You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Zhou Kang <zh...@outlook.com> on 2020/01/20 11:58:40 UTC
[Discuss] Add webhook to Kylin
Hi Kylin users & developers:
Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
Do we need to add webhook to Kylin? Such as when cubing job finished.
What do you think about this, and which is the better way in your environment ?
Re: [Discuss] Add webhook to Kylin
Posted by Luke Han <lu...@gmail.com>.
this is very cool stuff, looking forward for the first release.
Best Regards!
---------------------
Luke Han
On Thu, Feb 6, 2020 at 7:27 PM Yaqian Zhang <Ya...@126.com> wrote:
> Agree with xiaoxiang’s solution. I'm looking forward to this feature.
>
> 在 2020年2月6日，18:49，weibin zhu <co...@126.com> 写道：
>
> Thanks for the reply, if everyone agrees with this solution, I can create
> a jira and implement the code. I suggest that can define the interface and
> framework code first. If you are interested, you can review it. If there is
> no problem, then implement the complete code.
> @ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu
>
> ----------------------------------------
> weibin0516
>
> codingforfun@126.com
> http://rrd.me/fT6RW
> ----------------------------------------
>
> 在 2020年2月5日，下午11:19，Xiaoxiang Yu <xx...@apache.org> 写道：
>
> Hi codingforfun and zhoukang,
> I am agreed with codingforfun. Event, Listener, and ListenerBus, these
> components should be good abstraction/design for a monitor framework.
> Recently I am checking source code of System Cube and Dashboard
> Service in apache kylin, I have find some exist abstraction of these
> components(Event, Listener, and ListenerBus).
> We can find them under core-mertics module (
> https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). Maybe
> we can implement the webhook feature by extending this metrics framework
> other than reinventing the wheel. I am not very sure am I right but I guess
> it should works.
> It is a draft version of and I extend some basic interface
> under core-mertics module, I think it not perfect at the moment, this is my
> patch
> https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea .
> I supposed my draft is intended to the beginning of the conversation, not
> the final implementation. If you have any comments, please let me know,
> thank you in advance.
>
> To test it, use the following config in kylin.properties and
> kylinMetrics.xml.
>
> 1. kylin.properties
> kylin.metrics.reporter-job-enabled=true
> kylin.metrics.monitor-enabled=true
>
> 2. kylinMetrics.xml
> <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="http://www.springframework.org/schema/beans"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">
>
> <description>Kylin Metrics Related Configuration
> (SystemCube)</description>
>
> <!-- A Reservoir which don't staged metrics message at all, emit it in
> no time. Maybe good for debug purpose.-->
> <bean id="instantReservoir"
> class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
>
> <!-- A Reservoir which staged metrics message in memory, and emit them
> in fixed rate. -->
> <bean id="blockingReservoir"
> class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
> <!-- minReportSize, only if currently count of staged message
> exceed minReportSize, will Reservoir try to write message-->
> <constructor-arg index="0">
> <value>100</value>
> </constructor-arg>
>
> <!-- maxReportSize, max size of report in one time -->
> <constructor-arg index="1">
> <value>500</value>
> </constructor-arg>
>
> <!-- minReportTime, min duration(in minute) between two report
> action-->
> <constructor-arg index="2">
> <value>10</value>
> </constructor-arg>
> </bean>
>
> <bean id="hiveSink"
> class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
> <bean id="kafkaSink"
> class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
>
> <bean id="initMetricsManager"
> class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
> <property name="targetClass"
> value="org.apache.kylin.metrics.MetricsManager"/>
> <property name="targetMethod" value="initMetricsManager"/>
> <property name="arguments">
> <list>
> <!--
> Sink of System Cube.
> -->
> <ref bean="hiveSink"/>
>
> <!--
> Bind properties for each ActiveReservoirReporter.
> -->
> <map
> key-type="org.apache.kylin.metrics.lib.ActiveReservoir"
> value-type="java.util.List">
>
> <!-- Each ActiveReservoir can have multi
> ReservoirReporter -->
> <entry key-ref="instantReservoir">
> <list>
> <bean
> class="org.apache.kylin.common.util.Pair">
> <!-- Implementation of ReservoirReporter-->
> <property name="first"
>
> value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
> <!-- Properties for specific
> ReservoirReporter-->
> <property name="second">
> <props>
> <prop
> key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
> </props>
> </property>
> </bean>
>
> <bean
> class="org.apache.kylin.common.util.Pair">
> <!-- Implementation of ReservoirReporter-->
> <property name="first"
>
> value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
> <!-- Properties for specific
> ReservoirReporter-->
> <property name="second">
> <props>
> <prop
> key="callback.url">localhost:8888</prop>
> </props>
> </property>
> </bean>
> </list>
> </entry>
> </map>
> </list>
> </property>
> </bean>
> </beans>
>
>
>
>
> --
> *Best wishes to you ! *
> *From ：**Xiaoxiang Yu*
>
> At 2020-02-05 10:03:13, "codingforfun" <co...@126.com> wrote:
>
> I think we can refer to the implementation of spark listener, a few key
> points are as follows:
> 1. Define interface ListenerEvent, which can implement specific events.
> Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart,
> CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot
> of information, such as owner, success/failed, error msg, resource
> occupation, etc
> 2. Define interface Listener, which contains the monitoring of all events.
> When the user defines the Listener, some events can be selectively
> implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can
> be implemented to obtain the time cost.
> 3. Define a ListenerBus that provides the ability to register listeners(by
> configured custom listener class names), receive events, and dispatch
> events to the appropriate Listener in the background
>
>
>
> 在 2020年2月4日 11:17，ShaoFeng Shi <sh...@apache.org>写道：
>
> Good proposal. Is there some standard and popular framework for this? We
> can integrate with the best solution there.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Liukaige <et...@gmail.com> 于2020年1月25日周六 上午1:01写道：
>
>> Totally agreed. And this feature can be integrated with ETL scheduling,
>> data governance, approval flow etc. Brilliant idea. If you need any help,
>> count me in.
>>
>> 朱卫斌 <co...@126.com> 于2020年1月20日周一 下午10:10写道：
>>
>>> I think there should be a set of kylin's event, metrics mechanism,
>>> event-driven, provide event, metrics interfaces, we can implement specific
>>> listeners in the form of plugins. For example, we can implement dingtalk
>>> plugin, SMS plugin or any other plugin, which has a high Flexibility (Not
>>> only do notifications, but also do many things, such as unifying and even
>>> interfering with the task.). We can refer to the design and implementation
>>> of spark event and metrics.
>>> I think this is very valuable, we can do it together.
>>>
>>> weibin0516
>>> codingforfun@126.com
>>> Best wishes !
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>>
>>> On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> <xx...@apache.org>
>>> wrote：
>>>
>>> It looks good, and it should be useful for IT team, please go ahead!
>>>
>>>
>>> --
>>> *Best wishes to you ! *
>>> *From ：**Xiaoxiang Yu*
>>>
>>> At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
>>>
>>> Hi Kylin users & developers:
>>>
>>>
>>>
>>> Many apps support webhook, it is one way that apps can send
>>> automated messages or information to other apps.
>>>
>>> Use webhook, I think we can send messages to chat tools（slack,
>>> dingding）, send sms , trigger another workflow.
>>>
>>> Do we need to add webhook to Kylin? Such as when cubing job
>>> finished.
>>>
>>> What do you think about this, and which is the better way in your
>>> environment ?
>>>
>>>
>>
>> --
>> Best regards,
>>
>> Kaige Liu(刘凯歌)
>>
>> *"Do small things with great love." *
>>
>
>
>
Re: [Discuss] Add webhook to Kylin
Posted by Yaqian Zhang <Ya...@126.com>.
Agree with xiaoxiang’s solution. I'm looking forward to this feature.
> 在 2020年2月6日，18:49，weibin zhu <co...@126.com> 写道：
>
> Thanks for the reply, if everyone agrees with this solution, I can create a jira and implement the code. I suggest that can define the interface and framework code first. If you are interested, you can review it. If there is no problem, then implement the complete code.
> @ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu
>
> ----------------------------------------
> weibin0516
>
> codingforfun@126.com <ma...@126.com>
> http://rrd.me/fT6RW
> ----------------------------------------
>
>> 在 2020年2月5日，下午11:19，Xiaoxiang Yu <xxyu@apache.org <ma...@apache.org>> 写道：
>>
>> Hi codingforfun and zhoukang,
>> I am agreed with codingforfun. Event, Listener, and ListenerBus, these components should be good abstraction/design for a monitor framework.
>> Recently I am checking source code of System Cube and Dashboard Service in apache kylin, I have find some exist abstraction of these components(Event, Listener, and ListenerBus).
>> We can find them under core-mertics module (https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). <https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).> Maybe we can implement the webhook feature by extending this metrics framework other than reinventing the wheel. I am not very sure am I right but I guess it should works.
>> It is a draft version of and I extend some basic interface under core-mertics module, I think it not perfect at the moment, this is my patch https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea <https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea> . I supposed my draft is intended to the beginning of the conversation, not the final implementation. If you have any comments, please let me know, thank you in advance.
>>
>> To test it, use the following config in kylin.properties and kylinMetrics.xml.
>>
>> 1. kylin.properties
>> kylin.metrics.reporter-job-enabled=true
>> kylin.metrics.monitor-enabled=true
>>
>> 2. kylinMetrics.xml
>> <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance <http://www.w3.org/2001/XMLSchema-instance>"
>> xmlns="http://www.springframework.org/schema/beans <http://www.springframework.org/schema/beans>"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans <http://www.springframework.org/schema/beans>
>> http://www.springframework.org/schema/beans/spring-beans-3.1.xsd <http://www.springframework.org/schema/beans/spring-beans-3.1.xsd>">
>>
>> <description>Kylin Metrics Related Configuration (SystemCube)</description>
>>
>> <!-- A Reservoir which don't staged metrics message at all, emit it in no time. Maybe good for debug purpose.-->
>> <bean id="instantReservoir" class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
>>
>> <!-- A Reservoir which staged metrics message in memory, and emit them in fixed rate. -->
>> <bean id="blockingReservoir" class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
>> <!-- minReportSize, only if currently count of staged message exceed minReportSize, will Reservoir try to write message-->
>> <constructor-arg index="0">
>> <value>100</value>
>> </constructor-arg>
>>
>> <!-- maxReportSize, max size of report in one time -->
>> <constructor-arg index="1">
>> <value>500</value>
>> </constructor-arg>
>>
>> <!-- minReportTime, min duration(in minute) between two report action-->
>> <constructor-arg index="2">
>> <value>10</value>
>> </constructor-arg>
>> </bean>
>>
>> <bean id="hiveSink" class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
>> <bean id="kafkaSink" class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
>>
>> <bean id="initMetricsManager" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
>> <property name="targetClass" value="org.apache.kylin.metrics.MetricsManager"/>
>> <property name="targetMethod" value="initMetricsManager"/>
>> <property name="arguments">
>> <list>
>> <!--
>> Sink of System Cube.
>> -->
>> <ref bean="hiveSink"/>
>>
>> <!--
>> Bind properties for each ActiveReservoirReporter.
>> -->
>> <map key-type="org.apache.kylin.metrics.lib.ActiveReservoir" value-type="java.util.List">
>>
>> <!-- Each ActiveReservoir can have multi ReservoirReporter -->
>> <entry key-ref="instantReservoir">
>> <list>
>> <bean class="org.apache.kylin.common.util.Pair">
>> <!-- Implementation of ReservoirReporter-->
>> <property name="first"
>> value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
>> <!-- Properties for specific ReservoirReporter-->
>> <property name="second">
>> <props>
>> <prop key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
>> </props>
>> </property>
>> </bean>
>>
>> <bean class="org.apache.kylin.common.util.Pair">
>> <!-- Implementation of ReservoirReporter-->
>> <property name="first"
>> value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
>> <!-- Properties for specific ReservoirReporter-->
>> <property name="second">
>> <props>
>> <prop key="callback.url">localhost:8888</prop>
>> </props>
>> </property>
>> </bean>
>> </list>
>> </entry>
>> </map>
>> </list>
>> </property>
>> </bean>
>> </beans>
>>
>>
>>
>>
>> --
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>> At 2020-02-05 10:03:13, "codingforfun" <codingforfun@126.com <ma...@126.com>> wrote:
>>
>> I think we can refer to the implementation of spark listener, a few key points are as follows:
>> 1. Define interface ListenerEvent, which can implement specific events. Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart, CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of information, such as owner, success/failed, error msg, resource occupation, etc
>> 2. Define interface Listener, which contains the monitoring of all events. When the user defines the Listener, some events can be selectively implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can be implemented to obtain the time cost.
>> 3. Define a ListenerBus that provides the ability to register listeners(by configured custom listener class names), receive events, and dispatch events to the appropriate Listener in the background
>>
>>
>>
>> 在 2020年2月4日 11:17，ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org>>写道：
>> Good proposal. Is there some standard and popular framework for this? We can integrate with the best solution there.
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofengshi@apache.org <ma...@apache.org>
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html <https://kylin.apache.org/docs/gettingstarted/faq.html>
>> Join Kylin user mail group: user-subscribe@kylin.apache.org <ma...@kylin.apache.org>
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org <ma...@kylin.apache.org>
>>
>>
>>
>>
>> Liukaige <etherge@gmail.com <ma...@gmail.com>> 于2020年1月25日周六 上午1:01写道：
>> Totally agreed. And this feature can be integrated with ETL scheduling, data governance, approval flow etc. Brilliant idea. If you need any help, count me in.
>>
>> 朱卫斌 <codingforfun@126.com <ma...@126.com>> 于2020年1月20日周一 下午10:10写道：
>> I think there should be a set of kylin's event, metrics mechanism, event-driven, provide event, metrics interfaces, we can implement specific listeners in the form of plugins. For example, we can implement dingtalk plugin, SMS plugin or any other plugin, which has a high Flexibility (Not only do notifications, but also do many things, such as unifying and even interfering with the task.). We can refer to the design and implementation of spark event and metrics.
>> I think this is very valuable, we can do it together.
>>
>>
>>
>> weibin0516
>> codingforfun@126.com
>> Best wishes !
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>
>> On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> <ma...@apache.org> wrote：
>> It looks good, and it should be useful for IT team, please go ahead!
>>
>>
>> --
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>> At 2020-01-20 19:58:40, "Zhou Kang" <zhoukangcn@outlook.com <ma...@outlook.com>> wrote:
>>
>> Hi Kylin users & developers:
>>
>>
>>
>> Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
>>
>> Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
>>
>> Do we need to add webhook to Kylin? Such as when cubing job finished.
>>
>> What do you think about this, and which is the better way in your environment ?
>>
>>
>>
>> --
>> Best regards,
>>
>> Kaige Liu(刘凯歌)
>>
>> "Do small things with great love."
>
Re: [Discuss] Add webhook to Kylin
Posted by weibin zhu <co...@126.com>.
Thanks for the reply, if everyone agrees with this solution, I can create a jira and implement the code. I suggest that can define the interface and framework code first. If you are interested, you can review it. If there is no problem, then implement the complete code.
@ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu
----------------------------------------
weibin0516
codingforfun@126.com
http://rrd.me/fT6RW
----------------------------------------
> 在 2020年2月5日，下午11:19，Xiaoxiang Yu <xx...@apache.org> 写道：
>
> Hi codingforfun and zhoukang,
> I am agreed with codingforfun. Event, Listener, and ListenerBus, these components should be good abstraction/design for a monitor framework.
> Recently I am checking source code of System Cube and Dashboard Service in apache kylin, I have find some exist abstraction of these components(Event, Listener, and ListenerBus).
> We can find them under core-mertics module (https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). <https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).> Maybe we can implement the webhook feature by extending this metrics framework other than reinventing the wheel. I am not very sure am I right but I guess it should works.
> It is a draft version of and I extend some basic interface under core-mertics module, I think it not perfect at the moment, this is my patch https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea <https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea> . I supposed my draft is intended to the beginning of the conversation, not the final implementation. If you have any comments, please let me know, thank you in advance.
>
> To test it, use the following config in kylin.properties and kylinMetrics.xml.
>
> 1. kylin.properties
> kylin.metrics.reporter-job-enabled=true
> kylin.metrics.monitor-enabled=true
>
> 2. kylinMetrics.xml
> <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="http://www.springframework.org/schema/beans"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">
>
> <description>Kylin Metrics Related Configuration (SystemCube)</description>
>
> <!-- A Reservoir which don't staged metrics message at all, emit it in no time. Maybe good for debug purpose.-->
> <bean id="instantReservoir" class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
>
> <!-- A Reservoir which staged metrics message in memory, and emit them in fixed rate. -->
> <bean id="blockingReservoir" class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
> <!-- minReportSize, only if currently count of staged message exceed minReportSize, will Reservoir try to write message-->
> <constructor-arg index="0">
> <value>100</value>
> </constructor-arg>
>
> <!-- maxReportSize, max size of report in one time -->
> <constructor-arg index="1">
> <value>500</value>
> </constructor-arg>
>
> <!-- minReportTime, min duration(in minute) between two report action-->
> <constructor-arg index="2">
> <value>10</value>
> </constructor-arg>
> </bean>
>
> <bean id="hiveSink" class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
> <bean id="kafkaSink" class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
>
> <bean id="initMetricsManager" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
> <property name="targetClass" value="org.apache.kylin.metrics.MetricsManager"/>
> <property name="targetMethod" value="initMetricsManager"/>
> <property name="arguments">
> <list>
> <!--
> Sink of System Cube.
> -->
> <ref bean="hiveSink"/>
>
> <!--
> Bind properties for each ActiveReservoirReporter.
> -->
> <map key-type="org.apache.kylin.metrics.lib.ActiveReservoir" value-type="java.util.List">
>
> <!-- Each ActiveReservoir can have multi ReservoirReporter -->
> <entry key-ref="instantReservoir">
> <list>
> <bean class="org.apache.kylin.common.util.Pair">
> <!-- Implementation of ReservoirReporter-->
> <property name="first"
> value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
> <!-- Properties for specific ReservoirReporter-->
> <property name="second">
> <props>
> <prop key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
> </props>
> </property>
> </bean>
>
> <bean class="org.apache.kylin.common.util.Pair">
> <!-- Implementation of ReservoirReporter-->
> <property name="first"
> value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
> <!-- Properties for specific ReservoirReporter-->
> <property name="second">
> <props>
> <prop key="callback.url">localhost:8888</prop>
> </props>
> </property>
> </bean>
> </list>
> </entry>
> </map>
> </list>
> </property>
> </bean>
> </beans>
>
>
>
>
> --
> Best wishes to you !
> From ：Xiaoxiang Yu
>
> At 2020-02-05 10:03:13, "codingforfun" <co...@126.com> wrote:
>
> I think we can refer to the implementation of spark listener, a few key points are as follows:
> 1. Define interface ListenerEvent, which can implement specific events. Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart, CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of information, such as owner, success/failed, error msg, resource occupation, etc
> 2. Define interface Listener, which contains the monitoring of all events. When the user defines the Listener, some events can be selectively implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can be implemented to obtain the time cost.
> 3. Define a ListenerBus that provides the ability to register listeners(by configured custom listener class names), receive events, and dispatch events to the appropriate Listener in the background
>
>
>
> 在 2020年2月4日 11:17，ShaoFeng Shi <sh...@apache.org>写道：
> Good proposal. Is there some standard and popular framework for this? We can integrate with the best solution there.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org <ma...@apache.org>
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html <https://kylin.apache.org/docs/gettingstarted/faq.html>
> Join Kylin user mail group: user-subscribe@kylin.apache.org <ma...@kylin.apache.org>
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org <ma...@kylin.apache.org>
>
>
>
>
> Liukaige <etherge@gmail.com <ma...@gmail.com>> 于2020年1月25日周六 上午1:01写道：
> Totally agreed. And this feature can be integrated with ETL scheduling, data governance, approval flow etc. Brilliant idea. If you need any help, count me in.
>
> 朱卫斌 <codingforfun@126.com <ma...@126.com>> 于2020年1月20日周一 下午10:10写道：
> I think there should be a set of kylin's event, metrics mechanism, event-driven, provide event, metrics interfaces, we can implement specific listeners in the form of plugins. For example, we can implement dingtalk plugin, SMS plugin or any other plugin, which has a high Flexibility (Not only do notifications, but also do many things, such as unifying and even interfering with the task.). We can refer to the design and implementation of spark event and metrics.
> I think this is very valuable, we can do it together.
>
>
>
> weibin0516
> codingforfun@126.com
> Best wishes !
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> <ma...@apache.org> wrote：
> It looks good, and it should be useful for IT team, please go ahead!
>
>
> --
> Best wishes to you !
> From ：Xiaoxiang Yu
>
> At 2020-01-20 19:58:40, "Zhou Kang" <zhoukangcn@outlook.com <ma...@outlook.com>> wrote:
>
> Hi Kylin users & developers:
>
>
>
> Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
>
> Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
>
> Do we need to add webhook to Kylin? Such as when cubing job finished.
>
> What do you think about this, and which is the better way in your environment ?
>
>
>
> --
> Best regards,
>
> Kaige Liu(刘凯歌)
>
> "Do small things with great love."
Re: [Discuss] Add webhook to Kylin
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi codingforfun and zhoukang,
I am agreed with codingforfun. Event, Listener, and ListenerBus, these components should be good abstraction/design for a monitor framework.
Recently I am checking source code of System Cube and Dashboard Service in apache kylin, I have find some exist abstraction of these components(Event, Listener, and ListenerBus).
We can find them under core-mertics module (https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). Maybe we can implement the webhook feature by extending this metrics framework other than reinventing the wheel. I am not very sure am I right but I guess it should works.
It is a draft version of and I extend some basic interface under core-mertics module, I think it not perfect at the moment, this is my patch https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea . I supposed my draft is intended to the beginning of the conversation, not the final implementation. If you have any comments, please let me know, thank you in advance.
To test it, use the following config in kylin.properties and kylinMetrics.xml.
1. kylin.properties
kylin.metrics.reporter-job-enabled=true
kylin.metrics.monitor-enabled=true
2. kylinMetrics.xml
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.springframework.org/schema/beans"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">
<description>Kylin Metrics Related Configuration (SystemCube)</description>
<!-- A Reservoir which don't staged metrics message at all, emit it in no time. Maybe good for debug purpose.-->
<bean id="instantReservoir" class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
<!-- A Reservoir which staged metrics message in memory, and emit them in fixed rate. -->
<bean id="blockingReservoir" class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
<!-- minReportSize, only if currently count of staged message exceed minReportSize, will Reservoir try to write message-->
<constructor-arg index="0">
<value>100</value>
</constructor-arg>
<!-- maxReportSize, max size of report in one time -->
<constructor-arg index="1">
<value>500</value>
</constructor-arg>
<!-- minReportTime, min duration(in minute) between two report action-->
<constructor-arg index="2">
<value>10</value>
</constructor-arg>
</bean>
<bean id="hiveSink" class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
<bean id="kafkaSink" class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
<bean id="initMetricsManager" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetClass" value="org.apache.kylin.metrics.MetricsManager"/>
<property name="targetMethod" value="initMetricsManager"/>
<property name="arguments">
<list>
<!--
Sink of System Cube.
-->
<ref bean="hiveSink"/>
<!--
Bind properties for each ActiveReservoirReporter.
-->
<map key-type="org.apache.kylin.metrics.lib.ActiveReservoir" value-type="java.util.List">
<!-- Each ActiveReservoir can have multi ReservoirReporter -->
<entry key-ref="instantReservoir">
<list>
<bean class="org.apache.kylin.common.util.Pair">
<!-- Implementation of ReservoirReporter-->
<property name="first"
value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
<!-- Properties for specific ReservoirReporter-->
<property name="second">
<props>
<prop key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
</props>
</property>
</bean>
<bean class="org.apache.kylin.common.util.Pair">
<!-- Implementation of ReservoirReporter-->
<property name="first"
value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
<!-- Properties for specific ReservoirReporter-->
<property name="second">
<props>
<prop key="callback.url">localhost:8888</prop>
</props>
</property>
</bean>
</list>
</entry>
</map>
</list>
</property>
</bean>
</beans>
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-02-05 10:03:13, "codingforfun" <co...@126.com> wrote:
I think we can refer to the implementation of spark listener, a few key points are as follows:
1. Define interface ListenerEvent, which can implement specific events. Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart, CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of information, such as owner, success/failed, error msg, resource occupation, etc
2. Define interface Listener, which contains the monitoring of all events. When the user defines the Listener, some events can be selectively implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can be implemented to obtain the time cost.
3. Define a ListenerBus that provides the ability to register listeners(by configured custom listener class names), receive events, and dispatch events to the appropriate Listener in the background
在 2020年2月4日 11:17，ShaoFeng Shi <sh...@apache.org>写道：
Good proposal. Is there some standard and popular framework for this? We can integrate with the best solution there.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Liukaige <et...@gmail.com> 于2020年1月25日周六 上午1:01写道：
Totally agreed. And this feature can be integrated with ETL scheduling, data governance, approval flow etc. Brilliant idea. If you need any help, count me in.
朱卫斌 <co...@126.com> 于2020年1月20日周一 下午10:10写道：
I think there should be a set of kylin's event, metrics mechanism, event-driven, provide event, metrics interfaces, we can implement specific listeners in the form of plugins. For example, we can implement dingtalk plugin, SMS plugin or any other plugin, which has a high Flexibility (Not only do notifications, but also do many things, such as unifying and even interfering with the task.). We can refer to the design and implementation of spark event and metrics.
I think this is very valuable, we can do it together.
| |
weibin0516
|
|
codingforfun@126.com
Best wishes !
|
签名由网易邮箱大师定制
On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> wrote：
It looks good, and it should be useful for IT team, please go ahead!
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
Hi Kylin users & developers:
Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
Do we need to add webhook to Kylin? Such as when cubing job finished.
What do you think about this, and which is the better way in your environment ?
--
Best regards,
Kaige Liu(刘凯歌)
"Do small things with great love."
Re: [Discuss] Add webhook to Kylin
Posted by codingforfun <co...@126.com>.
I think we can refer to the implementation of spark listener, a few key points
are as follows:
1\. Define interface ListenerEvent, which can implement specific events.
Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart,
CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of
information, such as owner, success/failed, error msg, resource occupation,
etc
2\. Define interface Listener, which contains the monitoring of all events.
When the user defines the Listener, some events can be selectively implemented
as needed, for example, onCubeBuildStart and CubeBuildEnd can be implemented
to obtain the time cost.
3\. Define a ListenerBus that provides the ability to register listeners(by
configured custom listener class names), receive events, and dispatch events
to the appropriate Listener in the background
在 2020年2月4日 11:17，ShaoFeng Shi <sh...@apache.org>写道：
> Good proposal. Is there some standard and popular framework for this? We can
integrate with the best solution there.
>
>
>
>
> Best regards,
>
>
>
>
> Shaofeng Shi 史少锋
>
> Apache Kylin PMC
>
> Email: [shaofengshi@apache.org](mailto:shaofengshi@apache.org)
>
>
>
>
> Apache Kylin FAQ: <https://kylin.apache.org/docs/gettingstarted/faq.html>
>
> Join Kylin user mail group: [user-subscribe@kylin.apache.org](mailto:user-
subscribe@kylin.apache.org)
>
> Join Kylin dev mail group: [dev-subscribe@kylin.apache.org](mailto:dev-
subscribe@kylin.apache.org)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Liukaige <[etherge@gmail.com](mailto:etherge@gmail.com)> 于2020年1月25日周六
上午1:01写道：
>
>
>> Totally agreed. And this feature can be integrated with ETL scheduling,
data governance, approval flow etc. Brilliant idea. If you need any help,
count me in.
>>
>>
>
>>
>> 朱卫斌 <[codingforfun@126.com](mailto:codingforfun@126.com)> 于2020年1月20日周一
下午10:10写道：
>
>>
>>> I think there should be a set of kylin's event, metrics mechanism, event-
driven, provide event, metrics interfaces, we can implement specific listeners
in the form of plugins. For example, we can implement dingtalk plugin, SMS
plugin or any other plugin, which has a high Flexibility (Not only do
notifications, but also do many things, such as unifying and even interfering
with the task.). We can refer to the design and implementation of spark event
and metrics.
>>>
>>> I think this is very valuable, we can do it together.
>>>
>>>
>
>>>
>>> [  |
weibin0516
>>> ---|---
>>> codingforfun@126.com Best wishes !
>>> ](https://maas.mail.163.com/dashi-web-
extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-
online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D)
>>>
>>> 签名由 [网易邮箱大师](https://mail.163.com/dashi/dlpro.html?from=mail81) 定制
>>>
>>>
>
>>>
>>> On 01/21/2020 10:57，[Xiaoxiang
Yu<xx...@apache.org>](mailto:xxyu@apache.org) wrote：
>>>
>>>> It looks good, and it should be useful for IT team, please go ahead!
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>> \--
>
>>>>
>>>> **Best wishes to you !**
>>>>
>>>> **From ：** **Xiaoxiang Yu**
>>>>
>>>>
>
>>>>
>>>> At 2020-01-20 19:58:40, "Zhou Kang"
<[zhoukangcn@outlook.com](mailto:zhoukangcn@outlook.com)> wrote:
>>>>
>>>>> Hi Kylin users & developers: __ __
>>>>>
>>>>> ____
>>>>>
>>>>> Many apps support webhook, it is one way that apps can send automated
messages or information to other apps. __ __
>>>>>
>>>>> Use webhook, I think we can send messages to chat tools （slack,
dingding）, send sms , trigger another workflow. __ __
>>>>>
>>>>> Do we need to add webhook to Kylin? Such as when cubing job finished. __
__
>>>>>
>>>>> What do you think about this, and which is the better way in your
environment ? __ __
>>
>>
>
>>
>>
>
>>
>> \--
>
>>
>> Best regards,
>>
>>
>
>>
>> Kaige Liu(刘凯歌)
>>
>>
>
>>
>> _" Do small things with great love." _
>
Re: [Discuss] Add webhook to Kylin
Posted by ShaoFeng Shi <sh...@apache.org>.
Good proposal. Is there some standard and popular framework for this? We
can integrate with the best solution there.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Liukaige <et...@gmail.com> 于2020年1月25日周六 上午1:01写道：
> Totally agreed. And this feature can be integrated with ETL scheduling,
> data governance, approval flow etc. Brilliant idea. If you need any help,
> count me in.
>
> 朱卫斌 <co...@126.com> 于2020年1月20日周一 下午10:10写道：
>
>> I think there should be a set of kylin's event, metrics mechanism,
>> event-driven, provide event, metrics interfaces, we can implement specific
>> listeners in the form of plugins. For example, we can implement dingtalk
>> plugin, SMS plugin or any other plugin, which has a high Flexibility (Not
>> only do notifications, but also do many things, such as unifying and even
>> interfering with the task.). We can refer to the design and implementation
>> of spark event and metrics.
>> I think this is very valuable, we can do it together.
>>
>> weibin0516
>> codingforfun@126.com
>> Best wishes !
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>
>> On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> <xx...@apache.org>
>> wrote：
>>
>> It looks good, and it should be useful for IT team, please go ahead!
>>
>>
>> --
>> *Best wishes to you ! *
>> *From ：**Xiaoxiang Yu*
>>
>> At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
>>
>> Hi Kylin users & developers:
>>
>>
>>
>> Many apps support webhook, it is one way that apps can send
>> automated messages or information to other apps.
>>
>> Use webhook, I think we can send messages to chat tools（slack,
>> dingding）, send sms , trigger another workflow.
>>
>> Do we need to add webhook to Kylin? Such as when cubing job
>> finished.
>>
>> What do you think about this, and which is the better way in your
>> environment ?
>>
>>
>
> --
> Best regards,
>
> Kaige Liu(刘凯歌)
>
> *"Do small things with great love." *
>
Re: [Discuss] Add webhook to Kylin
Posted by Liukaige <et...@gmail.com>.
Totally agreed. And this feature can be integrated with ETL scheduling,
data governance, approval flow etc. Brilliant idea. If you need any help,
count me in.
朱卫斌 <co...@126.com> 于2020年1月20日周一 下午10:10写道：
> I think there should be a set of kylin's event, metrics mechanism,
> event-driven, provide event, metrics interfaces, we can implement specific
> listeners in the form of plugins. For example, we can implement dingtalk
> plugin, SMS plugin or any other plugin, which has a high Flexibility (Not
> only do notifications, but also do many things, such as unifying and even
> interfering with the task.). We can refer to the design and implementation
> of spark event and metrics.
> I think this is very valuable, we can do it together.
>
> weibin0516
> codingforfun@126.com
> Best wishes !
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> <xx...@apache.org>
> wrote：
>
> It looks good, and it should be useful for IT team, please go ahead!
>
>
> --
> *Best wishes to you ! *
> *From ：**Xiaoxiang Yu*
>
> At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
>
> Hi Kylin users & developers:
>
>
>
> Many apps support webhook, it is one way that apps can send
> automated messages or information to other apps.
>
> Use webhook, I think we can send messages to chat tools（slack,
> dingding）, send sms , trigger another workflow.
>
> Do we need to add webhook to Kylin? Such as when cubing job
> finished.
>
> What do you think about this, and which is the better way in your
> environment ?
>
>
--
Best regards,
Kaige Liu(刘凯歌)
*"Do small things with great love." *
Re:[Discuss] Add webhook to Kylin
Posted by 朱卫斌 <co...@126.com>.
I think there should be a set of kylin's event, metrics mechanism, event-driven, provide event, metrics interfaces, we can implement specific listeners in the form of plugins. For example, we can implement dingtalk plugin, SMS plugin or any other plugin, which has a high Flexibility (Not only do notifications, but also do many things, such as unifying and even interfering with the task.). We can refer to the design and implementation of spark event and metrics.
I think this is very valuable, we can do it together.
| |
weibin0516
|
|
codingforfun@126.com
Best wishes !
|
签名由网易邮箱大师定制
On 01/21/2020 10:57，Xiaoxiang Yu<xx...@apache.org> wrote：
It looks good, and it should be useful for IT team, please go ahead!
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
Hi Kylin users & developers:
Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
Do we need to add webhook to Kylin? Such as when cubing job finished.
What do you think about this, and which is the better way in your environment ?
Re:[Discuss] Add webhook to Kylin
Posted by Xiaoxiang Yu <xx...@apache.org>.
It looks good, and it should be useful for IT team, please go ahead!
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-01-20 19:58:40, "Zhou Kang" <zh...@outlook.com> wrote:
Hi Kylin users & developers:
Many apps support webhook, it is one way that apps can send automated messages or information to other apps.
Use webhook, I think we can send messages to chat tools（slack, dingding）, send sms , trigger another workflow.
Do we need to add webhook to Kylin? Such as when cubing job finished.
What do you think about this, and which is the better way in your environment ?