You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/07 03:00:46 UTC
[GitHub] [incubator-seatunnel] leo65535 opened a new issue #965: [DISCUSS][metrics] Support metrics statistics
leo65535 opened a new issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Support metrics statistics when transmitting data.
### Usage Scenario
-
### Related issues
-
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 edited a comment on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
xtr1993 edited a comment on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020765690
Spark metrics can be displayed in reports
but what can flink indicators be used to display? flink pre-job should be used to display
@yuangjiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
xtr1993 commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020765690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1013829725
this is a seatunnel metric system

and Description of monitoring indicators
#数据读取耗时/s
readTime
#任务读取输入数据总量
inputRecordsRead
#数据传输记录数
recordsRead
#读取字节大小
readBytes
#数据写入记录数
recordsWritten
#数据写入字节大小
writeBytes
#最终结果大小
resultSize
#任务GC时间
jvmGCTime
#任务执行CPU时间
taskCpuTime
#任务执行总共耗时
taskRunTime
#数据写入耗时/s
writeTime
#内存使用峰值
peakExecutionMemory
#任务创建时间
createTime

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] mosence edited a comment on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
mosence edited a comment on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023850802
I found the latest Spark (3.X) Metrics Document [https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers](https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers).
And Flink Metics Document [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics).
In per-job Metric Repotor:
Spark can config History Event and Use spark-historyserver. [https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files](https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files)
Flink can config Repotor. [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters .
And most of list the metics include them.
Should we discuss the Top design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020768362
If flink needs to collect indicators to generate reports, it should define a high-level abstraction, which will be better collected.
------------------ 原始邮件 ------------------
发件人: "apache/incubator-seatunnel" ***@***.***>;
发送时间: 2022年1月25日(星期二) 中午11:09
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/incubator-seatunnel] [DISCUSS][metrics] Support metrics statistics (Issue #965)
Spark metrics can be displayed in reports
but what can flink indicators be used to display? flink pre-job should be used to display
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you commented.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007223366
Good job, We can discuss what metrics are needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007460506
the plugin can consider adding the following monitoring indicators:
1. Read the number of records
2. Read size
3. CPU time
4. Consumption of memory size
5. Number of concurrent threads of execution
7. Number of records written
8. Write data size
9. Consume CPU time
10. Consumption of memory size
11. Number of concurrent threads
You can consider designing and integrating these monitoring indicators from SparkListener, but you need to consider how to design it reasonably. Recently, we are also considering adding this part of the function to discuss together

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020774217
We tried to use SparkListener to collect task metrics but the results are not accurate, only the number of records is more reliable
------------------ 原始邮件 ------------------
发件人: "apache/incubator-seatunnel" ***@***.***>;
发送时间: 2022年1月25日(星期二) 中午11:23
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/incubator-seatunnel] [DISCUSS][metrics] Support metrics statistics (Issue #965)
yes,I approve of your technical feasibility,
But flink metrics are real-time,Can the report be provided in real time
If an API interface is provided to the user such as doris streamLoad
@yuangjiang
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wu-sheng commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020782559
If you want to see how Meter APIs are designed in SkyWalking, these two are good
1. Source codes, https://github.com/apache/skywalking-java/blob/main/apm-application-toolkit/apm-toolkit-meter/src/main/java/org/apache/skywalking/apm/toolkit/meter/MeterFactory.java
2. APIs docs, https://skywalking.apache.org/docs/skywalking-java/v8.8.0/en/setup/service-agent/java-agent/application-toolkit-meter/
Also, Prometheus APIs should be a good reference(but Java's Prometheus APIs are not very widely used). MicroMeter(Sleuth) APIs are good too.
SkyWalking's APIs could adopt Sleuth APIs to meter APIs.
https://skywalking.apache.org/docs/skywalking-java/v8.8.0/en/setup/service-agent/java-agent/application-toolkit-micrometer/
__
The reason for sharing these APIs are meaning, metrics APIs are already a stable system, most systems don't need a complex internal mechanism to do that. Try to check those existing APIs, and make sure you have enough mechanisms to forward your metrics to their core. This should be better than doing these internally. After all, the metrics would not be managed, calculated, and visualized inside your project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 edited a comment on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
xtr1993 edited a comment on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020765690
Spark metrics can be displayed in reports
but what can flink indicators be used to display? flink pre-job should be used to display
@yuangjiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020759676
> I have a question, is Flink also able to provide corresponding indicators?
Moreover, if they have already provided corresponding monitoring, what is the point of us doing another layer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020754348
I have a question, is Flink also able to provide corresponding indicators?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023909394
> I found the latest Spark (3.X) Metrics Document https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers. And Flink Metics Document https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics.
>
> In per-job Metric Repotor: Spark can config History Event and Use spark-historyserver. https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files
>
> Flink can config Repotor. [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters .
>
> And most of list the metics include them.
>
> Should we discuss the Top design?
The only reason I can convince me is that we have implemented platform shielding for users. Whether it is index collection from Flink or Spark, we can collect their metrics. BTW, I would recommend that we integrate with other APM applications, such as SkyWalking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020777987
> > @wu-sheng @kezhenxu94 Hi, sorry to bother you, can you give us some suggestions?
>
> Hi, as this is a Java ecosystem project, most observability tool has no requirement for how the system is built. Typically, a common and widely used observability platform has provided metrics exporting/collecting agents to inject codes into your codes to collect metrics or traffic.
>
> What kind of suggestions do you ask about?
I want you to see the design here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007460506
the plugin can consider adding the following monitoring indicators:
1. Read the number of records
2. Read size
3. CPU time
4. Consumption of memory size
5. Number of concurrent threads of execution
7. Number of records written
8. Write data size
9. Consume CPU time
10. Consumption of memory size
11. Number of concurrent threads
You can consider designing and integrating these monitoring indicators from SparkListener, but you need to consider how to design it reasonably. Recently, we are also considering adding this part of the function to discuss together

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1019199357
Spark tasks are divided into shuffle tasks and non-shuffle tasks:
1. The shuffle task indicator is placed in @shuffleMetircs
2. Non-shuffle tasks are placed in input/outputMetrics
Question 1: How to ensure the accuracy of the indicator data is compatible with two types of indicator statistics at the same time?
Question 2: In the process of index statistics, we found that the data size index statistics are inaccurate and need to be solved
Question 3: The number of statistical data can be statistically accurate
@CalvinKirs @garyelephant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wu-sheng commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020774558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] mosence commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
mosence commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023850802
I found the latest Spark (3.X) Metrics Document [https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers](https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers).
And Flink Metics Document [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics).
In per-job Metric Repotor:
Spark can config History Event and Use spark-historyserver. [https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files](https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files)
Flink can config Repotor. [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters]https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters .
And most of list the metics include them.
Should we discuss the Top design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] tmljob commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
tmljob commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1063713382
> Is there a solution to this problem? How do you collect metrics for specific etl tasks in practice?
like DataX,as flows:
```
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 任务启动时刻 : 2022-03-05 14:00:01
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 任务结束时刻 : 2022-03-05 14:00:22
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 任务总计耗时 : 21s
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 任务平均流量 : 1.53MB/s
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 记录写入速度 : 1391rec/s
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 读出记录总数 : 27824
2022-03-05 14:00:22 [AnalysisStatistics.analysisStatisticsLog-53] 读写失败总数 : 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007996869
The following solutions can be used for plug-in status monitoring:
1. Create a SparkMetricListener and register it in sparkcontext.
2. Add the indicators that need to be output
the following is a sample output of our internal engineering test indicators:

need to confirm whether this option is feasible
@garyelephant
@CalvinKirs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
xtr1993 commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020765690
Spark metrics can be displayed in reports
but what can flink indicators be used to display? flink pre-job should be used to display
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wu-sheng commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020774558
> @wu-sheng @kezhenxu94 Hi, sorry to bother you, can you give us some suggestions?
Hi, as this is a Java ecosystem project, most observability tool has no requirement for how the system is built.
Typically, a common and widely used observability platform has provided metrics exporting/collecting agents to inject codes into your codes to collect metrics or traffic.
What kind of suggestions do you ask about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
xtr1993 commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020772882
>
yes,I approve of your technical feasibility,
But flink metrics are real-time,Can the report be provided in real time
If an API interface is provided to the user such as doris streamLoad
@yuangjiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] tmljob commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
tmljob commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1051661169
Is there a solution to this problem? How do you collect metrics for specific etl tasks in practice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020791998
> If you want to see how Meter APIs are designed in SkyWalking, these two are good
>
> 1. Source codes, https://github.com/apache/skywalking-java/blob/main/apm-application-toolkit/apm-toolkit-meter/src/main/java/org/apache/skywalking/apm/toolkit/meter/MeterFactory.java
> 2. APIs docs, https://skywalking.apache.org/docs/skywalking-java/v8.8.0/en/setup/service-agent/java-agent/application-toolkit-meter/
>
> Also, Prometheus APIs should be a good reference(but Java's Prometheus APIs are not very widely used). MicroMeter(Sleuth) APIs are good too. SkyWalking's APIs could adopt Sleuth APIs to meter APIs. https://skywalking.apache.org/docs/skywalking-java/v8.8.0/en/setup/service-agent/java-agent/application-toolkit-micrometer/
>
> __
>
> The reason for sharing these APIs are meaning, metrics APIs are already a stable system, most systems don't need a complex internal mechanism to do that. Try to check those existing APIs, and make sure you have enough mechanisms to forward your metrics to their core. This should be better than doing these internally. After all, the metrics would not be managed, calculated, and visualized inside your project.
Deeply thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023903645
Debezium's cdc based on spark structured streaming is feasible. Need to custom develop a spark structured streaming connector
The following is the defined test code
val parameters = new util.HashMap[String,String]()
parameters.put("database.hostname","localhost")
parameters.put("database.port","3306")
parameters.put("database.user","")
parameters.put("database.password","")
parameters.put("database.include.list","temp")
parameters.put("table.include.list","temp.data")
parameters.put("connector.class","io.debezium.connector.mysql.MySqlConnector")
val dataFrame = sparkSession
.readStream
.options(parameters)
.schema(structType)
.format("org.apache.spark.sql.execution.streaming.debezium.DefaultSource")
.load()
dataFrame.createOrReplaceTempView("test")
sparkSession.sql("select * from test")
.writeStream
.format("console")
.outputMode(OutputMode.Append())
.start()
.awaitTermination()
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023925577
SkyWalking how to collect spark internal metrics statistics,using java agent?
------------------ 原始邮件 ------------------
发件人: "apache/incubator-seatunnel" ***@***.***>;
发送时间: 2022年1月28日(星期五) 中午1:53
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/incubator-seatunnel] [DISCUSS][metrics] Support metrics statistics (Issue #965)
I found the latest Spark (3.X) Metrics Document https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers. And Flink Metics Document https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics.
In per-job Metric Repotor: Spark can config History Event and Use spark-historyserver. https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files
Flink can config Repotor. [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters .
And most of list the metics include them.
Should we discuss the Top design?
The only reason I can convince me is that we have implemented platform shielding for users. Whether it is index collection from Flink or Spark, we can collect their metrics. BTW, I would recommend that we integrate with other APM applications, such as SkyWalking.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1023959513
> SkyWalking how to collect spark internal metrics statistics,using java agent?
> […](#)
> ------------------ 原始邮件 ------------------ 发件人: "apache/incubator-seatunnel" ***@***.***>; 发送时间: 2022年1月28日(星期五) 中午1:53 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/incubator-seatunnel] [DISCUSS][metrics] Support metrics statistics (Issue #965) I found the latest Spark (3.X) Metrics Document https://spark.apache.org/docs/latest/monitoring.html#list-of-available-metrics-providers. And Flink Metics Document https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#system-metrics. In per-job Metric Repotor: Spark can config History Event and Use spark-historyserver. https://spark.apache.org/docs/latest/monitoring.html#applying-compaction-on-rolling-event-log-files Flink can config Repotor. [https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_reporters/#metric-reporters] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/metric_rep
orters/#metric-reporters . And most of list the metics include them. Should we discuss the Top design? The only reason I can convince me is that we have implemented platform shielding for users. Whether it is index collection from Flink or Spark, we can collect their metrics. BTW, I would recommend that we integrate with other APM applications, such as SkyWalking. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: ***@***.***>
I overlooked a point, our metrics are already calculated, (got it from Flink) I have no other problem with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020764434
@wu-sheng @kezhenxu94 Hi, sorry to bother you, can you give us some suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020768362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1020754348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang edited a comment on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
yuangjiang edited a comment on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007996869
The following solutions can be used for plugin status monitoring:
1. Create a SparkMetricListener and register it in sparkcontext.
2. Add the indicators that need to be output
the following is a sample output of our internal engineering test indicators:

need to confirm whether this option is feasible
@garyelephant
@CalvinKirs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #965: [DISCUSS][metrics] Support metrics statistics
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #965:
URL: https://github.com/apache/incubator-seatunnel/issues/965#issuecomment-1007223366
Good job, We can discuss what metrics are needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org