You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by "AdeelAslamUnimore (via GitHub)" <gi...@apache.org> on 2023/04/30 14:54:12 UTC
[GitHub] [incubator-wayang] AdeelAslamUnimore opened a new pull request, #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
AdeelAslamUnimore opened a new pull request, #311:
URL: https://github.com/apache/incubator-wayang/pull/311
I have made some updates to enhance the monitoring of runtime metrics for the underlying platform. Initially, I implement for Spark big data platform, however, I will extend it for other underlying platform of Apache Wayang for debugging purpose Platform-level runtime metrics monitoring allows for more granular resource utilization and performance metrics tracking during Spark job execution.
This will enable more effective monitoring and optimization of platform job performance, and provide greater visibility into resource utilization, allowing for more efficient resource allocation and troubleshooting.
Please let me know if you have any questions or feedback on these updates. Thank you for considering this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1539565450
Great! thank you! @2pk03 We can merge this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1534558013
Hi @AdeelAslamUnimore, there are some compilation errors. Can you please take a look?
https://github.com/apache/incubator-wayang/actions/runs/4881291333/jobs/8710465408?pr=311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] kbeedkar commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "kbeedkar (via GitHub)" <gi...@apache.org>.
kbeedkar commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529610595
Hi guys, I went through the code. Overall, while the PR extends WayangContext, it does not affect any existing methods so this should not break anything. A few other issues with the code are
1. package name should start in lowercase, also there is a typo (Intefaces --> interfaces)
2. package names should be more meaningful and be consistent with the code base. spark_monitoring --> spark-monitoring. I would suggest having another meaningful name as the package is already under /spark/monitoring
3. Comments should not have @author tags (or?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] kbeedkar commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "kbeedkar (via GitHub)" <gi...@apache.org>.
kbeedkar commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529503549
Good point. There is already a Monitor abstract class, and the current WayangContext provides an execute() method with Monitor. That would be a clean way IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529463234
If sqlcontext does the same (or some other module), then if someone needs to monitor jobs using the sql api, how would he do it? which context would he have to use? Just brainstorming here..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] 2pk03 commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1534153876
This PR is behind our current main, can you please rebase? We updated pom.xml to pass Eclipse licenses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] AdeelAslamUnimore commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "AdeelAslamUnimore (via GitHub)" <gi...@apache.org>.
AdeelAslamUnimore commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1534904182
Hi @zkaoudi The compilation problem occur is due to **spark_2.11**. Some definitions used in our SparkListener are defined only for wayang spark_2.12, i.e., executorInfo.resourceProfileId() is not present in spark_2.11. Similarly, for stageExecutorMetrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] 2pk03 merged pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 merged PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] AdeelAslamUnimore commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "AdeelAslamUnimore (via GitHub)" <gi...@apache.org>.
AdeelAslamUnimore commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1538551693
Hi @zkaoudi,
I have discovered that there are three conflicts for sparkListener with Wayang_Spark2.11 and Wayang_Spark2.12. Therefore, I am not utilizing the updated (2.12 SparkListener) attributes in my ownSparkListener for monitoring purposes. Now it should work for both releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529455386
Hi all,
I'm wondering if modifying WayangContext and Job to include the hackIT variables/methods is the "best" way to go. Another alternative would be to create a new class eg. WayangContextWithMonitoring which extends from wayang context and contains whatever is necessary. Same for the Job class.
What do you think? @JorgeQuiane @kbeedkar Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1530955027
Ok then! We keep the wayangcontext as is.
@AdeelAslamUnimore can you please make the required changes that @kbeedkar?
Then we can merge :)
BTW, I think there are no author tags in general in any class. I think it's an Apache decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] kbeedkar commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "kbeedkar (via GitHub)" <gi...@apache.org>.
kbeedkar commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529458655
I second that. It's also how sqlcontext does. In general, we should avoid modifying Wayangcontext as this may have undesirable side effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] JorgeQuiane commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "JorgeQuiane (via GitHub)" <gi...@apache.org>.
JorgeQuiane commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1529646451
IMHO the monitoring should be part of the WayangContext as it is a core functionality and the SQLContext should extend from the WayangContext so that all (current and future) functionalities are inherited. As Kaustubh pointed out, we already have a Monitor abstract class and Monitor functionality in the current WayangContext. I also believe this should be the way to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-wayang] zkaoudi commented on pull request #311: Updates for monitoring the runtime metrics of underlying platform (Spark)
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #311:
URL: https://github.com/apache/incubator-wayang/pull/311#issuecomment-1538337860
Hi @AdeelAslamUnimore, thank you for the explanation. The build is passing now. Did you change your implementation? Does the constraint of using spark_2.12 still hold?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org