You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "ycbcholy@gmail.com" <yc...@gmail.com> on 2022/07/02 09:43:38 UTC
jobmanager 与taskmanager间的对象传递
各位大佬：<br/> 请教个问题，有个jobmanager产生的map对象，传递到各个taskmanager的function里，保证每个taskmanager内的对象有且仅有一个，然后会随时根据数据改变map对象的值，我想定时从taskmanager收集这个map对象整合后到jobmanager端，然后进行存储，有什么API或者通讯方式来支持吗？<br/><br/><br/><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=ycbcholy%40gmail.com&uid=ycbcholy%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aycbcholy%40gmail.com%22%5D" style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px 0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<table cellpadding="0" style='width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;'>
<tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
<tr>
<td width="38" style="padding:0; box-sizing: border-box; width: 38px;">
<img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" />
</td>
<td style='padding: 0 0 0 10px; color: #31353b;'>
<div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">ycbcholy@gmail.com</div>
</td>
</tr>
<tr width="100%" style="font-size: 14px !important; width: 100%;">
<td colspan='2' style="padding:10px 0 0 0; font-size:14px !important; width: 100%;">
<div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">邮箱：ycbcholy@gmail.com</div>
</td>
</tr>
</tbody>
</table>
</a>
</body>
</html>
Re: jobmanager 与taskmanager间的对象传递
Posted by Yunfeng Zhou <fl...@gmail.com>.
你好。
如果只是需要从各个subtask中收集一些信息，在JobManager中汇总的话，我觉得可以用累加器和计数器[1]。
如果需要双向通信的话，可以考虑一下FLIP-27[2]引入的OperatorCoordinator。如何通过通信来传递对象可以通过自定义算子或函数来实现。
在自定义算子中使用OperatorCoordinator可能还有一些不方便的地方，可以追踪一下相关ticket的进展[3]。
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/#accumulators--counters
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[3] https://issues.apache.org/jira/browse/FLINK-26029
On Sat, Jul 2, 2022 at 5:43 PM ycbcholy@gmail.com <yc...@gmail.com>
wrote:
> 各位大佬：<br/>
> 请教个问题，有个jobmanager产生的map对象，传递到各个taskmanager的function里，保证每个taskmanager内的对象有且仅有一个，然后会随时根据数据改变map对象的值，我想定时从taskmanager收集这个map对象整合后到jobmanager端，然后进行存储，有什么API或者通讯方式来支持吗？<br/><br/><br/><html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
> </head>
> <body>
> <a href="
> https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=ycbcholy%40gmail.com&uid=ycbcholy%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aycbcholy%40gmail.com%22%5D"
> style="display:block;background:#fff; max-width: 400px; _width:
> 400px;padding:15px 0 10px 0;text-decoration: none;
> outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;">
> <table cellpadding="0" style='width: 100%; max-width: 100%;
> table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size:
> 14px;line-height:1.3;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;'>
> <tbody style="font-family: 'PingFang SC', 'Hiragino Sans
> GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important;
> word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none
> !important;text-size-adjust:none !important;">
> <tr>
> <td width="38" style="padding:0; box-sizing:
> border-box; width: 38px;">
> <img width="38" height="38"
> style="vertical-align:middle; width: 38px; height: 38px;
> border-radius:50%;" src="
> https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png" />
> </td>
> <td style='padding: 0 0 0 10px; color:
> #31353b;'>
> <div style="font-size:
> 16px;font-weight:bold; width:100%; white-space: nowrap;
> overflow:hidden;text-overflow: ellipsis;">ycbcholy@gmail.com</div>
> </td>
> </tr>
> <tr width="100%" style="font-size: 14px
> !important; width: 100%;">
> <td colspan='2' style="padding:10px 0 0 0;
> font-size:14px !important; width: 100%;">
> <div style="width: 100%;font-size:
> 14px !important;word-wrap:break-word;word-break:break-all;">邮箱：
> ycbcholy@gmail.com</div>
> </td>
> </tr>
> </tbody>
> </table>
> </a>
> </body>
> </html>