You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andy Hoang <an...@parcelperform.com> on 2019/06/24 05:04:03 UTC
Akka tcp address on AWS EMR between job/task managers
Hi guys,
I want to use some client lib that integrated with akka system
Flink aslo already created akka system job manager and those task manager can use akka by its akka.tcp addess
My architecture stucture currently is:
EMR (flink 1.8)
1 core node
1 master node
Flink job on Yarn
I have on task manager and one job manager:
On my task manager I got this log:
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,171 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:46267]
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,183 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:46267
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,284 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Actor system started at akka.tcp://flink-metrics@ip-10-0-5-203.ap-southeast-1.compute.internal:36753
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,731 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 .
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,755 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:33547/user/resourcemanager(00000000000000000000000000000000).
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:53,153 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:33547/user/resourcemanager under registration id 31415730214d58ecaf8396835ccce613.
On my job manager log:
container_1560142756526_0001_01_000001/jobmanager.out:2019-06-10 06:32:43,473 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager 00000000000000000000000000000000@akka.tcp://flink@ip-10-
0-5-203.ap-southeast-1.compute.internal:33547/user/jobmanager_0 for job 3111b27f8e4a873ad4301bbe2f7e7069.
container_1560142756526_0001_01_000001/jobmanager.out:2019-06-10 06:32:43,484 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager 00000000000000000000000000000000@akka.tcp://flink@ip-10-0
-5-203.ap-southeast-1.compute.internal:33547/user/jobmanager_0 for job 3111b27f8e4a873ad4301bbe2f7e7069.
I want to access to this actor system for my client lib (like akka http, redisscala…). Those need to declare akka.tcp address too.
implicit val akkaSystem = akka.actor.ActorSystem(“akka.tcp”)
How do I find those address dynamically so that I can use them in AWS emr?
Thanks