You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Samuel Brotherton (Jira)" <ji...@apache.org> on 2020/10/01 20:35:00 UTC
[jira] [Updated] (BEAM-11000) NullPointerException in Flink manager
when submitting job with environment_type=EXTERNAL
[ https://issues.apache.org/jira/browse/BEAM-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samuel Brotherton updated BEAM-11000:
-------------------------------------
Description:
Apologies if this is user error, as I am still learning Beam. I am attempting to run a python-based pipeline on the Flink runner, with Flink and an external beam image running inside docker-compose (see "Environment"). Whenever I submit a job, I get the following error. Am I doing something wrong? Thanks in advance.
* Edit: Jira mangled the traceback, you can see it here: https://gist.github.com/sbrother/b9f62580fbab84e2cfc925e0ea8ffc07
was:
Apologies if this is user error, as I am still learning Beam. I am attempting to run a python-based pipeline on the Flink runner, with Flink and an external beam image running inside docker-compose (see "Environment"). Whenever I submit a job, I get the following error. Am I doing something wrong? Thanks in advance.
{code:java}
jobmanager_1 | 2020-10-01 20:23:07,622 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 0b199c080493017fd532138056d27d57 (BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 0b199c080493017fd532138056d27d57 (BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 0b199c080493017fd532138056d27d57 (BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,625 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 .jobmanager_1 | 2020-10-01 20:23:07,626 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job BeamApp-root-1001202304-4e0d2f62 (0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,627 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy NoRestartBackoffTimeStrategy for BeamApp-root-1001202304-4e0d2f62 (0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,630 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job BeamApp-root-1001202304-4e0d2f62 (0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,654 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 24 ms.jobmanager_1 | 2020-10-01 20:23:07,655 ERROR org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Failed to submit job 0b199c080493017fd532138056d27d57.jobmanager_1 | org.apache.flink.runtime.client.JobExecutionException: Could not instantiate JobManager.jobmanager_1 | at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:398) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_265]jobmanager_1 | at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | Caused by: java.lang.NullPointerExceptionjobmanager_1 | at java.util.Collections$UnmodifiableCollection.<init>(Collections.java:1028) ~[?:1.8.0_265]jobmanager_1 | at java.util.Collections$UnmodifiableList.<init>(Collections.java:1304) ~[?:1.8.0_265]jobmanager_1 | at java.util.Collections.unmodifiableList(Collections.java:1289) ~[?:1.8.0_265]jobmanager_1 | at org.apache.flink.runtime.jobgraph.JobVertex.getOperatorCoordinators(JobVertex.java:352) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:232) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:814) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:228) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:269) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:242) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:229) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:119) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:284) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:272) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:140) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:388) ~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | ... 7 morejobmanager_1 | 2020-10-01 20:23:07,656 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler [] - Unhandled exception.jobmanager_1 | org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
{code}
> NullPointerException in Flink manager when submitting job with environment_type=EXTERNAL
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-11000
> URL: https://issues.apache.org/jira/browse/BEAM-11000
> Project: Beam
> Issue Type: Bug
> Components: beam-community
> Affects Versions: 2.24.0
> Environment: jobmanager:
> image: flink:1.11.2-scala_2.11
> ports:
> - "8081:8081"
> command: jobmanager
> environment:
> - |
> FLINK_PROPERTIES=
> jobmanager.rpc.address: jobmanager
> taskmanager:
> image: flink:1.11.2-scala_2.11
> depends_on:
> - jobmanager
> command: taskmanager
> environment:
> - |
> FLINK_PROPERTIES=
> jobmanager.rpc.address: jobmanager
> taskmanager.numberOfTaskSlots: 2
> beamworker:
> image: apache/beam_python3.8_sdk
> ports:
> - "50000:50000"
> command: --worker_pool
> Reporter: Samuel Brotherton
> Assignee: Aizhamal Nurmamat kyzy
> Priority: P2
>
> Apologies if this is user error, as I am still learning Beam. I am attempting to run a python-based pipeline on the Flink runner, with Flink and an external beam image running inside docker-compose (see "Environment"). Whenever I submit a job, I get the following error. Am I doing something wrong? Thanks in advance.
> * Edit: Jira mangled the traceback, you can see it here: https://gist.github.com/sbrother/b9f62580fbab84e2cfc925e0ea8ffc07
--
This message was sent by Atlassian Jira
(v8.3.4#803005)