You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Papadopoulos, Konstantinos" <Ko...@IRIworldwide.com> on 2019/03/14 12:16:16 UTC
ProgramInvocationException when trying to submit a job by running a
jar using Monitoring REST API
Hi all,
As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files.
When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error response with the following body:
{ "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] }
On the other hand, when I am trying the same with the 'Word Count' batch example, execution succeeds.
It must be pointed out that I tried to execute the respective POST request both via 'curl' and Postman, but both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
RE: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by "Papadopoulos, Konstantinos" <Ko...@IRIworldwide.com>.
Yes, we are submitting more than one job and we choose which one is going to be executed depending on the first program argument (i.e., ‘job’ argument).
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 12:53 μμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
In your jar, are you submitting multiple jobs in parallel?
On 15.03.2019 10:05, Papadopoulos, Konstantinos wrote:
We had some progress since the job seems to be submitted and its execution has been started, but, now, I am getting a ProgramAbortException as follows:
05:01:01.788 [ERROR] SpringApplication – Application run failed
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException: null
at org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:54) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at com.iri.aa.etl.lmx.entity.service.AbstractLmxService.getCampaigns(AbstractLmxService.java:105) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.entity.service.MediaSpentServiceImpl.transform(MediaSpentServiceImpl.java:25) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:58) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.AbstractLmxJob.abstractExecute(AbstractLmxJob.java:27) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:38) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.executeDry(MediaSpentJob.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.job.JobExecutor.lambda$executeDryRunners$6(JobExecutor.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_201]
at com.iri.aa.etl.job.JobExecutor.executeDryRunners(JobExecutor.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:35) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_201]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_201]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 10:56 πμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please separate your program arguments by a space instead of a comma and try again.
On 15.03.2019 09:34, Papadopoulos, Konstantinos wrote:
Hi Chesnay,
Sorry for the misunderstanding. I get the following exception:
2019-03-15 04:31:26,826 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception occurred in REST handler.
org.apache.flink.runtime.rest.handler.RestHandlerException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
... 4 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
... 5 more
Caused by: java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 10 more
Caused by: java.lang.IllegalArgumentException: Argument 'initialScopeId' is missing.
at com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
... 20 more
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please provide the logged exception, I cannot help you otherwise.
On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
It seems that Flink cluster does not retrieve program arguments correctly. For reference, I sent the following request:
Method Type: POST
URL: http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run<https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cf5437d3c54884da2c66308d6a9345720%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882439599635866&sdata=Ax%2BVHmCcnsJ7GFbpgN09Tp7WgDGCBYDmDtttXpPmt%2BY%3D&reserved=0>
Body: {"programArgs" : "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
Content-Type: application/json
From: Chesnay Schepler <ch...@apache.org>
Sent: Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
Hi all,
As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files.
When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error response with the following body:
{ "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] }
On the other hand, when I am trying the same with the ‘Word Count’ batch example, execution succeeds.
It must be pointed out that I tried to execute the respective POST request both via ‘curl’ and Postman, but both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
Re: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by Chesnay Schepler <ch...@apache.org>.
In your jar, are you submitting multiple jobs in parallel?
On 15.03.2019 10:05, Papadopoulos, Konstantinos wrote:
>
> We had some progress since the job seems to be submitted and its
> execution has been started, but, now, I am getting a
> ProgramAbortException as follows:
>
> 05:01:01.788 [ERROR] SpringApplication – Application run failed
>
> org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException:
> null
>
> at
> org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:54)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> com.iri.aa.etl.lmx.entity.service.AbstractLmxService.getCampaigns(AbstractLmxService.java:105)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> com.iri.aa.etl.lmx.entity.service.MediaSpentServiceImpl.transform(MediaSpentServiceImpl.java:25)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:58)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> com.iri.aa.etl.lmx.AbstractLmxJob.abstractExecute(AbstractLmxJob.java:27)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:38)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at com.iri.aa.etl.lmx.MediaSpentJob.executeDry(MediaSpentJob.java:44)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> com.iri.aa.etl.job.JobExecutor.lambda$executeDryRunners$6(JobExecutor.java:44)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_201]
>
> at
> com.iri.aa.etl.job.JobExecutor.executeDryRunners(JobExecutor.java:44)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:35)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
> ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_201]
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> ~[?:1.8.0_201]
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> ~[?:1.8.0_201]
>
> at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_201]
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
> ~[flink-dist_2.12-1.7.1.jar:1.7.1]
>
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
> Source) [?:1.8.0_201]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [?:1.8.0_201]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [?:1.8.0_201]
>
> at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
>
> *From:*Chesnay Schepler <ch...@apache.org>
> *Sent:* Παρασκευή, 15 Μαρτίου 2019 10:56 πμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>; user@flink.apache.org
> *Subject:* Re: ProgramInvocationException when trying to submit a job
> by running a jar using Monitoring REST API
>
> Please separate your program arguments by a space instead of a comma
> and try again.
>
> On 15.03.2019 09:34, Papadopoulos, Konstantinos wrote:
>
> Hi Chesnay,
>
> Sorry for the misunderstanding. I get the following exception:
>
> 2019-03-15 04:31:26,826 ERROR
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler -
> Exception occurred in REST handler.
>
> org.apache.flink.runtime.rest.handler.RestHandlerException:
> org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error.
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
>
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(Unknown
> Source)
>
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown
> Source)
>
> at
> java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
>
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>
> at java.lang.Thread.run(Unknown Source)
>
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error.
>
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
>
> ... 4 more
>
> Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error.
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>
> at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
>
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
>
> ... 5 more
>
> Caused by: java.lang.IllegalStateException: Failed to execute
> ApplicationRunner
>
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
>
> at
> org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
>
> at
> com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>
> at java.lang.reflect.Method.invoke(Unknown Source)
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>
> ... 10 more
>
> Caused by: java.lang.IllegalArgumentException: Argument
> 'initialScopeId' is missing.
>
> at
> com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
>
> at
> com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
>
> at
> com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
>
> at
> com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
>
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
>
> ... 20 more
>
> *From:*Chesnay Schepler <ch...@apache.org>
> <ma...@apache.org>
> *Sent:* Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>
> <ma...@IRIworldwide.com>;
> user@flink.apache.org <ma...@flink.apache.org>
> *Subject:* Re: ProgramInvocationException when trying to submit a
> job by running a jar using Monitoring REST API
>
> Please provide the logged exception, I cannot help you otherwise.
>
> On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
>
> It seems that Flink cluster does not retrieve program
> arguments correctly. For reference, I sent the following request:
>
> Method Type: POST
>
> URL:
> http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cdf15b45eb7814617865208d6a9240306%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882369474729054&sdata=2i%2BnBpElTlCgf8SAyc13xtCY8%2FMk%2B2TNA3sLMzSJ5O4%3D&reserved=0>
>
> Body: {"programArgs" :
> "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
>
> Content-Type: application/json
>
> *From:*Chesnay Schepler <ch...@apache.org>
> <ma...@apache.org>
> *Sent:* Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>
> <ma...@IRIworldwide.com>;
> user@flink.apache.org <ma...@flink.apache.org>
> *Subject:* Re: ProgramInvocationException when trying to
> submit a job by running a jar using Monitoring REST API
>
> Please enable debug logging, re-submit the job, check the
> JobManager logs and report back the logged exception.
>
> On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
>
> Hi all,
>
> As part of our projects, I experiment with Flink
> Monitoring REST API and, especially, its capabilities of
> uploading and running jar files.
>
> When I am trying to submit one of our jobs by running a
> jar previously uploaded via '/jars/upload', I am getting
> an 500 Internal Server Error response with the following body:
>
> { "errors": [
> "org.apache.flink.client.program.ProgramInvocationException:
> The main method caused an error." ] }
>
> On the other hand, when I am trying the same with the
> ‘Word Count’ batch example, execution succeeds.
>
> It must be pointed out that I tried to execute the
> respective POST request both via ‘curl’ and Postman, but
> both failed with the same error.
>
> Do anyone have idea why this may happen?
>
> Thanks in advance,
>
> Konstantinos
>
RE: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by "Papadopoulos, Konstantinos" <Ko...@IRIworldwide.com>.
We had some progress since the job seems to be submitted and its execution has been started, but, now, I am getting a ProgramAbortException as follows:
05:01:01.788 [ERROR] SpringApplication - Application run failed
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException: null
at org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:54) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at com.iri.aa.etl.lmx.entity.service.AbstractLmxService.getCampaigns(AbstractLmxService.java:105) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.entity.service.MediaSpentServiceImpl.transform(MediaSpentServiceImpl.java:25) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:58) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.AbstractLmxJob.abstractExecute(AbstractLmxJob.java:27) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:38) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.executeDry(MediaSpentJob.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.job.JobExecutor.lambda$executeDryRunners$6(JobExecutor.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_201]
at com.iri.aa.etl.job.JobExecutor.executeDryRunners(JobExecutor.java:44) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:35) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21) ~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_201]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151) ~[flink-dist_2.12-1.7.1.jar:1.7.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_201]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 10:56 πμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please separate your program arguments by a space instead of a comma and try again.
On 15.03.2019 09:34, Papadopoulos, Konstantinos wrote:
Hi Chesnay,
Sorry for the misunderstanding. I get the following exception:
2019-03-15 04:31:26,826 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception occurred in REST handler.
org.apache.flink.runtime.rest.handler.RestHandlerException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
... 4 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
... 5 more
Caused by: java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 10 more
Caused by: java.lang.IllegalArgumentException: Argument 'initialScopeId' is missing.
at com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
... 20 more
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please provide the logged exception, I cannot help you otherwise.
On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
It seems that Flink cluster does not retrieve program arguments correctly. For reference, I sent the following request:
Method Type: POST
URL: http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run<https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cdf15b45eb7814617865208d6a9240306%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882369474729054&sdata=2i%2BnBpElTlCgf8SAyc13xtCY8%2FMk%2B2TNA3sLMzSJ5O4%3D&reserved=0>
Body: {"programArgs" : "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
Content-Type: application/json
From: Chesnay Schepler <ch...@apache.org>
Sent: Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
Hi all,
As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files.
When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error response with the following body:
{ "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] }
On the other hand, when I am trying the same with the ‘Word Count’ batch example, execution succeeds.
It must be pointed out that I tried to execute the respective POST request both via ‘curl’ and Postman, but both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
Re: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by Chesnay Schepler <ch...@apache.org>.
Please separate your program arguments by a space instead of a comma and
try again.
On 15.03.2019 09:34, Papadopoulos, Konstantinos wrote:
>
> Hi Chesnay,
>
> Sorry for the misunderstanding. I get the following exception:
>
> 2019-03-15 04:31:26,826 ERROR
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception
> occurred in REST handler.
>
> org.apache.flink.runtime.rest.handler.RestHandlerException:
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
>
> at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
>
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown
> Source)
>
> at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
>
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>
> at java.lang.Thread.run(Unknown Source)
>
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
>
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
>
> ... 4 more
>
> Caused by: org.apache.flink.client.program.ProgramInvocationException:
> The main method caused an error.
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>
> at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
>
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
>
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
>
> ... 5 more
>
> Caused by: java.lang.IllegalStateException: Failed to execute
> ApplicationRunner
>
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
>
> at
> org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
>
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
>
> at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>
> at java.lang.reflect.Method.invoke(Unknown Source)
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>
> ... 10 more
>
> Caused by: java.lang.IllegalArgumentException: Argument
> 'initialScopeId' is missing.
>
> at
> com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
>
> at
> com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
>
> at
> com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
>
> at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
>
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
>
> ... 20 more
>
> *From:*Chesnay Schepler <ch...@apache.org>
> *Sent:* Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>; user@flink.apache.org
> *Subject:* Re: ProgramInvocationException when trying to submit a job
> by running a jar using Monitoring REST API
>
> Please provide the logged exception, I cannot help you otherwise.
>
> On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
>
> It seems that Flink cluster does not retrieve program arguments
> correctly. For reference, I sent the following request:
>
> Method Type: POST
>
> URL:
> http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cc66d382ad8cb47a0547608d6a91f0367%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882348004604648&sdata=eQF8PpHADRppeT7y9pkPdfbG0sAaKkgpzTa%2BTaBb9mQ%3D&reserved=0>
>
> Body: {"programArgs" :
> "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
>
> Content-Type: application/json
>
> *From:*Chesnay Schepler <ch...@apache.org>
> <ma...@apache.org>
> *Sent:* Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>
> <ma...@IRIworldwide.com>;
> user@flink.apache.org <ma...@flink.apache.org>
> *Subject:* Re: ProgramInvocationException when trying to submit a
> job by running a jar using Monitoring REST API
>
> Please enable debug logging, re-submit the job, check the
> JobManager logs and report back the logged exception.
>
> On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
>
> Hi all,
>
> As part of our projects, I experiment with Flink Monitoring
> REST API and, especially, its capabilities of uploading and
> running jar files.
>
> When I am trying to submit one of our jobs by running a jar
> previously uploaded via '/jars/upload', I am getting an 500
> Internal Server Error response with the following body:
>
> { "errors": [
> "org.apache.flink.client.program.ProgramInvocationException:
> The main method caused an error." ] }
>
> On the other hand, when I am trying the same with the ‘Word
> Count’ batch example, execution succeeds.
>
> It must be pointed out that I tried to execute the respective
> POST request both via ‘curl’ and Postman, but both failed with
> the same error.
>
> Do anyone have idea why this may happen?
>
> Thanks in advance,
>
> Konstantinos
>
RE: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by "Papadopoulos, Konstantinos" <Ko...@IRIworldwide.com>.
Hi Chesnay,
Sorry for the misunderstanding. I get the following exception:
2019-03-15 04:31:26,826 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception occurred in REST handler.
org.apache.flink.runtime.rest.handler.RestHandlerException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
... 4 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
... 5 more
Caused by: java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 10 more
Caused by: java.lang.IllegalArgumentException: Argument 'initialScopeId' is missing.
at com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
at com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
... 20 more
From: Chesnay Schepler <ch...@apache.org>
Sent: Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please provide the logged exception, I cannot help you otherwise.
On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
It seems that Flink cluster does not retrieve program arguments correctly. For reference, I sent the following request:
Method Type: POST
URL: http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run<https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cc66d382ad8cb47a0547608d6a91f0367%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882348004604648&sdata=eQF8PpHADRppeT7y9pkPdfbG0sAaKkgpzTa%2BTaBb9mQ%3D&reserved=0>
Body: {"programArgs" : "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
Content-Type: application/json
From: Chesnay Schepler <ch...@apache.org>
Sent: Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
Hi all,
As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files.
When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error response with the following body:
{ "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] }
On the other hand, when I am trying the same with the ‘Word Count’ batch example, execution succeeds.
It must be pointed out that I tried to execute the respective POST request both via ‘curl’ and Postman, but both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
Re: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by Chesnay Schepler <ch...@apache.org>.
Please provide the logged exception, I cannot help you otherwise.
On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
>
> It seems that Flink cluster does not retrieve program arguments
> correctly. For reference, I sent the following request:
>
> Method Type: POST
>
> URL:
> http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run
>
> Body: {"programArgs" :
> "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
>
> Content-Type: application/json
>
> *From:*Chesnay Schepler <ch...@apache.org>
> *Sent:* Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
> *To:* Papadopoulos, Konstantinos
> <Ko...@IRIworldwide.com>; user@flink.apache.org
> *Subject:* Re: ProgramInvocationException when trying to submit a job
> by running a jar using Monitoring REST API
>
> Please enable debug logging, re-submit the job, check the JobManager
> logs and report back the logged exception.
>
> On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
>
> Hi all,
>
> As part of our projects, I experiment with Flink Monitoring REST
> API and, especially, its capabilities of uploading and running jar
> files.
>
> When I am trying to submit one of our jobs by running a jar
> previously uploaded via '/jars/upload', I am getting an 500
> Internal Server Error response with the following body:
>
> { "errors": [
> "org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error." ] }
>
> On the other hand, when I am trying the same with the ‘Word Count’
> batch example, execution succeeds.
>
> It must be pointed out that I tried to execute the respective POST
> request both via ‘curl’ and Postman, but both failed with the same
> error.
>
> Do anyone have idea why this may happen?
>
> Thanks in advance,
>
> Konstantinos
>
RE: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by "Papadopoulos, Konstantinos" <Ko...@IRIworldwide.com>.
It seems that Flink cluster does not retrieve program arguments correctly. For reference, I sent the following request:
Method Type: POST
URL: http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run
Body: {"programArgs" : "--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
Content-Type: application/json
From: Chesnay Schepler <ch...@apache.org>
Sent: Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
To: Papadopoulos, Konstantinos <Ko...@IRIworldwide.com>; user@flink.apache.org
Subject: Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API
Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
Hi all,
As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files.
When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error response with the following body:
{ "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] }
On the other hand, when I am trying the same with the 'Word Count' batch example, execution succeeds.
It must be pointed out that I tried to execute the respective POST request both via 'curl' and Postman, but both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
Re: ProgramInvocationException when trying to submit a job by running
a jar using Monitoring REST API
Posted by Chesnay Schepler <ch...@apache.org>.
Please enable debug logging, re-submit the job, check the JobManager
logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
>
> Hi all,
>
> As part of our projects, I experiment with Flink Monitoring REST API
> and, especially, its capabilities of uploading and running jar files.
>
> When I am trying to submit one of our jobs by running a jar previously
> uploaded via '/jars/upload', I am getting an 500 Internal Server Error
> response with the following body:
>
> { "errors": [
> "org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error." ] }
>
> On the other hand, when I am trying the same with the ‘Word Count’
> batch example, execution succeeds.
>
> It must be pointed out that I tried to execute the respective POST
> request both via ‘curl’ and Postman, but both failed with the same error.
>
> Do anyone have idea why this may happen?
>
> Thanks in advance,
>
> Konstantinos
>