Submiting jobs via UI/Rest API

[eSKa <es...@gmail.com>]

Hi guys,

We were trying to use UI's "Submit new job" functionality (and later REST
endpoints for that).
There were few problems we found:
1. When we ran job that had additional code done after env execution (or any
sink) the code was not executed. E.g. our job was calculating some data,
writing it to temp location in sink and when everything was successfully,
move files to proper location on HDFS. Running job using Java's
YARNClusterClient API worked fine.
2. We wanted to test job using "Show Plan" option but it seems that running
this option for job that did not have anything to run (e.g. calculated input
paths list was empty) results in killing the container on YARN. I didnt find
any suspicious logs in jobManager:


/2018-03-09 14:13:53,979 INFO  com.my_job.CustomFlinkJob                      
- All job done :)
2018-03-09 14:13:53,996 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
dashboard root cache directory
/tmp/flink-web-1fe30b99-9ad1-4531-b14b-143ea6c3d9ed
2018-03-09 14:13:54,004 INFO  org.apache.flink.runtime.blob.BlobServer                     
- Stopped BLOB server at 0.0.0.0:60727
2018-03-09 14:13:54,007 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
dashboard jar upload directory
/tmp/flink-web-8d7b68fc-1ef7-4869-91c1-5bebb370b529
/

We are using Flink 1.3.1 version, next week will play with 1.4.1.
Any chances for fixing that bugs in next versions?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Submiting jobs via UI/Rest API

[Nico Kruber <ni...@data-artisans.com>]

Thanks for reporting these issues,

1. This behaviour is actually intended since we do not spawn any thread
that is waiting for the job completion (which may or may not occur
eventually). Therefore, the web UI always submits jobs in detached mode
and you could not wait for job completion anyway. Any call after
env.execute() may thus not give you any more data than you already had
before. As a safety precaution, we stop the execution of the main()
method after env.execute().
If there was a possibility to wait for job completion, you would be able
to block the whole web UI with it.

2. This seems to be solved: I tried to submit this skeleton to Flink
1.5-SNAPSHOT and only got a failure message like this:
{"errors":["The main method caused an error."]}
The code I tried was
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.execute("Socket Window WordCount");

I also tried with a StreamTransformation only but got the same message.



Regards
Nico

On 09/03/18 14:33, eSKa wrote:
> Hi guys,
> 
> We were trying to use UI's "Submit new job" functionality (and later REST
> endpoints for that).
> There were few problems we found:
> 1. When we ran job that had additional code done after env execution (or any
> sink) the code was not executed. E.g. our job was calculating some data,
> writing it to temp location in sink and when everything was successfully,
> move files to proper location on HDFS. Running job using Java's
> YARNClusterClient API worked fine.
> 2. We wanted to test job using "Show Plan" option but it seems that running
> this option for job that did not have anything to run (e.g. calculated input
> paths list was empty) results in killing the container on YARN. I didnt find
> any suspicious logs in jobManager:
> 
> 
> /2018-03-09 14:13:53,979 INFO  com.my_job.CustomFlinkJob                      
> - All job done :)
> 2018-03-09 14:13:53,996 INFO 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
> dashboard root cache directory
> /tmp/flink-web-1fe30b99-9ad1-4531-b14b-143ea6c3d9ed
> 2018-03-09 14:13:54,004 INFO  org.apache.flink.runtime.blob.BlobServer                     
> - Stopped BLOB server at 0.0.0.0:60727
> 2018-03-09 14:13:54,007 INFO 
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
> dashboard jar upload directory
> /tmp/flink-web-8d7b68fc-1ef7-4869-91c1-5bebb370b529
> /
> 
> We are using Flink 1.3.1 version, next week will play with 1.4.1.
> Any chances for fixing that bugs in next versions?
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>