You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by nragon <nu...@wedotechnologies.com> on 2017/07/20 15:51:39 UTC

Detached execution API

It would be nice to let users deploy detached jobs through api.
For instance>

*StreamExecutionEnviroment*

  public JobExecutionResult execute() throws Exception {
    return execute(DEFAULT_JOB_NAME, false);
  }
Which keep backward compatibility

  public abstract JobExecutionResult execute(String jobName, boolean
detached) throws Exception;

*RemoteStreamEnvironment*

@Override
  public JobExecutionResult execute(String jobName, boolean detached) throws
ProgramInvocationException {
    StreamGraph streamGraph = getStreamGraph();
    streamGraph.setJobName(jobName);
    transformations.clear();
    return executeRemotely(streamGraph, jarFiles, detached);
  }

---------------------

protected JobExecutionResult executeRemotely(StreamGraph streamGraph,
List<URL> jarFiles, boolean detached) throws ProgramInvocationException {
.....
ClusterClient client;
    try {
      client = new StandaloneClusterClient(configuration);
      client.setDetached(detached);
     
client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
    }
....
}

Just an idea. I'm using the previous approach and it works fine.

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Detached-execution-API-tp14366.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Detached execution API

Posted by nragon <nu...@wedotechnologies.com>.
Yes, something like that.
Although, this would be an MVP for this purpose which has minimal impacts on
next releases. Moreover, I think clients are too specific to be included as
API because each user will have their own way to implement a stop, start, or
whatever (of course a base client is welcome :)). For instance, we have
implemented one, using REST API + async(detached) job.

Also, one thing I forgot in the previous example is that abstract execute
should return a JobSubmissionResult instead of JobExecutionResult.

But of course, just my point of view.




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Detached-execution-API-tp14366p14369.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Detached execution API

Posted by Aljoscha Krettek <al...@apache.org>.
Hi,

I think you might find those two issues interesting:

- https://issues.apache.org/jira/browse/FLINK-2313: Change Streaming Driver Execution Model
- https://issues.apache.org/jira/browse/FLINK-4272: Create a JobClient for job control and monitoring

Best,
Aljoscha

> On 20. Jul 2017, at 17:51, nragon <nu...@wedotechnologies.com> wrote:
> 
> It would be nice to let users deploy detached jobs through api.
> For instance>
> 
> *StreamExecutionEnviroment*
> 
>  public JobExecutionResult execute() throws Exception {
>    return execute(DEFAULT_JOB_NAME, false);
>  }
> Which keep backward compatibility
> 
>  public abstract JobExecutionResult execute(String jobName, boolean
> detached) throws Exception;
> 
> *RemoteStreamEnvironment*
> 
> @Override
>  public JobExecutionResult execute(String jobName, boolean detached) throws
> ProgramInvocationException {
>    StreamGraph streamGraph = getStreamGraph();
>    streamGraph.setJobName(jobName);
>    transformations.clear();
>    return executeRemotely(streamGraph, jarFiles, detached);
>  }
> 
> ---------------------
> 
> protected JobExecutionResult executeRemotely(StreamGraph streamGraph,
> List<URL> jarFiles, boolean detached) throws ProgramInvocationException {
> .....
> ClusterClient client;
>    try {
>      client = new StandaloneClusterClient(configuration);
>      client.setDetached(detached);
> 
> client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
>    }
> ....
> }
> 
> Just an idea. I'm using the previous approach and it works fine.
> 
> Thanks
> 
> 
> 
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Detached-execution-API-tp14366.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.