You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by sachingoel0101 <gi...@git.apache.org> on 2015/10/02 15:33:27 UTC

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

GitHub user sachingoel0101 opened a pull request:

    https://github.com/apache/flink/pull/1214

    [FLINK-2797][cli] Add support for running jobs in detached mode from CLI

    Adds an option `-d` to run jobs in detached mode.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sachingoel0101/flink detached_cli

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1214.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1214
    
----
commit f75467cddbc182fdc42061c9ca602e20690a4ef5
Author: Sachin Goel <sa...@gmail.com>
Date:   2015-10-02T12:41:09Z

    [FLINK-2797][cli] Add support for running jobs in detached mode from CLI

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146908658
  
    The way we do this with the Optimizer plan is a bit pragmatic. We wick the first runnable part and use that. We could do the same here. Detached mode would submit the first part of the program. If the program has more than one submission to the jobManager, the other ones would not get executed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155043211
  
    Hi @mxm , I'm not sure about directing sysout and syserr to null. Detached mode just means flink jobs are being run in detached mode; the rest of the program should go whatever way the user requested.
    
    As for refactoring into a separate environment, I agree. I've already done that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146875588
  
    I think it would be fine to not have HA for interactive programs. After all, Flink need no client/master interactivity for streaming and iterations, which is where the HA is most important.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-152934449
  
    @mxm, since you already delegated job graph creation to the Client in one of your commits, I've removed that from my commits. Rebasing required squashing all commits first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772606
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -172,5 +184,74 @@ public ExecutionEnvironment createExecutionEnvironment() {
     			}
     			return env;
     		}
    +
    +		public static void setPlan(JobWithJars plan) {
    +			if (_plan == null) {
    +				_plan = plan;
    +			} else {
    +				throw new DetachedProgramException("The Job was submitted in detached mode. Only " +
    +						"one execution is allowed. Please make sure your program doesn't call " +
    +						"execute twice and/or doesn't call an eager execution function[collect, " +
    +						"print, printToErr, count].\n");
    +			}
    +		}
    +
    +		public static JobWithJars getPlan() {
    +			return _plan;
    +		}
    +
    +		public static JobGraph getGraph() {
    +			return _graph;
    +		}
    +
    +		public static void setGraph(JobGraph graph) {
    +			if (_graph == null) {
    +				_graph = graph;
    +			} else {
    +				throw new DetachedProgramException("The Job was submitted in detached mode. Only " +
    +						"one call to execute is allowed.");
    +			}
    +		}
    +	}
    +
    +	public static final class DetachedJobExecutionResult extends JobExecutionResult {
    +
    +		public static final DetachedJobExecutionResult INSTANCE = new DetachedJobExecutionResult();
    +
    +		public DetachedJobExecutionResult() {
    --- End diff --
    
    Constructor should be private to be a singleton.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155381011
  
    Looks like your recent changes broke the YARN tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098234
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
    @@ -271,18 +272,21 @@ public JobSubmissionResult runDetached(PackagedProgram prog, int parallelism)
     		}
     		else if (prog.isUsingInteractiveMode()) {
     			LOG.info("Starting program in interactive mode");
    -			ContextEnvironment.setAsContext(this, prog.getAllLibraries(), prog.getClasspaths(),
    -				prog.getUserCodeClassLoader(), parallelism, false);
    -
    +			ContextEnvironment.ContextEnvironmentFactory factory = ContextEnvironment.setAsContext(this,
    +					prog.getAllLibraries(), prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
     			// invoke here
     			try {
     				prog.invokeInteractiveModeForExecution();
    +				ContextEnvironment ctx = factory.getLastEnvironment();
    +				if (ctx == null) {
    +					throw new InvalidProgramException("No execution environment was created.");
    --- End diff --
    
    Do we really want to fail here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772695
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    These are no proper tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146899718
  
    OK....now I'm confused. This PR was all about disallowing "multiple round trips" and that's what Sachin tried to solve. Failing all interactive programs detached is a quick solution because solving the "multiple round trips" problem is not trivial. 
    
    How do you want to detect another execution after the current one? Essentially we have to forbid all user code that follows execute(), collect(), print(), count().


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146911998
  
    With proper documentation, failing on the second execute could be an acceptable workaround but it's bound to cause trouble. In addition we might want to warn users on the command line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146897768
  
    @sachingoel0101 you are right, interactive mode is actually pretty broad (as defined in the Client).
    
    What I have been taking about in prior comments was programs that do multiple round trips between JobManager and client (because of using `collect()` `count()` or so).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153094985
  
    @mxm , I've made some modifications and removed `ClassLoader` code from the `StreamContextEnvironment`. 
    Further, made the code branch to set the `lastEnv` more explicit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145513084
  
    That sounds good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155373495
  
    >This is a full fledged execution of the use code, and should show any user generated out and err messages in my opinion.
    
    You're probably right. It may just seem odd to the user that code gets executed even though the job won't be submitted. We might fix that in the future if users don't like it.
    
    >Of course, anything Flink will print, say, log messages, will already be disabled as there is no actor to receive the messages.
    
    There are still jobmanager/taskmanager log files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151402904
  
    @mxm I have made some changes and now the *current* plan resides in the `ContextEnvironment`. This required some changes in `StreamContextEnvironment`, which now wraps the `ContextEnvironment` which was initially created to check the existence of context environment.
    Furthermore, `ContextEnvironmentFactory` needs to store the last context environment created, which can be used then to access the plan, so it can be executed after the call to interactive program's `main` finishes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42774659
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
    +			out.runDetached(prg, 1);
    +		} catch (ProgramInvocationException e) {
    +			assertEquals("The Job was submitted in detached mode. Only one execution is allowed. " +
    --- End diff --
    
    Will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43753100
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -81,13 +60,13 @@ public JobExecutionResult execute(String jobName) throws Exception {
     
     		transformations.clear();
     
    -		// execute the programs
    -		if (wait) {
    -			return client.runBlocking(streamGraph, jars, classpaths, userCodeClassLoader);
    +		if (ctx.isWait()) {
    +			return ctx.getClient().runBlocking(streamGraph, ctx.getJars(), ctx.getClasspaths(),
    +					ctx.getUserCodeClassLoader());
     		} else {
    -			JobSubmissionResult result = client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
    +			ctx.setCurrentPlan(streamGraph);
     			LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    -			return JobExecutionResult.fromJobSubmissionResult(result);
    +			return ContextEnvironment.DetachedJobExecutionResult.INSTANCE;
    --- End diff --
    
    Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155385085
  
    >>> You're probably right. It may just seem odd to the user that code gets executed even though the job won't be submitted. We might fix that in the future if users don't like it.
    
    I think most people will be aware that Flink executes lazily, and just because the program goes right till the end printing whatever it does, it doesn't guarantee a true submission. But there is no point debating on this. We can adapt to whatever people prefer. This'll likely go through a major change before 1.0 to support truly detached mode anyways.
    
    >>>Looks like your recent changes broke the YARN tests?
    Yes. It's also kind of hard to debug them. I can't seem to run them on my VM, and travis build takes a full one and a half hour to produce anything useful, and even then, I can't access the complete logs. It probably requires AWS keys or something. I'm not sure.
    Do you know of any change I can make to travis conf so all modules are built but only Yarn tests get executed? 
    
    I have one more concern:
    There are four types of Yarn runs we can perform:
    1. Yarn session in detached mode: Equivalent to `bin/start-cluster.sh`
    2. Yarn session: Doesn't have a counterpart in standalone mode
    3. Per job yarn in detached mode: No counterpart in standalone mode, even after we add this PR.
    4. Per job yarn: No counterpart in standalone mode.
    
    Now, when `bin/flink -d` is run, user will expect there to be a single semantics; that a cluster will be brought up, job will be submitted and the cluster will be terminated. However, there doesn't seem to be any way of achieving this in standalone mode. Will this create any confusion? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44051406
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -81,13 +60,15 @@ public JobExecutionResult execute(String jobName) throws Exception {
     
     		transformations.clear();
     
    -		// execute the programs
    -		if (wait) {
    -			return client.runBlocking(streamGraph, jars, classpaths, userCodeClassLoader);
    +		if (ctx.isWait()) {
    +			return ctx.getClient().runBlocking(streamGraph, ctx.getJars(), ctx.getClasspaths(),
    +					ctx.getUserCodeClassLoader());
     		} else {
    -			JobSubmissionResult result = client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
    +			ctx.setCurrentPlan(streamGraph);
    --- End diff --
    
    @mxm , fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43600897
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -36,28 +31,16 @@
     
     	private static final Logger LOG = LoggerFactory.getLogger(StreamContextEnvironment.class);
     
    -	private final List<URL> jars;
    -
    -	private final List<URL> classpaths;
    -	
    -	private final Client client;
    +	private final ContextEnvironment ctx;
     
     	private final ClassLoader userCodeClassLoader;
    -	
    -	private final boolean wait;
     
    -	protected StreamContextEnvironment(Client client, List<URL> jars, List<URL> classpaths, int parallelism,
    -			boolean wait) {
    -		this.client = client;
    -		this.jars = jars;
    -		this.classpaths = classpaths;
    -		this.wait = wait;
    -		
    -		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(jars, classpaths,
    -				getClass().getClassLoader());
    +	protected StreamContextEnvironment(ContextEnvironment ctx) {
    +		this.ctx = ctx;
    +		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(), getClass().getClassLoader());
    --- End diff --
    
    I'm not very familiar with classloaders. Is it necessary that the classloader for Streaming jobs be built here and not via the usual `Program#getUserCodeClassLoader` in `Client` class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43610508
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -165,12 +181,65 @@ public ContextEnvironmentFactory(Client client, List<URL> jarFilesToAttach,
     		
     		@Override
     		public ExecutionEnvironment createExecutionEnvironment() {
    -			ContextEnvironment env = new ContextEnvironment(client, jarFilesToAttach, classpathsToAttach,
    -					userCodeClassLoader, wait);
    -			if (defaultParallelism > 0) {
    -				env.setParallelism(defaultParallelism);
    +			if (wait || lastEnv == null) {
    --- End diff --
    
    Could you at least add a comment here? This condition is not very intuitive. Why do you even set `lastEnv` here for blocking execution mode?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44641729
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java ---
    @@ -0,0 +1,116 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.client.program;
    +
    +import org.apache.flink.api.common.InvalidProgramException;
    +import org.apache.flink.api.common.JobExecutionResult;
    +import org.apache.flink.api.common.JobID;
    +import org.apache.flink.api.common.JobSubmissionResult;
    +import org.apache.flink.api.common.Plan;
    +import org.apache.flink.optimizer.plan.FlinkPlan;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.net.URL;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Execution Environment for remote execution with the Client in detached mode.
    + */
    +public class DetachedEnvironment extends ContextEnvironment {
    +
    +	/** Keeps track of the program plan for the Client to access. */
    +	private FlinkPlan detachedPlan;
    +
    +	private static final Logger LOG = LoggerFactory.getLogger(DetachedEnvironment.class);
    +
    +	public DetachedEnvironment(Client remoteConnection, List<URL> jarFiles, List<URL> classpaths, ClassLoader userCodeClassLoader) {
    +		super(remoteConnection, jarFiles, classpaths, userCodeClassLoader);
    +	}
    +
    +	@Override
    +	public JobExecutionResult execute(String jobName) throws Exception {
    +		Plan p = createProgramPlan(jobName);
    +		setDetachedPlan(Client.getOptimizedPlan(client.compiler, p, getParallelism()));
    +		LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    +		this.lastJobExecutionResult = DetachedJobExecutionResult.INSTANCE;
    +		return this.lastJobExecutionResult;
    +	}
    +
    +	public void setDetachedPlan(FlinkPlan plan) {
    +		if (detachedPlan == null) {
    +			detachedPlan = plan;
    +		} else {
    +			throw new InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
    +					DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
    +		}
    +	}
    +
    +	/**
    +	 * Finishes this Context Environment's execution by explicitly running the plan constructed.
    +	 */
    +	public JobSubmissionResult finalizeExecute() throws ProgramInvocationException {
    --- End diff --
    
    Wouldn't it make more sense to overwrite `execute()` here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772667
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
    +			out.runDetached(prg, 1);
    +		} catch (ProgramInvocationException e) {
    +			assertEquals("The Job was submitted in detached mode. Only one execution is allowed. " +
    --- End diff --
    
    Can we make this message available somewhere such that it doesn't have to be copied over and over?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44044799
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -81,13 +60,15 @@ public JobExecutionResult execute(String jobName) throws Exception {
     
     		transformations.clear();
     
    -		// execute the programs
    -		if (wait) {
    -			return client.runBlocking(streamGraph, jars, classpaths, userCodeClassLoader);
    +		if (ctx.isWait()) {
    +			return ctx.getClient().runBlocking(streamGraph, ctx.getJars(), ctx.getClasspaths(),
    +					ctx.getUserCodeClassLoader());
     		} else {
    -			JobSubmissionResult result = client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
    +			ctx.setCurrentPlan(streamGraph);
    --- End diff --
    
    Ah. Yes. I was earlier throwing the exception in `setCurrentPlan`. Forgot to change this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146896402
  
    Yes, that is true. We're failing them which is a similar to failing for eager executions but a much cleaner and easier solution. Essentially, we cannot detect what a user does in interactive mode, that's why we shouldn't allow detached mode for interactive programs until we have a more sophisticated solution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155861149
  
    @mxm , rebased. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146912518
  
    We can go with the `Program` interface ,but I think that is such a heavy restriction that this effectively renders the feature useless...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-149172503
  
    The test failures are on the flaky scala shell integration test.
    
    @mxm @StephanEwen , can you take a look at this so we can add proper documentation to the programming guide.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-147114132
  
    @hsaputra Yes. However, the travis build log doesn't display the exact cause of the failure. In the code, there is code to log the file with prohibited strings at error level, however, it isn't actually happening. 
    The problem is I'm unable to test the module locally. Yarn works without any issues if I use, say, `bin/yarn-session.sh` or start a cluster using `start-yarn.sh`, but the tests are getting stuck. I was just wondering if it requires any special configuration to run the tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146909986
  
    I would solve it pragmatically and simply make the Program interface a requirement for detached programs. Otherwise, fail with an explanation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772489
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
    @@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
     			// invoke here
     			try {
     				prog.invokeInteractiveModeForExecution();
    +				// if we're here, it means the program is valid for detached mode execution.
    +				JobWithJars plan = ContextEnvironment.ContextEnvironmentFactory.getPlan();
    +				JobGraph graph = ContextEnvironment.ContextEnvironmentFactory.getGraph();
    +				if (plan != null && graph == null) {
    --- End diff --
    
    What would happen if we created a streaming program and a batch program in one jar? Then both would be set and only the batch program would be created...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146898360
  
    When we grab the Optimizer Plan from interactive programs, we also have a similar problem - we would in theory need to execute the whole program. What we do instead is to grab the program part that constitutes the first portion (up to the first job submission to the JobManager)
    
    I agree with you, maybe we can do something like that. Failing all interactive programs (in the sense of programs with a `main` function rather than implementing the `Program` interface) is too much of a restriction.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153599182
  
    Build passes. Let's include this in the next rc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-154428753
  
    There are still two issues I noticed. Printing to standard out should be disabled during the dry-run phase. Just like in the OptimizerPlanEnvironment. The other one is ContextEnvironment. Do you think you could refactor your changes in ContextEnvironment into a separate DetachedEnvironment? We don't really need the context for the dry run. Thus, we have a better separation of concerns with an extra class for the detached execution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-152989560
  
    It's fine to have separate code for detached and blocking submission where necessary. I was more concerned with this pull request that it introduces new code branches which distort the existing submission logic. Wherever possible we should try to be explicit about the submission code path we're in instead of mixing them all together.
    
    For instance, you're setting `lastEnv“ on blocking execution but then you never access it during blocking execution. This is not a clear code style.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098424
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
    @@ -271,18 +272,21 @@ public JobSubmissionResult runDetached(PackagedProgram prog, int parallelism)
     		}
     		else if (prog.isUsingInteractiveMode()) {
     			LOG.info("Starting program in interactive mode");
    -			ContextEnvironment.setAsContext(this, prog.getAllLibraries(), prog.getClasspaths(),
    -				prog.getUserCodeClassLoader(), parallelism, false);
    -
    +			ContextEnvironment.ContextEnvironmentFactory factory = ContextEnvironment.setAsContext(this,
    +					prog.getAllLibraries(), prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
     			// invoke here
     			try {
     				prog.invokeInteractiveModeForExecution();
    +				ContextEnvironment ctx = factory.getLastEnvironment();
    +				if (ctx == null) {
    +					throw new InvalidProgramException("No execution environment was created.");
    --- End diff --
    
    It's not strictly necessary. This should return a `JobSubmissionResult` with null job id then.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42774635
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
    --- End diff --
    
    I was following the code in `testGetExecutionPlan`. Can you elaborate a bit more?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42886049
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    Yes. But marking the class `private` won't interfere with that. I've already pushed a fix and it does pass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153381184
  
    Cool. :)
    Squashing commits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145824055
  
    Yes. I agree. It started out as a simple issue to support detached jobs but gotten a bit messy.
    1. In detached mode, whether the program is run in interactive mode or normal mode, if there is a call to an eager execution function, it will lead to an exception. [As of now.]
    2. This is where the problem started to appear more difficult. Since the accumulator addition is entirely handled by Runtime Context, there can be no check whether the job is in detached mode. Of course, having to store this information in two places[`ExecutionEnvironment` and `RuntimeContext`] is poor design. 
    If the job is being run in detached mode, the accumulators are lost and user can never access them or use them.
    But disallowing accumulators will effectively break most of existing programs. So that's bad choice.
    Eager execution functions use just accumulators actually, so, to maintain harmony, we have to maintain them too.
    If we just consider the first commit of this PR which adds support for executing detached jobs, on a eager call, the main program throws an exception, because the job execution result doesn't contain any accumulators.
    ```java
    this.flatMap(new Utils.CollectHelper<T>(id, serializer)).name("collect()")
    	.output(new DiscardingOutputFormat<T>()).name("collect() sink");
    JobExecutionResult res = getExecutionEnvironment().execute();
    ```
    I'm really not sure how to handle accumulators in detached programs. What we can do is:
    In `JobExecutionResult`, whenever an accumulator is requested, we check for the runtime to be -1, which indicates job was exited in detached mode. Display  a warning to the user on the Cli itself and indicate that the results of accumulators are available on the web frontend [which will require writing them to a file on the job manager]. 
    One more thing: Since the detached mode is available on Yarn, it must throw the exception too in case of `collect` or `print` calls. Is this the case? I can't check on a yarn cluster myself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44126978
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -114,15 +115,33 @@ public Client getClient() {
     	public List<URL> getClasspaths(){
     		return classpathsToAttach;
     	}
    -	
    +
    +	public ClassLoader getUserCodeClassLoader() {
    +		return userCodeClassLoader;
    +	}
    +
    +	public void setCurrentPlan(FlinkPlan plan) {
    --- End diff --
    
    Fair enough. Changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155059238
  
    Yes, the idea is suppress output during the dry-run and plan creation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146895195
  
    @mxm, just to confirm one thing:
    Interactive mode is defined when the the main class isn't of the type `Program`. In that case, doesn't it mean we're essentially failing most of the programs in detached mode? Unless my understanding is wrong of course, in which case could you clarify?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151430438
  
    Thanks. Looks much better. Let's make sure we don't break any classloader or jar dependencies during job submission. This can cause annoyances for users and we have to fix it afterwards :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44643881
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java ---
    @@ -0,0 +1,116 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.client.program;
    +
    +import org.apache.flink.api.common.InvalidProgramException;
    +import org.apache.flink.api.common.JobExecutionResult;
    +import org.apache.flink.api.common.JobID;
    +import org.apache.flink.api.common.JobSubmissionResult;
    +import org.apache.flink.api.common.Plan;
    +import org.apache.flink.optimizer.plan.FlinkPlan;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.net.URL;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Execution Environment for remote execution with the Client in detached mode.
    + */
    +public class DetachedEnvironment extends ContextEnvironment {
    +
    +	/** Keeps track of the program plan for the Client to access. */
    +	private FlinkPlan detachedPlan;
    +
    +	private static final Logger LOG = LoggerFactory.getLogger(DetachedEnvironment.class);
    +
    +	public DetachedEnvironment(Client remoteConnection, List<URL> jarFiles, List<URL> classpaths, ClassLoader userCodeClassLoader) {
    +		super(remoteConnection, jarFiles, classpaths, userCodeClassLoader);
    +	}
    +
    +	@Override
    +	public JobExecutionResult execute(String jobName) throws Exception {
    +		Plan p = createProgramPlan(jobName);
    +		setDetachedPlan(Client.getOptimizedPlan(client.compiler, p, getParallelism()));
    +		LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    +		this.lastJobExecutionResult = DetachedJobExecutionResult.INSTANCE;
    +		return this.lastJobExecutionResult;
    +	}
    +
    +	public void setDetachedPlan(FlinkPlan plan) {
    +		if (detachedPlan == null) {
    +			detachedPlan = plan;
    +		} else {
    +			throw new InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
    +					DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
    +		}
    +	}
    +
    +	/**
    +	 * Finishes this Context Environment's execution by explicitly running the plan constructed.
    +	 */
    +	public JobSubmissionResult finalizeExecute() throws ProgramInvocationException {
    --- End diff --
    
    I'm not sure I understand. `execute()` should default to `execute(null)`, no? 
    We don't really want to allow the user to trigger execution explicitly, which also makes me realize this should be a package private method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145483163
  
    Fair enough, we can do this but the error reporting must be pretty good. Otherwise it might confuse users because not everyone gets that eager execution spawns multiple jobs. The documentation should also explain why detached jobs cannot contain eager executions. I'm not sure whether we can still include this in 0.10 but please go ahead if you want to improve the pull request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145818892
  
    Can we step back a bit with this issue? The design for this seems quite crucial to get right to not get extreme code complexity/bloat and make everything about job submission, accumulators, etc very hard in the future.
    
    What is the design behind this pull request?
      - Can the "detached mode" handle interactive programs?
      - What about programs that use accumulators?
      - If not, how would they be recognized and what errors can be given?
      - The problem here seems very similar to intercepting the execution plan of a job and returning it. Any code sharing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145044584
  
    The current version of the detached mode doesn't really work for eager execution calls like `count()` or `collect()`. Those calls will also be executed detached and thus the result will never be available. I think we can only get this to work if we move the handling of the eager execution into the JobManager.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43015201
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    Okay. Cool. Lemme know if there's any other issues to address. :smile:


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155137822
  
    I'm not sure why we should do this in Detached mode. When `info` is run, user has only directly explicitly requested the plan. This is a full fledged execution of the use code, and should show any user generated out and err messages in my opinion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153690460
  
    That's a good point. It might inadvertently break things.
    For the release candidate, how about including just the first part, i.e., just the detached mode for standalone clusters. That would not break any functionality, and bring the yarn and standalone mode at par as far as detached mode is concerned. It will definitely stay broken for both in case of eager jobs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155047586
  
    Thanks for the timely update @sachingoel0101. 
    
    I was referring to the acquisition of the plan. When we create the plan, (e.g. ./bin/flink info JAR) we suppress printing to standard out. I thought we should do similar when we create the detached plan which we submit only after we have determined whether we can run the jar in detached mode. I ran some of the examples in detached mode and found it odd that it prints to stdout during the creation of the plan. The actual execution of the plan should of course be able to print to stdout. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-152987728
  
    @mxm , there is a lot of code duplication between the detached and blocking submission modes. I'm gonna try to clean that up while I address your concerns.
    Does it make sense to have a single run method which is called as `run(..., mode)` where mode is an enum?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43609728
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
     	@Override
     	public JobExecutionResult execute(String jobName) throws Exception {
     
    -		JobGraph jobGraph;
    -		if (jobName == null) {
    -			jobGraph = this.getStreamGraph().getJobGraph();
    -		} else {
    -			jobGraph = this.getStreamGraph().getJobGraph(jobName);
    -		}
    -
    -		transformations.clear();
    -
    -		// attach all necessary jar files to the JobGraph
    -		for (URL file : jars) {
    --- End diff --
    
    I pushed a fix for this issue: bf29de981c2bcd5cb5d33c68b158c95c8820f43d
    It think you have already adapted to this change. Nice :) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146877512
  
    +1 for now, just let submission fail for detached interactive programs (with a nice exception) 
    
    @sachingoel0101 Could you change the PR accordingly? This would also fix the detached YARN mode for interactive programs.
    
    +1 for having a dedicated execution container for YARN in the long run


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772661
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
    --- End diff --
    
    This should write to System.getProperty("java.io.tmpdir") and have some other Random input to be cross-platform compatible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/1214


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145949994
  
    @sachingoel0101 You're right. The detached YARN execution mode is broken for jobs which use eager execution and build other jobs using the results. However, the detached YARN execution could work if the ApplicationMaster executed the program. Essentially, this means shifting the Client to the YARN cluster.
    
    @StephanEwen The way Sachin was going to handle the eager execution during detached mode is to throw an Exception. However, the current code doesn't handle all cases. It is in the nature of an interactive session that multiple jobs can be submitted. Results from submitted jobs can be reused not only via count/collect but also using the returned JobExecutionResult. Therefore, it is very difficult to deal with all these cases. IMHO we can only support non-interactive detached job submissions.
    
    All in all, I think this is a nice idea but implementing it has too much overhead. The implications of the changes are not quite clear. I pledge to change this to "Won't Fix" until we think about a proper implementation. In the meantime, we should fix the detached YARN mode or remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-147235260
  
    Seems like each build fail in different places.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42879750
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    Oh didn't see you actually use these. Then its fine. Perhaps make them private.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155052238
  
    When the main method is invoked via `PackagedProgram#invokeInteractiveModeForExecution`, it executes everything, and the flink plan it builds is stored in the `DetachedEnvironment`[now]. 
    I just ran an example with the new code, and it prints something like this on the cli:
    ![capture](https://cloud.githubusercontent.com/assets/8874261/11033786/2cfad902-870c-11e5-8605-8bca4782443f.PNG)
    Is this what you mean? If not, can you check again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42774255
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
    @@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
     			// invoke here
     			try {
     				prog.invokeInteractiveModeForExecution();
    +				// if we're here, it means the program is valid for detached mode execution.
    +				JobWithJars plan = ContextEnvironment.ContextEnvironmentFactory.getPlan();
    +				JobGraph graph = ContextEnvironment.ContextEnvironmentFactory.getGraph();
    +				if (plan != null && graph == null) {
    --- End diff --
    
    Ah. I had not considered this possibility. What should be done in such a case? Execute both programs? That does make sense, but what should be the return value? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-147112611
  
    @sachingoel0101 , could you access this URL: https://travis-ci.org/apache/flink/jobs/84664370
    
    You could always set your own Apache Flink forked repo. That way you could try to send PR against your own copy of the repo to kick off Travis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42882261
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    > I guess then the invocation of the main method doesn't work properly...
    What do you mean by that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098099
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
     	@Override
     	public JobExecutionResult execute(String jobName) throws Exception {
     
    -		JobGraph jobGraph;
    -		if (jobName == null) {
    -			jobGraph = this.getStreamGraph().getJobGraph();
    -		} else {
    -			jobGraph = this.getStreamGraph().getJobGraph(jobName);
    -		}
    -
    -		transformations.clear();
    -
    -		// attach all necessary jar files to the JobGraph
    -		for (URL file : jars) {
    --- End diff --
    
    No Jars for detached jobs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145496208
  
    Okay. I will push a fix. I intend to solve it like this:
    Maintain a static variable in `ExecutionEnvironment`, which is set to true whenever the job is submitted in a detached mode. And the DataSet api throws exceptions accordingly, which can again be handled by the Client to display proper error message. Does that sound okay?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43635433
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -165,12 +181,65 @@ public ContextEnvironmentFactory(Client client, List<URL> jarFilesToAttach,
     		
     		@Override
     		public ExecutionEnvironment createExecutionEnvironment() {
    -			ContextEnvironment env = new ContextEnvironment(client, jarFilesToAttach, classpathsToAttach,
    -					userCodeClassLoader, wait);
    -			if (defaultParallelism > 0) {
    -				env.setParallelism(defaultParallelism);
    +			if (wait || lastEnv == null) {
    --- End diff --
    
    Could you reformulate the if-else? That way the code would be more explicit:
    
    ```java
    // We only allow one detached environment in detached mode
    if (!wait && lastEnv != null) {
      throw new DetachedProgramException(..);
    }
    // regular submission flow
    lastEnv = ...
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42772634
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List<URL> jarFilesToAttach,
     			this.userCodeClassLoader = userCodeClassLoader;
     			this.defaultParallelism = defaultParallelism;
     			this.wait = wait;
    +			_plan = null;
    +			_graph = null;
    --- End diff --
    
    Is there a reason for these odd names? `lastBatchPlan` and `lastStreamJobGraph` would be more appropriate. The first one is for batch programs, the latter for JobGraph created by the streaming API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42884467
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    A `PackagedProgram` needs to have a main class with a main method with a `public` modifier. That's why it wouldn't work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098720
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -909,7 +919,21 @@ private int handleError(Throwable t) {
     		}
     		LOG.error("Error while running the command.", t);
     
    -		t.printStackTrace();
    +		// check if the error was due to an invalid program in detached mode.
    +		if (t instanceof ProgramInvocationException && t.getCause() instanceof DetachedProgramException) {
    +			System.err.println(t.getCause().getMessage());
    +			// now trace to the user's main method. We don't wanna show unnecessary information
    +			// in this particular case.
    --- End diff --
    
    True. I didn't wanna change the existing stack traces.
    This will require maintaining a global variable for whether the program was run in interactive mode. The `DetachedProgramException` is a special case where we don't wanna clutter the stack trace with flink's trace, instead just show the user program's stack trace.
    In other cases, this might not be the case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44040745
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -81,13 +60,15 @@ public JobExecutionResult execute(String jobName) throws Exception {
     
     		transformations.clear();
     
    -		// execute the programs
    -		if (wait) {
    -			return client.runBlocking(streamGraph, jars, classpaths, userCodeClassLoader);
    +		if (ctx.isWait()) {
    +			return ctx.getClient().runBlocking(streamGraph, ctx.getJars(), ctx.getClasspaths(),
    +					ctx.getUserCodeClassLoader());
     		} else {
    -			JobSubmissionResult result = client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
    +			ctx.setCurrentPlan(streamGraph);
    --- End diff --
    
    Here, the last call to execute() would override all the previous ones. I think you should throw an Exception here like in ContextEnvironment. Or you throw the Exception on setCurrentPlan. We want to allow only one job to be submitted detached, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155842188
  
    There is a merge conflict. Looks good otherwise.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146914064
  
    There are three root causes(please add if I've missed any.)
    1. Call to execute twice
    2. Access job execution results.
    3. Executing one of the eager execution methods as the first part of program. [Second time is automatically a failure]
    We can handle all three easily. If a program doesn't have any of these, it shouldn't be failed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146913514
  
    IMO even removing the detached mode might be better than restricting to `Program` interface. Instead of failing late in the execution, or with a non-informative warning, it would be better to fail before submitting anything to the JobManager, and throw a nice exception message documenting the exact cause of failure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146874010
  
    Before interactive programs were allowed, the detached mode would have been worked fine because a single JobGraph would have been transferred to the job manager and executed there. This doesn't make sense now, unless we have another dedicated Executor (like you proposed). If that one is on the job manager, we need a fault tolerance mechanism there. Even after the HA changes, the Client is still the a single point of failure for interactive programs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44645161
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java ---
    @@ -0,0 +1,116 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.client.program;
    +
    +import org.apache.flink.api.common.InvalidProgramException;
    +import org.apache.flink.api.common.JobExecutionResult;
    +import org.apache.flink.api.common.JobID;
    +import org.apache.flink.api.common.JobSubmissionResult;
    +import org.apache.flink.api.common.Plan;
    +import org.apache.flink.optimizer.plan.FlinkPlan;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.net.URL;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Execution Environment for remote execution with the Client in detached mode.
    + */
    +public class DetachedEnvironment extends ContextEnvironment {
    +
    +	/** Keeps track of the program plan for the Client to access. */
    +	private FlinkPlan detachedPlan;
    +
    +	private static final Logger LOG = LoggerFactory.getLogger(DetachedEnvironment.class);
    +
    +	public DetachedEnvironment(Client remoteConnection, List<URL> jarFiles, List<URL> classpaths, ClassLoader userCodeClassLoader) {
    +		super(remoteConnection, jarFiles, classpaths, userCodeClassLoader);
    +	}
    +
    +	@Override
    +	public JobExecutionResult execute(String jobName) throws Exception {
    +		Plan p = createProgramPlan(jobName);
    +		setDetachedPlan(Client.getOptimizedPlan(client.compiler, p, getParallelism()));
    +		LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    +		this.lastJobExecutionResult = DetachedJobExecutionResult.INSTANCE;
    +		return this.lastJobExecutionResult;
    +	}
    +
    +	public void setDetachedPlan(FlinkPlan plan) {
    +		if (detachedPlan == null) {
    +			detachedPlan = plan;
    +		} else {
    +			throw new InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
    +					DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
    +		}
    +	}
    +
    +	/**
    +	 * Finishes this Context Environment's execution by explicitly running the plan constructed.
    +	 */
    +	public JobSubmissionResult finalizeExecute() throws ProgramInvocationException {
    --- End diff --
    
    Ah you're overwriting`execute()` already. Sorry. Yes, please make the finalize method package private.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146918295
  
    Good points. 
    
    I think that should do it, Sachin. Curious, how you'll implement it :) Don't we run into the problem that we never know if another execution followed unless we execute the whole user code (might have some other side effects)? However, it's the only way to check whether we can submit the job detached. Another approach I can think of is bytecode analysis.. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44117601
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -114,15 +115,33 @@ public Client getClient() {
     	public List<URL> getClasspaths(){
     		return classpathsToAttach;
     	}
    -	
    +
    +	public ClassLoader getUserCodeClassLoader() {
    +		return userCodeClassLoader;
    +	}
    +
    +	public void setCurrentPlan(FlinkPlan plan) {
    --- End diff --
    
    Isn't `setDetachedPlan` a better name? It's only used for the detached mode and it's either set or null.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43610296
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -36,28 +31,16 @@
     
     	private static final Logger LOG = LoggerFactory.getLogger(StreamContextEnvironment.class);
     
    -	private final List<URL> jars;
    -
    -	private final List<URL> classpaths;
    -	
    -	private final Client client;
    +	private final ContextEnvironment ctx;
     
     	private final ClassLoader userCodeClassLoader;
    -	
    -	private final boolean wait;
     
    -	protected StreamContextEnvironment(Client client, List<URL> jars, List<URL> classpaths, int parallelism,
    -			boolean wait) {
    -		this.client = client;
    -		this.jars = jars;
    -		this.classpaths = classpaths;
    -		this.wait = wait;
    -		
    -		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(jars, classpaths,
    -				getClass().getClassLoader());
    +	protected StreamContextEnvironment(ContextEnvironment ctx) {
    +		this.ctx = ctx;
    +		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(), getClass().getClassLoader());
    --- End diff --
    
    Okay. I will go through all usages of classloaders and try to delegate them to the `Client`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43273230
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
    @@ -271,18 +272,21 @@ public JobSubmissionResult runDetached(PackagedProgram prog, int parallelism)
     		}
     		else if (prog.isUsingInteractiveMode()) {
     			LOG.info("Starting program in interactive mode");
    -			ContextEnvironment.setAsContext(this, prog.getAllLibraries(), prog.getClasspaths(),
    -				prog.getUserCodeClassLoader(), parallelism, false);
    -
    +			ContextEnvironment.ContextEnvironmentFactory factory = ContextEnvironment.setAsContext(this,
    +					prog.getAllLibraries(), prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
     			// invoke here
     			try {
     				prog.invokeInteractiveModeForExecution();
    +				ContextEnvironment ctx = factory.getLastEnvironment();
    +				if (ctx == null) {
    +					throw new InvalidProgramException("No execution environment was created.");
    --- End diff --
    
    Ok let's leave it like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44117608
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -114,15 +115,33 @@ public Client getClient() {
     	public List<URL> getClasspaths(){
     		return classpathsToAttach;
     	}
    -	
    +
    +	public ClassLoader getUserCodeClassLoader() {
    +		return userCodeClassLoader;
    +	}
    +
    +	public void setCurrentPlan(FlinkPlan plan) {
    +		if (currentPlan == null) {
    +			currentPlan = plan;
    +		} else {
    +			throw new DetachedProgramException(
    +					DetachedJobExecutionResult.DETACHED_MESSAGE + DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE
    +			);
    +		}
    +	}
    +
    +	public FlinkPlan getCurrentPlan() {
    --- End diff --
    
    Same as above here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-156094351
  
    @mxm , should be ready to merge now. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-150299549
  
    @mxm , I have addressed some of your comments. 
    Two things still remain: batch + stream jobs, and better tests. Need advice on those.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43609625
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -36,28 +31,16 @@
     
     	private static final Logger LOG = LoggerFactory.getLogger(StreamContextEnvironment.class);
     
    -	private final List<URL> jars;
    -
    -	private final List<URL> classpaths;
    -	
    -	private final Client client;
    +	private final ContextEnvironment ctx;
     
     	private final ClassLoader userCodeClassLoader;
    -	
    -	private final boolean wait;
     
    -	protected StreamContextEnvironment(Client client, List<URL> jars, List<URL> classpaths, int parallelism,
    -			boolean wait) {
    -		this.client = client;
    -		this.jars = jars;
    -		this.classpaths = classpaths;
    -		this.wait = wait;
    -		
    -		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(jars, classpaths,
    -				getClass().getClassLoader());
    +	protected StreamContextEnvironment(ContextEnvironment ctx) {
    +		this.ctx = ctx;
    +		this.userCodeClassLoader = JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(), getClass().getClassLoader());
    --- End diff --
    
    Classloaders rules are the same for streaming and batch. We should try to reuse code when possible. So probably a good idea to delgate this to the Client class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151435402
  
    Agreed. I'll build and test with the example programs in standalone mode and on yarn. It should work perfectly though since `YarnSessionFIFOITCase` checks both batch and streaming jobs on detached clusters, and the standalone mode is tested in `ClientTest#testDetached`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098268
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -909,7 +919,21 @@ private int handleError(Throwable t) {
     		}
     		LOG.error("Error while running the command.", t);
     
    -		t.printStackTrace();
    +		// check if the error was due to an invalid program in detached mode.
    +		if (t instanceof ProgramInvocationException && t.getCause() instanceof DetachedProgramException) {
    +			System.err.println(t.getCause().getMessage());
    +			// now trace to the user's main method. We don't wanna show unnecessary information
    +			// in this particular case.
    --- End diff --
    
    Why in this case? Doesn't that apply to all errors that occur during interactive execution?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145023847
  
    I tested the normal execution mode. However, I don't have a yarn setup. Can someone give it a quick check with yarn?
    Quick question though: In detached mode, print or collect throw exceptions. Is that intended?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43498784
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
     	@Override
     	public JobExecutionResult execute(String jobName) throws Exception {
     
    -		JobGraph jobGraph;
    -		if (jobName == null) {
    -			jobGraph = this.getStreamGraph().getJobGraph();
    -		} else {
    -			jobGraph = this.getStreamGraph().getJobGraph(jobName);
    -		}
    -
    -		transformations.clear();
    -
    -		// attach all necessary jar files to the JobGraph
    -		for (URL file : jars) {
    --- End diff --
    
    Jars and classpaths also need to be attached to the JobGraph of the non-detached streaming job, right? Wouldn't it be nice to always do that at the Client? It's always good to avoid multiple branches for doing the same thing. `Client.getJobGraph(FlinkPlan,..` enables you to pass the streamgraph directly with the jars and classpaths.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098367
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
     	@Override
     	public JobExecutionResult execute(String jobName) throws Exception {
     
    -		JobGraph jobGraph;
    -		if (jobName == null) {
    -			jobGraph = this.getStreamGraph().getJobGraph();
    -		} else {
    -			jobGraph = this.getStreamGraph().getJobGraph(jobName);
    -		}
    -
    -		transformations.clear();
    -
    -		// attach all necessary jar files to the JobGraph
    -		for (URL file : jars) {
    --- End diff --
    
    The detached job goes through `Client#getJobGraph(FlinkPlan, List<URL>, List<URL>)` where the jars and classpaths are added to the job graph. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146876057
  
    Having a dedicated executor on the JobManager seems like a very big thing, for a not so important issue. Why not just fail when someone tries to submit a interactive program detached?
    
    Long term, we can think about deploying the driver program to a YARN container, if desired. That way, the submission process is detached, and the execution follows its core model.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42774775
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    I was concerned about this too. Perhaps a better idea would be have these classes in the test-jar?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151849439
  
    @mxm , if there are any more comments, let me know. I'll try to resolve them quickly so we can include this in the next rc (which will likely be the last one(?)). A round of testing might be needed to make sure everything works fine. I'm not much aware of how streaming jobs work, so I might have missed something while testing myself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r41248646
  
    --- Diff: docs/apis/programming_guide.md ---
    @@ -440,6 +440,46 @@ The `execute()` method is returning the `JobExecutionResult`, including executio
     accumulator results. `print()` and `collect()` are not returning the result, but it can be
     accessed from the `getLastJobExecutionResult()` method.
     
    +Further, `collect`, `print`, `printToErr` and `count` methods are *eager execution* methods. This
    --- End diff --
    
    I think parenthesis make these easier to recognize as methods when skimming over text.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155138232
  
    Of course, anything Flink will print, say, log messages, will already be disabled as there is no actor to receive the messages.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42882120
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +141,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class);
    +			out.runDetached(prg, 1);
    +		} catch (ProgramInvocationException e) {
    --- End diff --
    
    Aha. Good point. Will fix it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42778575
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
    --- End diff --
    
    Actually, these arguments are not needed. I will simply remove them.
    Also, in `textGetExecutionPlan` where these values are used, the locations are not actually accessed. Only plan is tested. So shouldn't be an issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-147091911
  
    I tested this out on yarn and without yarn. Works like a charm. 
    However, the travis builds are failing in `flink-yarn-tests`. Is there any way I can access the travis logs? I tried debugging in IntelliJ but it fails to find the jar. Running on linux is getting stuck at `Connecting to ResourceManager at ubuntu:8032`. Is there anything specific to keep in mind while running yarn tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151224527
  
    I took a look at this again because I wanted to merge it for 0.10. However, I think it still needs a bit of work. The `ContextEnvironmentFactory` shouldn't hold the state for detached batch and stream executions. Could you store the batch plan in the `ContextEnvironment` and the stream graph in the `StreamContextEnvironment` (non-static)? Please give them appropriate names.
    
    Since the `ContextEnvironmentFactory` is currently the entry point for batch and streaming execution (the streaming environment just checks whether the `ContextEnvironment` is instantiated), you may set a flag in the factory to prevent multiple executions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-147997899
  
    Failures on unrelated cases: JIRA 2847, and the scala shell test is flaky. Till is working on it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-151458018
  
    @mxm , I have tested the build with the following commands:
    1. Standalone cluster: `bin/flink run -d` for both streaming and batch wordcount examples.
    2. Yarn: `bin/flink run -d -m yarn-cluster` and `bin/flink run -m yarn-cluster -yd` for both streaming and batch wordcount.
    Lemme know if there's any other checks to be made.
    Also, what should I do about https://github.com/apache/flink/pull/1214#discussion_r43098424? Let it be or return a null job id?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-152990620
  
    The lastEnv is set for blocking also, because it still makes sense. It just stores what the last environment created by the factory was.
    I worked on this a while back while fixing a bug(?) in `TestEnvironment` which was meant to allow accessing last created environments. [https://github.com/apache/flink/commit/166b3705c9079efafe9217d98e7edaf54e6a84cf]
    Whether it is accessed or not is an entirely different matter IMO.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-152973475
  
    Thanks for adapting the pull request to the latest master. It is in better shape now but I still think it can be improved. Please make sure there is an explicit way to submit a job in both detached and blocking execution mode. Some classes mix up the two modes (e.g. `StreamContextEnvironment`).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145810314
  
    @mxm , what I'm doing right now is throwing an exception whenever a eager execution function is called in detached mode. However, like you said, maybe the results should be written to a file on the job manager, and we can maintain that information in `JobInfo`. These results can be served to the user from the web frontend under a separate tab for `Outputs`.
    I can see where the relevant changes would be. In `JobManager.scala`, under the case check for `JobStatusChanged`, whenever the `Client` ref is none, we just write the accumulator results to a file. 
    However, I'm not very familiar with how to access type serializes and type information for de-serializing the results. Can you help with that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146901927
  
    That won't suffice. Even if we allow one execute call, we cannot be sure that user won't try to access the `JobExecutionResult`. That also is an issue.
    Disallowing every program which doesn't implement the `Program` interface isn't a good solution. I'm not an expert but most people would usually have a main method. In that case, it doesn't make sense to have a detached mode if we cannot allow such executions.
    We might be able to detect the exact program structure by exploiting the Context Environment. Before actually executing the program, we instead first do a validation.
    1. On an execute call, instead of actually executing the program, we instead build the plan and store it. If another call to execute comes in, we fail.
    2. The `JobExecutionResult` can be wrapped in a `DetachedExecutionResult` which will fail if user tries to access, say accumulators.
    
    In the documentation for detached mode, we should specify that only programs with one execute call, and no usage of the execution result will be allowed. This handles the eager execution issue also.
    But I'm concerned this isn't the best possible design.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42774405
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java ---
    @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List<URL> jarFilesToAttach,
     			this.userCodeClassLoader = userCodeClassLoader;
     			this.defaultParallelism = defaultParallelism;
     			this.wait = wait;
    +			_plan = null;
    +			_graph = null;
    --- End diff --
    
    I tend to use `_field` appended names instead of `this.field`. But yeah. `last*` makes more sense here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-154339878
  
    Failure on an unrelated test: JobManagerSubmittedJobGraphsRecoveryITCase


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153377995
  
    Looks good @sachingoel0101. Good to merge from my side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146980054
  
    @mxm @StephanEwen I have an implementation ready. I have also added unit tests to verify the behavior. Let me know if this is an acceptable way. All the relevant changes are in the second commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155717073
  
    @mxm , fortunately, this time I was able to run the tests on my machine. :)
    I'm seeing a strange error. This is the error in Task manager log file for per job detached streaming test:
    ```java
    java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.
    	at org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:549)
    	at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:226)
    	at org.apache.flink.api.java.io.TextOutputFormat.open(TextOutputFormat.java:77)
    	at org.apache.flink.streaming.api.functions.sink.FileSinkFunction.open(FileSinkFunction.java:62)
    	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
    	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
    	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:286)
    	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:213)
    	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
    	at java.lang.Thread.run(Thread.java:745)
    ```
    
    This is surprising because I have made no changes to the way file writes or overwrites are handled. 
    Here's the thing though: it should fail, because the folder is created at line 475 in `YarnSessionFIFOITCase` and the program will need to overwrite it.
    ```java
    File tmpOutFolder = null;
    try{
    	tmpOutFolder = tmp.newFolder();
    }
    catch(IOException e) {
    	throw new RuntimeException(e);
    }
    ```
    But the tests always pass on the master branch. 
    The rest of the tests which follow this automatically fail because they again check all the logs files for exception strings.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42979520
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -304,4 +368,60 @@ public String getDescription() {
     			return "TestOptimizerPlan <input-file-path> <output-file-path>";
     		}
     	}
    +
    +	public static final class TestExecuteTwice {
    +
    +		public static void main(String args[]) throws Exception {
    --- End diff --
    
    Ok that's better then! Thanks



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r42879700
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
    @@ -139,6 +141,69 @@ public void shouldSubmitToJobClient() {
     	}
     
     	/**
    +	 * Tests that invalid detached mode programs fail.
    +	 */
    +	@Test
    +	public void testDetachedMode() throws Exception{
    +		jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class), JobManager.JOB_MANAGER_NAME());
    +		Client out = new Client(config);
    +
    +		try {
    +			PackagedProgram prg = new PackagedProgram(TestExecuteTwice.class);
    +			out.runDetached(prg, 1);
    +		} catch (ProgramInvocationException e) {
    --- End diff --
    
    The problem with this test is that it doesn't fail if the Program actually runs (which is the aim of the test...).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155764884
  
    Okay. Figured it out. The Parallelism in the `ContextEnvironment` wasn't being respected in `StreamContextEnvironment`. Passes on my local machine now; waiting for travis to give the green light now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43098098
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
     	@Override
     	public JobExecutionResult execute(String jobName) throws Exception {
     
    -		JobGraph jobGraph;
    -		if (jobName == null) {
    -			jobGraph = this.getStreamGraph().getJobGraph();
    -		} else {
    -			jobGraph = this.getStreamGraph().getJobGraph(jobName);
    -		}
    -
    -		transformations.clear();
    -
    -		// attach all necessary jar files to the JobGraph
    -		for (URL file : jars) {
    -			jobGraph.addJar(new Path(file.toURI()));
    -		}
    -
    -		jobGraph.setClasspaths(classpaths);
    --- End diff --
    
    No classpaths for detached jobs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145059638
  
    It doesn't really make sense to call `print` or `collect` in detached mode. Even if the result is printed on JobManager, client cannot access it anyway [short of going through the std out logs of JM].
    Also, right now, the stack trace for this failure is uninformative.
    I propose to explicitly disallow functions which access Accumulator results in detached mode *viz* `count`, `collect` and `print`. What do you think?
    We can provide a hint that a sink should be used instead in the exception message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146909628
  
    This will fail hard when users rely on the result of the first job. So imagine your program always worked. You submit it detached and then it fails after 12 hours when the first job completes. Granted, we could alleviate the problem a bit by only allowing execute() for detached programs because then the date has been written to a proper sink. Still, the accumulator values might be used after job completion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-155391848
  
    >Do you know of any change I can make to travis conf so all modules are built but only Yarn tests get executed? 
    
    Yes change the build instructions in .travis.yml to run `mvn clean install -DskipTests` followed by a `cd flink-yarn-tests && mvn verify`. Locally, you can find the logs in the target directory of flink-yarn-tests. There is also a way to upload these log files with an AWS account.
    
    I think the different semantics for YARN and standalone are fine. The `-d` flag always starts the job detached. That's independent of YARN or standalone mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r43750488
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java ---
    @@ -81,13 +60,13 @@ public JobExecutionResult execute(String jobName) throws Exception {
     
     		transformations.clear();
     
    -		// execute the programs
    -		if (wait) {
    -			return client.runBlocking(streamGraph, jars, classpaths, userCodeClassLoader);
    +		if (ctx.isWait()) {
    +			return ctx.getClient().runBlocking(streamGraph, ctx.getJars(), ctx.getClasspaths(),
    +					ctx.getUserCodeClassLoader());
     		} else {
    -			JobSubmissionResult result = client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
    +			ctx.setCurrentPlan(streamGraph);
     			LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    -			return JobExecutionResult.fromJobSubmissionResult(result);
    +			return ContextEnvironment.DetachedJobExecutionResult.INSTANCE;
    --- End diff --
    
    Could you add a comment here which explains why you delay the execution?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145635600
  
    @uce , I have pushed a commit with my fix for disabling eager execution in detached mode. Please have a look. If it's okay, I can go ahead with adding docs for it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-150283627
  
    Thanks for the update @sachingoel0101. I've made some comments. I think the pull request needs a bit of cleanup to make things more explicit. The general approach looks good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-153689197
  
    I'd like to merge this on the master but I don't think it should be included in 0.10 because it is too big of a change for a release candidate. Additionally, it's not a pressing issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145972139
  
    I agree. We need to have a proper handling of eager execution, and why exactly it shouldn't be allowed. The client exits is not a very good explanation from the user viewpoint, and it might end up breaking programs.
    Further, the collect call is used in other API methods as well, for example, in Gelly. Detached mode in the current version will lead to failure of every such code.
    
    One question: What exactly is the significance of a detached submission and what does the user expect when they submit the job in detached mode? From what I understand, it should mean that the user should be able to shut down the machine on which the client is present. They should be able to monitor the output for however long they want, stop and resume at a later point. I'm drawing the analogy from how the `screen` program works on linux systems. 
    According to this understanding of mine, the following organization would make sense:
                                                            __ __ __ __ __ __ __ __ __ __
                                                           |  ExecutorClient(invokes main)  |
                                                           |   [Output available on web        |
    User(command line/ webclient) ->  |    if detached mode. Otherwise |
    [Actor system receives updates     |    sent to user client]                  |
    if not in detached mode otherwise |   [Compute job graph and         |
    shuts down after submitting]          |       submit to Job Manager]       |  -> Task Managers
                                                           |                                                   |
                                                           |             Job Manager                 |
                                                           |__ __ __ __ __ __ __ __ __ __  |
                                                              Job Manager actor system
    
    However, if the executor program is in the job manager actor system, what happens in case of a failure? Can we recover the state of the main thread, its memory space etc. on a different leader?
    
    I hope that added something to a discussion that is now bound to happen on detached mode. :)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1214#discussion_r44645434
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java ---
    @@ -0,0 +1,116 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.client.program;
    +
    +import org.apache.flink.api.common.InvalidProgramException;
    +import org.apache.flink.api.common.JobExecutionResult;
    +import org.apache.flink.api.common.JobID;
    +import org.apache.flink.api.common.JobSubmissionResult;
    +import org.apache.flink.api.common.Plan;
    +import org.apache.flink.optimizer.plan.FlinkPlan;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.net.URL;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Execution Environment for remote execution with the Client in detached mode.
    + */
    +public class DetachedEnvironment extends ContextEnvironment {
    +
    +	/** Keeps track of the program plan for the Client to access. */
    +	private FlinkPlan detachedPlan;
    +
    +	private static final Logger LOG = LoggerFactory.getLogger(DetachedEnvironment.class);
    +
    +	public DetachedEnvironment(Client remoteConnection, List<URL> jarFiles, List<URL> classpaths, ClassLoader userCodeClassLoader) {
    +		super(remoteConnection, jarFiles, classpaths, userCodeClassLoader);
    +	}
    +
    +	@Override
    +	public JobExecutionResult execute(String jobName) throws Exception {
    +		Plan p = createProgramPlan(jobName);
    +		setDetachedPlan(Client.getOptimizedPlan(client.compiler, p, getParallelism()));
    +		LOG.warn("Job was executed in detached mode, the results will be available on completion.");
    +		this.lastJobExecutionResult = DetachedJobExecutionResult.INSTANCE;
    +		return this.lastJobExecutionResult;
    +	}
    +
    +	public void setDetachedPlan(FlinkPlan plan) {
    +		if (detachedPlan == null) {
    +			detachedPlan = plan;
    +		} else {
    +			throw new InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
    +					DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
    +		}
    +	}
    +
    +	/**
    +	 * Finishes this Context Environment's execution by explicitly running the plan constructed.
    +	 */
    +	public JobSubmissionResult finalizeExecute() throws ProgramInvocationException {
    --- End diff --
    
    Already done. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by sachingoel0101 <gi...@git.apache.org>.
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-146911063
  
    I think we need to ensure that interactive programs which can execute successfully should still be allowed.
    What I proposed should work. I can push a commit and we can discuss whether it's a nice solution or not. But it most certainly makes the detached mode more useful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145029090
  
    Looks good but the Maven Checkstyle check doesn't pass...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---