You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2017/12/31 17:45:17 UTC

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/5219

    [FLINK-8332] [flip6] Move savepoint dispose into ClusterClient

    ## What is the purpose of the change
    
    Move the savepoint disposal logic from the CliFrontend into the ClusterClient. This gives
    a better separation of concerns and allows the CliFrontend to be used with different
    ClusterClient implementations.
    
    ## Verifying this change
    
    This change is a trivial rework / code cleanup without any test coverage.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink refactorSavepointCommand

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5219.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5219
    
----
commit c48324702b0ebff3fd7a16e480ead90fc4c9a30b
Author: Till Rohrmann <tr...@...>
Date:   2017-12-07T12:57:24Z

    [FLINK-8328] [flip6] Move Yarn ApplicationStatus polling out of YarnClusterClient
    
    Introduce YarnApplicationStatusMonitor which does the Yarn ApplicationStatus polling in
    the FlinkYarnSessionCli. This decouples the YarnClusterClient from the actual communication
    with Yarn and, thus, gives a better separation of concerns.

commit 783bd1daf36b34a74cbe718a52a1043ba38d5a44
Author: Till Rohrmann <tr...@...>
Date:   2017-12-20T15:43:21Z

    [FLINK-8329] [flip6] Move YarnClient to AbstractYarnClusterDescriptor
    
    Moves the YarnClient from the YarnClusterClient to the AbstractYarnClusterDescriptor.
    This makes the latter responsible for the lifecycle management of the client and gives
    a better separation of concerns.

commit 0cd22bf559eb820f3e2d381686752f583f4f16ff
Author: Till Rohrmann <tr...@...>
Date:   2017-12-18T17:59:30Z

    [FLINK-8332] [flip6] Move savepoint dispose into ClusterClient
    
    Move the savepoint disposal logic from the CliFrontend into the ClusterClient. This gives
    a better separation of concerns and allows the CliFrontend to be used with different
    ClusterClient implementations.

----


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160704416
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -178,65 +185,30 @@ public void testTriggerSavepointCustomTarget() throws Exception {
     	public void testDisposeSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			String savepointPath = "expectedSavepointPath";
    -			ActorGateway jobManager = mock(ActorGateway.class);
    +		String savepointPath = "expectedSavepointPath";
     
    -			Promise<Object> triggerResponse = new scala.concurrent.impl.Promise.DefaultPromise<>();
    +		final CompletableFuture<Acknowledge> disposeCallFuture = new CompletableFuture<>();
     
    -			when(jobManager.ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class))).thenReturn(triggerResponse.future());
    +		ClusterClient clusterClient = new DisposeSavepointClusterClient((String path, Time timeout) -> {
    +			disposeCallFuture.complete(Acknowledge.get());
    +			return CompletableFuture.completedFuture(Acknowledge.get());
    +		});
     
    -			triggerResponse.success(getDisposeSavepointSuccess());
    +		try {
     
    -			CliFrontend frontend = new MockCliFrontend(
    -					CliFrontendTestUtils.getConfigDir(), jobManager);
    +			CliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { "-d", savepointPath };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    -			verify(jobManager, times(1)).ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class));
    +			disposeCallFuture.get();
     
     			String outMsg = buffer.toString();
     			assertTrue(outMsg.contains(savepointPath));
     			assertTrue(outMsg.contains("disposed"));
     		}
     		finally {
    -			restoreStdOutAndStdErr();
    -		}
    -	}
    -
    -	/**
    -	 * Tests that a disposal failure due a  ClassNotFoundException triggers a
    -	 * note about the JAR option.
    -	 */
    -	@Test
    -	public void testDisposeClassNotFoundException() throws Exception {
    --- End diff --
    
    Yes indeed, will add it with the referenced commit.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160699710
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -128,13 +131,14 @@ public void testTriggerSavepointFailureIllegalJobID() throws Exception {
     		replaceStdOutAndStdErr();
     
     		try {
    -			CliFrontend frontend = new CliFrontend(CliFrontendTestUtils.getConfigDir());
    +			CliFrontend frontend = new MockedCliFrontend(new StandaloneClusterClient(
    +				new Configuration(),
    +				new TestingHighAvailabilityServices()));
     
     			String[] parameters = { "invalid job id" };
     			int returnCode = frontend.savepoint(parameters);
     
    -			assertTrue(returnCode != 0);
    -			assertTrue(buffer.toString().contains("not a valid ID"));
    --- End diff --
    
    I think it's not important. We all know it will get removed anyways.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159248959
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -659,128 +655,107 @@ protected int savepoint(String[] args) {
     			return 0;
     		}
     
    -		if (options.isDispose()) {
    -			// Discard
    -			return disposeSavepoint(options);
    -		} else {
    -			// Trigger
    -			String[] cleanedArgs = options.getArgs();
    -			JobID jobId;
    +		CustomCommandLine<?> customCommandLine = getActiveCustomCommandLine(options.getCommandLine());
     
    -			if (cleanedArgs.length >= 1) {
    -				String jobIdString = cleanedArgs[0];
    -				try {
    -					jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    -				} catch (Exception e) {
    -					return handleArgException(new IllegalArgumentException(
    -							"Error: The value for the Job ID is not a valid ID."));
    -				}
    +		ClusterClient clusterClient = customCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    +
    +		try {
    +			if (options.isDispose()) {
    +				// Discard
    +				return disposeSavepoint(clusterClient, options.getSavepointPath());
     			} else {
    -				return handleArgException(new IllegalArgumentException(
    +				// Trigger
    +				String[] cleanedArgs = options.getArgs();
    +				JobID jobId;
    +
    +				if (cleanedArgs.length >= 1) {
    +					String jobIdString = cleanedArgs[0];
    +					try {
    +						jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    --- End diff --
    
    `JobID.fromHexString`


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160706043
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -178,65 +185,30 @@ public void testTriggerSavepointCustomTarget() throws Exception {
     	public void testDisposeSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			String savepointPath = "expectedSavepointPath";
    -			ActorGateway jobManager = mock(ActorGateway.class);
    +		String savepointPath = "expectedSavepointPath";
     
    -			Promise<Object> triggerResponse = new scala.concurrent.impl.Promise.DefaultPromise<>();
    +		final CompletableFuture<Acknowledge> disposeCallFuture = new CompletableFuture<>();
     
    -			when(jobManager.ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class))).thenReturn(triggerResponse.future());
    +		ClusterClient clusterClient = new DisposeSavepointClusterClient((String path, Time timeout) -> {
    +			disposeCallFuture.complete(Acknowledge.get());
    +			return CompletableFuture.completedFuture(Acknowledge.get());
    +		});
     
    -			triggerResponse.success(getDisposeSavepointSuccess());
    +		try {
     
    -			CliFrontend frontend = new MockCliFrontend(
    -					CliFrontendTestUtils.getConfigDir(), jobManager);
    +			CliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { "-d", savepointPath };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    -			verify(jobManager, times(1)).ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class));
    +			disposeCallFuture.get();
    --- End diff --
    
    True. Will remove it.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160697769
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -659,128 +655,107 @@ protected int savepoint(String[] args) {
     			return 0;
     		}
     
    -		if (options.isDispose()) {
    -			// Discard
    -			return disposeSavepoint(options);
    -		} else {
    -			// Trigger
    -			String[] cleanedArgs = options.getArgs();
    -			JobID jobId;
    +		CustomCommandLine<?> customCommandLine = getActiveCustomCommandLine(options.getCommandLine());
     
    -			if (cleanedArgs.length >= 1) {
    -				String jobIdString = cleanedArgs[0];
    -				try {
    -					jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    -				} catch (Exception e) {
    -					return handleArgException(new IllegalArgumentException(
    -							"Error: The value for the Job ID is not a valid ID."));
    -				}
    +		ClusterClient clusterClient = customCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    +
    +		try {
    +			if (options.isDispose()) {
    +				// Discard
    +				return disposeSavepoint(clusterClient, options.getSavepointPath());
     			} else {
    -				return handleArgException(new IllegalArgumentException(
    +				// Trigger
    +				String[] cleanedArgs = options.getArgs();
    +				JobID jobId;
    +
    +				if (cleanedArgs.length >= 1) {
    +					String jobIdString = cleanedArgs[0];
    +					try {
    +						jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    --- End diff --
    
    Good catch. Will change it.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160699213
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -128,13 +131,14 @@ public void testTriggerSavepointFailureIllegalJobID() throws Exception {
     		replaceStdOutAndStdErr();
     
     		try {
    -			CliFrontend frontend = new CliFrontend(CliFrontendTestUtils.getConfigDir());
    +			CliFrontend frontend = new MockedCliFrontend(new StandaloneClusterClient(
    +				new Configuration(),
    +				new TestingHighAvailabilityServices()));
     
     			String[] parameters = { "invalid job id" };
     			int returnCode = frontend.savepoint(parameters);
     
    -			assertTrue(returnCode != 0);
    -			assertTrue(buffer.toString().contains("not a valid ID"));
    --- End diff --
    
    True. Will re-add for the sake of correctness.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159255690
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -128,13 +131,14 @@ public void testTriggerSavepointFailureIllegalJobID() throws Exception {
     		replaceStdOutAndStdErr();
     
     		try {
    -			CliFrontend frontend = new CliFrontend(CliFrontendTestUtils.getConfigDir());
    +			CliFrontend frontend = new MockedCliFrontend(new StandaloneClusterClient(
    +				new Configuration(),
    +				new TestingHighAvailabilityServices()));
     
     			String[] parameters = { "invalid job id" };
     			int returnCode = frontend.savepoint(parameters);
     
    -			assertTrue(returnCode != 0);
    -			assertTrue(buffer.toString().contains("not a valid ID"));
    --- End diff --
    
    What happened to this assertion? It would still pass.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159400326
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -149,23 +153,26 @@ public void testTriggerSavepointFailureIllegalJobID() throws Exception {
     	public void testTriggerSavepointCustomTarget() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			JobID jobId = new JobID();
    +		JobID jobId = new JobID();
     
    -			String savepointDirectory = "customTargetDirectory";
    +		String savepointDirectory = "customTargetDirectory";
     
    -			MockedCliFrontend frontend = new SavepointTestCliFrontend(savepointDirectory);
    +		final ClusterClient clusterClient = createClusterClient(savepointDirectory);
    +
    +		try {
    +			MockedCliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { jobId.toString(), savepointDirectory };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    --- End diff --
    
    Is there a reason why this assertion was removed? I think it would still pass.


---

[GitHub] flink issue #5219: [FLINK-8332] [flip6] Move savepoint dispose into ClusterC...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:

    https://github.com/apache/flink/pull/5219
  
    Thanks for the review @GJL. I've addressed your comments. As soon as Travis gives green light, I'll merge this PR.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159249258
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -659,128 +655,107 @@ protected int savepoint(String[] args) {
     			return 0;
     		}
     
    -		if (options.isDispose()) {
    -			// Discard
    -			return disposeSavepoint(options);
    -		} else {
    -			// Trigger
    -			String[] cleanedArgs = options.getArgs();
    -			JobID jobId;
    +		CustomCommandLine<?> customCommandLine = getActiveCustomCommandLine(options.getCommandLine());
     
    -			if (cleanedArgs.length >= 1) {
    -				String jobIdString = cleanedArgs[0];
    -				try {
    -					jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    -				} catch (Exception e) {
    -					return handleArgException(new IllegalArgumentException(
    -							"Error: The value for the Job ID is not a valid ID."));
    -				}
    +		ClusterClient clusterClient = customCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    +
    +		try {
    +			if (options.isDispose()) {
    +				// Discard
    +				return disposeSavepoint(clusterClient, options.getSavepointPath());
     			} else {
    -				return handleArgException(new IllegalArgumentException(
    +				// Trigger
    +				String[] cleanedArgs = options.getArgs();
    +				JobID jobId;
    +
    +				if (cleanedArgs.length >= 1) {
    +					String jobIdString = cleanedArgs[0];
    +					try {
    +						jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    --- End diff --
    
    nit: `JobID.fromHexString(jobIdString)`


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160698812
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -77,23 +74,25 @@
     	public void testTriggerSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			JobID jobId = new JobID();
    +		JobID jobId = new JobID();
     
    -			String savepointPath = "expectedSavepointPath";
    +		String savepointPath = "expectedSavepointPath";
     
    -			MockedCliFrontend frontend = new SavepointTestCliFrontend(savepointPath);
    +		final ClusterClient clusterClient = createClusterClient(savepointPath);
    +
    +		try {
    +			MockedCliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { jobId.toString() };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    --- End diff --
    
    Because I moved the tests from a later commit where I change the signature of the `savepoint` command to return `void` and throw an `Exception` in case of a failure.  Will add the assertions again.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159401004
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -659,128 +655,107 @@ protected int savepoint(String[] args) {
     			return 0;
     		}
     
    -		if (options.isDispose()) {
    -			// Discard
    -			return disposeSavepoint(options);
    -		} else {
    -			// Trigger
    -			String[] cleanedArgs = options.getArgs();
    -			JobID jobId;
    +		CustomCommandLine<?> customCommandLine = getActiveCustomCommandLine(options.getCommandLine());
     
    -			if (cleanedArgs.length >= 1) {
    -				String jobIdString = cleanedArgs[0];
    -				try {
    -					jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    -				} catch (Exception e) {
    -					return handleArgException(new IllegalArgumentException(
    -							"Error: The value for the Job ID is not a valid ID."));
    -				}
    +		ClusterClient clusterClient = customCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    +
    +		try {
    +			if (options.isDispose()) {
    +				// Discard
    +				return disposeSavepoint(clusterClient, options.getSavepointPath());
     			} else {
    -				return handleArgException(new IllegalArgumentException(
    +				// Trigger
    +				String[] cleanedArgs = options.getArgs();
    +				JobID jobId;
    +
    +				if (cleanedArgs.length >= 1) {
    +					String jobIdString = cleanedArgs[0];
    +					try {
    +						jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    +					} catch (Exception e) {
    +						return handleArgException(new IllegalArgumentException(
    +							"Error: The value for the Job ID is not a valid ID."));
    +					}
    +				} else {
    +					return handleArgException(new IllegalArgumentException(
     						"Error: The value for the Job ID is not a valid ID. " +
    -								"Specify a Job ID to trigger a savepoint."));
    -			}
    +							"Specify a Job ID to trigger a savepoint."));
    +				}
     
    -			String savepointDirectory = null;
    -			if (cleanedArgs.length >= 2) {
    -				savepointDirectory = cleanedArgs[1];
    -			}
    +				String savepointDirectory = null;
    +				if (cleanedArgs.length >= 2) {
    +					savepointDirectory = cleanedArgs[1];
    +				}
     
    -			// Print superfluous arguments
    -			if (cleanedArgs.length >= 3) {
    -				logAndSysout("Provided more arguments than required. Ignoring not needed arguments.");
    -			}
    +				// Print superfluous arguments
    +				if (cleanedArgs.length >= 3) {
    +					logAndSysout("Provided more arguments than required. Ignoring not needed arguments.");
    +				}
     
    -			return triggerSavepoint(options, jobId, savepointDirectory);
    +				return triggerSavepoint(clusterClient, jobId, savepointDirectory);
    +			}
    +		} catch (Exception e) {
    +			return handleError(e);
    +		} finally {
    +			try {
    +				clusterClient.shutdown();
    +			} catch (Exception e) {
    +				LOG.info("Could not shutdown the cluster client.", e);
    +			}
     		}
     	}
     
     	/**
     	 * Sends a {@link org.apache.flink.runtime.messages.JobManagerMessages.TriggerSavepoint}
     	 * message to the job manager.
     	 */
    -	private int triggerSavepoint(SavepointOptions options, JobID jobId, String savepointDirectory) {
    -		try {
    -			CustomCommandLine<?> activeCommandLine = getActiveCustomCommandLine(options.getCommandLine());
    -			ClusterClient client = activeCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    -			try {
    -				logAndSysout("Triggering savepoint for job " + jobId + ".");
    -				CompletableFuture<String> savepointPathFuture = client.triggerSavepoint(jobId, savepointDirectory);
    -
    -				String savepointPath;
    -				try {
    -					logAndSysout("Waiting for response...");
    -					savepointPath = savepointPathFuture.get();
    -				}
    -				catch (ExecutionException ee) {
    -					Throwable cause = ExceptionUtils.stripExecutionException(ee);
    -					throw new FlinkException("Triggering a savepoint for the job " + jobId + " failed.", cause);
    -				}
    -
    -				logAndSysout("Savepoint completed. Path: " + savepointPath);
    -				logAndSysout("You can resume your program from this savepoint with the run command.");
    +	private int triggerSavepoint(ClusterClient clusterClient, JobID jobId, String savepointDirectory) throws Exception {
    +		logAndSysout("Triggering savepoint for job " + jobId + ".");
    +		CompletableFuture<String> savepointPathFuture = clusterClient.triggerSavepoint(jobId, savepointDirectory);
     
    -				return 0;
    -			}
    -			finally {
    -				client.shutdown();
    -			}
    +		String savepointPath;
    +		try {
    +			logAndSysout("Waiting for response...");
    +			savepointPath = savepointPathFuture.get();
     		}
    -		catch (Throwable t) {
    -			return handleError(t);
    +		catch (ExecutionException ee) {
    +			Throwable cause = ExceptionUtils.stripExecutionException(ee);
    +			throw new FlinkException("Triggering a savepoint for the job " + jobId + " failed.", cause);
     		}
    +
    +		logAndSysout("Savepoint completed. Path: " + savepointPath);
    +		logAndSysout("You can resume your program from this savepoint with the run command.");
    +
    +		return 0;
     	}
     
     	/**
     	 * Sends a {@link org.apache.flink.runtime.messages.JobManagerMessages.DisposeSavepoint}
     	 * message to the job manager.
     	 */
    -	private int disposeSavepoint(SavepointOptions options) {
    -		try {
    -			String savepointPath = options.getSavepointPath();
    -			if (savepointPath == null) {
    -				throw new IllegalArgumentException("Missing required argument: savepoint path. " +
    -						"Usage: bin/flink savepoint -d <savepoint-path>");
    -			}
    -
    -			ActorGateway jobManager = getJobManagerGateway(options);
    +	private int disposeSavepoint(ClusterClient clusterClient, String savepointPath) {
    +		Preconditions.checkNotNull(savepointPath, "Missing required argument: savepoint path. " +
    +			"Usage: bin/flink savepoint -d <savepoint-path>");
     
    -			logAndSysout("Disposing savepoint '" + savepointPath + "'.");
    +		logAndSysout("Disposing savepoint '" + savepointPath + "'.");
     
    -			Object msg = new DisposeSavepoint(savepointPath);
    -			Future<Object> response = jobManager.ask(msg, clientTimeout);
    +		CompletableFuture<Acknowledge> disposeFuture = null;
    --- End diff --
    
    nit: `null` initialization is not needed because you return in the `catch` block. Can be even declared final.
      


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159252679
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -77,23 +74,25 @@
     	public void testTriggerSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			JobID jobId = new JobID();
    +		JobID jobId = new JobID();
     
    -			String savepointPath = "expectedSavepointPath";
    +		String savepointPath = "expectedSavepointPath";
     
    -			MockedCliFrontend frontend = new SavepointTestCliFrontend(savepointPath);
    +		final ClusterClient clusterClient = createClusterClient(savepointPath);
    +
    +		try {
    +			MockedCliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { jobId.toString() };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    --- End diff --
    
    Is there a reason why this assertion was removed? I think it would still pass.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160697901
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -659,128 +655,107 @@ protected int savepoint(String[] args) {
     			return 0;
     		}
     
    -		if (options.isDispose()) {
    -			// Discard
    -			return disposeSavepoint(options);
    -		} else {
    -			// Trigger
    -			String[] cleanedArgs = options.getArgs();
    -			JobID jobId;
    +		CustomCommandLine<?> customCommandLine = getActiveCustomCommandLine(options.getCommandLine());
     
    -			if (cleanedArgs.length >= 1) {
    -				String jobIdString = cleanedArgs[0];
    -				try {
    -					jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    -				} catch (Exception e) {
    -					return handleArgException(new IllegalArgumentException(
    -							"Error: The value for the Job ID is not a valid ID."));
    -				}
    +		ClusterClient clusterClient = customCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    +
    +		try {
    +			if (options.isDispose()) {
    +				// Discard
    +				return disposeSavepoint(clusterClient, options.getSavepointPath());
     			} else {
    -				return handleArgException(new IllegalArgumentException(
    +				// Trigger
    +				String[] cleanedArgs = options.getArgs();
    +				JobID jobId;
    +
    +				if (cleanedArgs.length >= 1) {
    +					String jobIdString = cleanedArgs[0];
    +					try {
    +						jobId = new JobID(StringUtils.hexStringToByte(jobIdString));
    +					} catch (Exception e) {
    +						return handleArgException(new IllegalArgumentException(
    +							"Error: The value for the Job ID is not a valid ID."));
    +					}
    +				} else {
    +					return handleArgException(new IllegalArgumentException(
     						"Error: The value for the Job ID is not a valid ID. " +
    -								"Specify a Job ID to trigger a savepoint."));
    -			}
    +							"Specify a Job ID to trigger a savepoint."));
    +				}
     
    -			String savepointDirectory = null;
    -			if (cleanedArgs.length >= 2) {
    -				savepointDirectory = cleanedArgs[1];
    -			}
    +				String savepointDirectory = null;
    +				if (cleanedArgs.length >= 2) {
    +					savepointDirectory = cleanedArgs[1];
    +				}
     
    -			// Print superfluous arguments
    -			if (cleanedArgs.length >= 3) {
    -				logAndSysout("Provided more arguments than required. Ignoring not needed arguments.");
    -			}
    +				// Print superfluous arguments
    +				if (cleanedArgs.length >= 3) {
    +					logAndSysout("Provided more arguments than required. Ignoring not needed arguments.");
    +				}
     
    -			return triggerSavepoint(options, jobId, savepointDirectory);
    +				return triggerSavepoint(clusterClient, jobId, savepointDirectory);
    +			}
    +		} catch (Exception e) {
    +			return handleError(e);
    +		} finally {
    +			try {
    +				clusterClient.shutdown();
    +			} catch (Exception e) {
    +				LOG.info("Could not shutdown the cluster client.", e);
    +			}
     		}
     	}
     
     	/**
     	 * Sends a {@link org.apache.flink.runtime.messages.JobManagerMessages.TriggerSavepoint}
     	 * message to the job manager.
     	 */
    -	private int triggerSavepoint(SavepointOptions options, JobID jobId, String savepointDirectory) {
    -		try {
    -			CustomCommandLine<?> activeCommandLine = getActiveCustomCommandLine(options.getCommandLine());
    -			ClusterClient client = activeCommandLine.retrieveCluster(options.getCommandLine(), config, configurationDirectory);
    -			try {
    -				logAndSysout("Triggering savepoint for job " + jobId + ".");
    -				CompletableFuture<String> savepointPathFuture = client.triggerSavepoint(jobId, savepointDirectory);
    -
    -				String savepointPath;
    -				try {
    -					logAndSysout("Waiting for response...");
    -					savepointPath = savepointPathFuture.get();
    -				}
    -				catch (ExecutionException ee) {
    -					Throwable cause = ExceptionUtils.stripExecutionException(ee);
    -					throw new FlinkException("Triggering a savepoint for the job " + jobId + " failed.", cause);
    -				}
    -
    -				logAndSysout("Savepoint completed. Path: " + savepointPath);
    -				logAndSysout("You can resume your program from this savepoint with the run command.");
    +	private int triggerSavepoint(ClusterClient clusterClient, JobID jobId, String savepointDirectory) throws Exception {
    +		logAndSysout("Triggering savepoint for job " + jobId + ".");
    +		CompletableFuture<String> savepointPathFuture = clusterClient.triggerSavepoint(jobId, savepointDirectory);
     
    -				return 0;
    -			}
    -			finally {
    -				client.shutdown();
    -			}
    +		String savepointPath;
    +		try {
    +			logAndSysout("Waiting for response...");
    +			savepointPath = savepointPathFuture.get();
     		}
    -		catch (Throwable t) {
    -			return handleError(t);
    +		catch (ExecutionException ee) {
    +			Throwable cause = ExceptionUtils.stripExecutionException(ee);
    +			throw new FlinkException("Triggering a savepoint for the job " + jobId + " failed.", cause);
     		}
    +
    +		logAndSysout("Savepoint completed. Path: " + savepointPath);
    +		logAndSysout("You can resume your program from this savepoint with the run command.");
    +
    +		return 0;
     	}
     
     	/**
     	 * Sends a {@link org.apache.flink.runtime.messages.JobManagerMessages.DisposeSavepoint}
     	 * message to the job manager.
     	 */
    -	private int disposeSavepoint(SavepointOptions options) {
    -		try {
    -			String savepointPath = options.getSavepointPath();
    -			if (savepointPath == null) {
    -				throw new IllegalArgumentException("Missing required argument: savepoint path. " +
    -						"Usage: bin/flink savepoint -d <savepoint-path>");
    -			}
    -
    -			ActorGateway jobManager = getJobManagerGateway(options);
    +	private int disposeSavepoint(ClusterClient clusterClient, String savepointPath) {
    +		Preconditions.checkNotNull(savepointPath, "Missing required argument: savepoint path. " +
    +			"Usage: bin/flink savepoint -d <savepoint-path>");
     
    -			logAndSysout("Disposing savepoint '" + savepointPath + "'.");
    +		logAndSysout("Disposing savepoint '" + savepointPath + "'.");
     
    -			Object msg = new DisposeSavepoint(savepointPath);
    -			Future<Object> response = jobManager.ask(msg, clientTimeout);
    +		CompletableFuture<Acknowledge> disposeFuture = null;
    --- End diff --
    
    You're right. Will change it.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r160699257
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -149,23 +153,26 @@ public void testTriggerSavepointFailureIllegalJobID() throws Exception {
     	public void testTriggerSavepointCustomTarget() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			JobID jobId = new JobID();
    +		JobID jobId = new JobID();
     
    -			String savepointDirectory = "customTargetDirectory";
    +		String savepointDirectory = "customTargetDirectory";
     
    -			MockedCliFrontend frontend = new SavepointTestCliFrontend(savepointDirectory);
    +		final ClusterClient clusterClient = createClusterClient(savepointDirectory);
    +
    +		try {
    +			MockedCliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { jobId.toString(), savepointDirectory };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    --- End diff --
    
    Backport of tests.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159402067
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -178,65 +185,30 @@ public void testTriggerSavepointCustomTarget() throws Exception {
     	public void testDisposeSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			String savepointPath = "expectedSavepointPath";
    -			ActorGateway jobManager = mock(ActorGateway.class);
    +		String savepointPath = "expectedSavepointPath";
     
    -			Promise<Object> triggerResponse = new scala.concurrent.impl.Promise.DefaultPromise<>();
    +		final CompletableFuture<Acknowledge> disposeCallFuture = new CompletableFuture<>();
     
    -			when(jobManager.ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class))).thenReturn(triggerResponse.future());
    +		ClusterClient clusterClient = new DisposeSavepointClusterClient((String path, Time timeout) -> {
    +			disposeCallFuture.complete(Acknowledge.get());
    +			return CompletableFuture.completedFuture(Acknowledge.get());
    +		});
     
    -			triggerResponse.success(getDisposeSavepointSuccess());
    +		try {
     
    -			CliFrontend frontend = new MockCliFrontend(
    -					CliFrontendTestUtils.getConfigDir(), jobManager);
    +			CliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { "-d", savepointPath };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    -			verify(jobManager, times(1)).ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class));
    +			disposeCallFuture.get();
    --- End diff --
    
    Everything is run in the main thread:
    ```
    		@Override
    		public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath, Time timeout) {
    			return disposeSavepointFunction.apply(savepointPath, timeout);
    		}
    ```
    There is no need to wait for the completion of this future. The future can be removed.


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5219#discussion_r159403066
  
    --- Diff: flink-clients/src/test/java/org/apache/flink/client/CliFrontendSavepointTest.java ---
    @@ -178,65 +185,30 @@ public void testTriggerSavepointCustomTarget() throws Exception {
     	public void testDisposeSavepointSuccess() throws Exception {
     		replaceStdOutAndStdErr();
     
    -		try {
    -			String savepointPath = "expectedSavepointPath";
    -			ActorGateway jobManager = mock(ActorGateway.class);
    +		String savepointPath = "expectedSavepointPath";
     
    -			Promise<Object> triggerResponse = new scala.concurrent.impl.Promise.DefaultPromise<>();
    +		final CompletableFuture<Acknowledge> disposeCallFuture = new CompletableFuture<>();
     
    -			when(jobManager.ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class))).thenReturn(triggerResponse.future());
    +		ClusterClient clusterClient = new DisposeSavepointClusterClient((String path, Time timeout) -> {
    +			disposeCallFuture.complete(Acknowledge.get());
    +			return CompletableFuture.completedFuture(Acknowledge.get());
    +		});
     
    -			triggerResponse.success(getDisposeSavepointSuccess());
    +		try {
     
    -			CliFrontend frontend = new MockCliFrontend(
    -					CliFrontendTestUtils.getConfigDir(), jobManager);
    +			CliFrontend frontend = new MockedCliFrontend(clusterClient);
     
     			String[] parameters = { "-d", savepointPath };
    -			int returnCode = frontend.savepoint(parameters);
    +			frontend.savepoint(parameters);
     
    -			assertEquals(0, returnCode);
    -			verify(jobManager, times(1)).ask(
    -					Mockito.eq(new DisposeSavepoint(savepointPath)),
    -					any(FiniteDuration.class));
    +			disposeCallFuture.get();
     
     			String outMsg = buffer.toString();
     			assertTrue(outMsg.contains(savepointPath));
     			assertTrue(outMsg.contains("disposed"));
     		}
     		finally {
    -			restoreStdOutAndStdErr();
    -		}
    -	}
    -
    -	/**
    -	 * Tests that a disposal failure due a  ClassNotFoundException triggers a
    -	 * note about the JAR option.
    -	 */
    -	@Test
    -	public void testDisposeClassNotFoundException() throws Exception {
    --- End diff --
    
    The behaviour moved to `ClusterClient` but we are not testing it anymore.
      


---

[GitHub] flink pull request #5219: [FLINK-8332] [flip6] Move savepoint dispose into C...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5219


---

[GitHub] flink issue #5219: [FLINK-8332] [flip6] Move savepoint dispose into ClusterC...

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on the issue:

    https://github.com/apache/flink/pull/5219
  
    This PR is based on #5216 


---