You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2015/01/09 17:21:56 UTC

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

GitHub user rmetzger opened a pull request:

    https://github.com/apache/flink/pull/292

    [FLINK-1295][FLINK-883] Allow to deploy 'job only' YARN cluster. Add tests to YARN

    - users can now also deploy Flink on YARN for executing a single job.
    - the flink-yarn project has been moved out of the flink-addons module
    - the MiniYARNCluster is used for testing Flink on YARN
    - There is now a (undocumented) Java interface Flink's YARN client, allowing users to manually control the YARN session.
    - ALL ports used by Flink when running on YARN are automatically determined. In the past users reported problems with blocked ports (YARN is telling the client the RPC address of the application master)
    - The checks before deployment have been improved to give better error messages if the user is requesting too many resources for a YARN session
    
    The change has been tested on google compute cloud ("click to deploy" with google cloud storage, not hdfs) and amazon emr (hdfs).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rmetzger/flink flink1295-flink883-rebased-after-akka

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #292
    
----
commit 273cafbf44b5796b96ff2396b0f6e68f79ce9185
Author: Robert Metzger <rm...@apache.org>
Date:   2014-12-01T17:59:49Z

    [FLINK-1295][FLINK-883] Allow to deploy 'job only' YARN cluster. Add tests to YARN
    
    - users can now also deploy Flink on YARN for executing a single job.
    - The flink-yarn project has been moved out of the flink-addons module
    - the MiniYARNCluster is used for testing Flink on YARN
    - There is now a (undocumented) Java interface Flink's YARN client, allowing users to manually control the Yarn session.
    - ALL ports used by Flink when running on YARN are automatically determined. In the past users reported problems with blocked ports (YARN is telling the client the RPC address of the application master)
    - The checks before deployment have been improved to give better error messages if the user is requesting too many resources for a YARN session

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/292#issuecomment-70415876
  
    A good piece of work!. I like that there is a much better test coverage for the Yarn cluster now.
    
    @rmetzger Can you comment on a few open questions? 
    
     - There are two projects for yarn: "flink-yarn" and "flink-yarn-tests". Can we combine them in one project? The reason why we put the tests in "flink-scala" into a different project was only beause the Scala macros uses in "flink-scala" caused problems otherwise. Since the Yarn project is not using macros, I see no reason for the separation.
     
     - The example for the YARN session should probably not refer to a local file ("file:///...")
     
     - There is a bit of a strange mechanism in computing the ports for the application master (base port plus attempt-id or so). Can this be simplified? Also, what happens if that port is already in use, per chance?
       
     - How hard would it be to make the client parameter that starts a yarn session for the job simpler? Something like `-y` rather than `-m yarn-cluser`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23175611
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnCluster.java ---
    @@ -0,0 +1,363 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import akka.actor.ActorRef;
    +import akka.actor.ActorSystem;
    +
    +import static akka.pattern.Patterns.ask;
    +
    +import akka.actor.Props;
    +import akka.util.Timeout;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.net.NetUtils;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import scala.None$;
    +import scala.Some;
    +import scala.concurrent.Await;
    +import scala.concurrent.Awaitable;
    +import scala.concurrent.Future;
    +import scala.concurrent.duration.Duration;
    +import scala.concurrent.duration.FiniteDuration;
    +
    +import java.io.IOException;
    +import java.net.InetAddress;
    +import java.net.InetSocketAddress;
    +import java.util.ArrayList;
    +import java.util.List;
    +import java.util.concurrent.TimeUnit;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +
    +public class FlinkYarnCluster extends AbstractFlinkYarnCluster {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnCluster.class);
    +
    +	private static final int POLLING_THREAD_INTERVAL_MS = 1000;
    +
    +	private YarnClient yarnClient;
    +	private Thread actorRunner;
    +	private Thread clientShutdownHook = new ClientShutdownHook();
    +	private PollingThread pollingRunner;
    +	private Configuration hadoopConfig;
    +	// (HDFS) location of the files required to run on YARN. Needed here to delete them on shutdown.
    +	private Path sessionFilesDir;
    +	private InetSocketAddress jobManagerAddress;
    +
    +	//---------- Class internal fields -------------------
    +
    +	private ActorSystem actorSystem;
    +	private ActorRef applicationClient;
    +	private ApplicationReport intialAppReport;
    +	private static FiniteDuration akkaDuration = Duration.apply(5, TimeUnit.SECONDS);
    +	private static Timeout akkaTimeout = Timeout.durationToTimeout(akkaDuration);
    +
    +	public FlinkYarnCluster(final YarnClient yarnClient, final ApplicationId appId,
    +							Configuration hadoopConfig, Path sessionFilesDir) throws IOException, YarnException {
    +		this.yarnClient = yarnClient;
    +		this.hadoopConfig = hadoopConfig;
    +		this.sessionFilesDir = sessionFilesDir;
    +
    +		// get one application report manually
    +		intialAppReport = yarnClient.getApplicationReport(appId);
    +		String jobManagerHost = intialAppReport.getHost();
    +		int jobManagerPort = intialAppReport.getRpcPort();
    +		this.jobManagerAddress = new InetSocketAddress(jobManagerHost, jobManagerPort);
    +
    +		// start actor system
    +		LOG.info("Start actor system.");
    +		InetAddress ownHostname = NetUtils.resolveAddress(jobManagerAddress); // find name of own public interface, able to connect to the JM
    +		actorSystem = YarnUtils.createActorSystem(ownHostname.getCanonicalHostName(), 0, GlobalConfiguration.getConfiguration()); // set port automatically.
    +
    +		// start application client
    +		LOG.info("Start application client.");
    +
    +		applicationClient = actorSystem.actorOf(Props.create(ApplicationClient.class));
    +
    +		// instruct ApplicationClient to start a periodical status polling
    +		applicationClient.tell(new Messages.LocalRegisterClient(jobManagerHost + ":" + jobManagerPort), applicationClient);
    +
    +
    +		// add hook to ensure proper shutdown
    +		Runtime.getRuntime().addShutdownHook(clientShutdownHook);
    +
    +		actorRunner = new Thread(new Runnable() {
    +			@Override
    +			public void run() {
    +				// blocks until ApplicationMaster has been stopped
    +				actorSystem.awaitTermination();
    +
    +				// get final application report
    +				try {
    +					ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    +
    +					LOG.info("Application " + appId + " finished with state " + appReport
    +							.getYarnApplicationState() + " and final state " + appReport
    +							.getFinalApplicationStatus() + " at " + appReport.getFinishTime());
    +
    +					if(appReport.getYarnApplicationState() == YarnApplicationState.FAILED || appReport.getYarnApplicationState()
    +							== YarnApplicationState.KILLED	) {
    +						LOG.warn("Application failed. Diagnostics "+appReport.getDiagnostics());
    +						LOG.warn("If log aggregation is activated in the Hadoop cluster, we recommend to retrieve "
    +								+ "the full application log using this command:\n"
    +								+ "\tyarn logs -applicationId "+appReport.getApplicationId()+"\n"
    +								+ "(It sometimes takes a few seconds until the logs are aggregated)");
    +					}
    +				} catch(Exception e) {
    +					LOG.warn("Error while getting final application report", e);
    +				}
    +			}
    +		});
    +		actorRunner.setDaemon(true);
    +		actorRunner.start();
    +
    +		pollingRunner = new PollingThread(yarnClient, appId);
    +		pollingRunner.setDaemon(true);
    +		pollingRunner.start();
    +	}
    +
    +	// -------------------------- Interaction with the cluster ------------------------
    +
    +	@Override
    +	public InetSocketAddress getJobManagerAddress() {
    +		return jobManagerAddress;
    +	}
    +
    +	@Override
    +	public String getWebInterfaceURL() {
    +		return this.intialAppReport.getTrackingUrl();
    +	}
    +
    +
    +	@Override
    +	public FlinkYarnClusterStatus getClusterStatus() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		Future<Object> clusterStatusOption = ask(applicationClient, Messages.LocalGetYarnClusterStatus$.MODULE$, akkaTimeout);
    +		Object clusterStatus = awaitUtil(clusterStatusOption, "Unable to get Cluster status from Application Client");
    +		if(clusterStatus instanceof None$) {
    +			return null;
    +		} else if(clusterStatus instanceof Some) {
    +			return (FlinkYarnClusterStatus) (((Some) clusterStatus).get());
    +		} else {
    +			throw new RuntimeException("Unexpected type: "+clusterStatus.getClass().getCanonicalName());
    +		}
    +	}
    +
    +	@Override
    +	public boolean hasFailed() {
    +		if(pollingRunner == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on an uninitialized cluster." +
    +					"The system might be in an erroneous state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if(lastReport == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on a cluster. that didn't receive a status so far." +
    +					"The system might be in an erroneous state");
    +			return false;
    +		} else {
    +			return (lastReport.getYarnApplicationState() == YarnApplicationState.FAILED ||
    +					lastReport.getYarnApplicationState() == YarnApplicationState.KILLED);
    +		}
    +	}
    +
    +	@Override
    +	public String getDiagnostics() {
    +		if (!hasFailed()) {
    +			LOG.warn("getDiagnostics() called for cluster which is not in failed state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if (lastReport == null) {
    +			LOG.warn("Last report is null");
    +			return null;
    +		} else {
    +			return lastReport.getDiagnostics();
    +		}
    +	}
    +
    +	@Override
    +	public List<String> getNewMessages() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		List<String> ret = new ArrayList<String>();
    +		// get messages from ApplicationClient (locally)
    +		while(true) {
    +			Future<Object> messageOptionFuture = ask(applicationClient, Messages.LocalGetYarnMessage$.MODULE$, akkaTimeout);
    +			Object messageOption = awaitUtil(messageOptionFuture, "Error getting new messages from Appliation Client");
    +			if(messageOption instanceof None$) {
    +				break;
    +			} else if(messageOption instanceof org.apache.flink.yarn.Messages.YarnMessage) {
    +				Messages.YarnMessage msg = (Messages.YarnMessage) messageOption;
    +				ret.add("["+msg.date()+"] "+msg.message());
    +			} else {
    +				LOG.warn("LocalGetYarnMessage returned unexpected type: "+messageOption);
    +			}
    +		}
    +		return ret;
    +	}
    +
    +	private static <T> T awaitUtil(Awaitable<T> awaitable, String message) {
    +		try {
    +			return Await.result(awaitable, akkaDuration);
    +		} catch (Exception e) {
    +			throw new RuntimeException(message, e);
    +		}
    +	}
    +
    +	// -------------------------- Shutdown handling ------------------------
    +
    +	private AtomicBoolean hasBeenShutDown = new AtomicBoolean(false);
    +	@Override
    +	public void shutdown() {
    +		shutdownInternal(true);
    +	}
    +
    +	private void shutdownInternal(boolean removeShutdownHook) {
    +		if(hasBeenShutDown.getAndSet(true)) {
    +			return;
    +		}
    +		// the session is being stopped explicitly.
    +		if(removeShutdownHook) {
    +			Runtime.getRuntime().removeShutdownHook(clientShutdownHook);
    +		}
    +		if(actorSystem != null){
    +			LOG.info("Sending shutdown request to the Application Master");
    +			if(applicationClient != ActorRef.noSender()) {
    +				Future<Object> future = ask(applicationClient, new Messages.StopYarnSession(FinalApplicationStatus.SUCCEEDED), akkaTimeout);
    --- End diff --
    
    AkkaUtils.ask()


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23173072
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    --- End diff --
    
    Why GlobalConfiguration? There is at least a configuration field in this class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23169866
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/FlinkYarnSessionCli.java ---
    @@ -0,0 +1,429 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.client;
    +
    +import org.apache.commons.cli.CommandLine;
    +import org.apache.commons.cli.CommandLineParser;
    +import org.apache.commons.cli.HelpFormatter;
    +import org.apache.commons.cli.Option;
    +import org.apache.commons.cli.Options;
    +import org.apache.commons.cli.PosixParser;
    +import org.apache.commons.lang3.StringUtils;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.flink.util.InstantiationUtil;
    +import org.apache.hadoop.fs.Path;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.BufferedReader;
    +import java.io.File;
    +import java.io.FileOutputStream;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.InputStreamReader;
    +import java.io.OutputStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * Class handling the command line interface to the YARN session.
    + */
    +public class FlinkYarnSessionCli {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnSessionCli.class);
    +
    +	//------------------------------------ Constants   -------------------------
    +
    +	private static final String CONFIG_FILE_NAME = "flink-conf.yaml";
    +	public static final String CONFIG_FILE_LOGBACK_NAME = "logback.xml";
    +	public static final String CONFIG_FILE_LOG4J_NAME = "log4j.properties";
    +
    +
    +	private static final int CLIENT_POLLING_INTERVALL = 3;
    +
    +
    +	//------------------------------------ Command Line argument options -------------------------
    +	// the prefix transformation is used by the CliFrontend static constructor.
    +	private final Option QUERY;
    +	// --- or ---
    +	private final Option QUEUE;
    +	private final Option SHIP_PATH;
    +	private final Option FLINK_JAR;
    +	private final Option JM_MEMORY;
    +	private final Option TM_MEMORY;
    +	private final Option CONTAINER;
    +	private final Option SLOTS;
    +
    +	/**
    +	 * Dynamic properties allow the user to specify additional configuration values with -D, such as
    +	 *  -Dfs.overwrite-files=true  -Dtaskmanager.network.numberOfBuffers=16368
    +	 */
    +	private final Option DYNAMIC_PROPERTIES;
    +
    +	private AbstractFlinkYarnCluster yarnCluster = null;
    +
    +	public FlinkYarnSessionCli(String shortPrefix, String longPrefix) {
    +		QUERY = new Option(shortPrefix + "q", longPrefix + "query", false, "Display available YARN resources (memory, cores)");
    +		QUEUE = new Option(shortPrefix + "qu", longPrefix + "queue", true, "Specify YARN queue.");
    +		SHIP_PATH = new Option(shortPrefix + "t", longPrefix + "ship", true, "Ship files in the specified directory (t for transfer)");
    +		FLINK_JAR = new Option(shortPrefix + "j", longPrefix + "jar", true, "Path to Flink jar file");
    +		JM_MEMORY = new Option(shortPrefix + "jm", longPrefix + "jobManagerMemory", true, "Memory for JobManager Container [in MB]");
    +		TM_MEMORY = new Option(shortPrefix + "tm", longPrefix + "taskManagerMemory", true, "Memory per TaskManager Container [in MB]");
    +		CONTAINER = new Option(shortPrefix + "n", longPrefix + "container", true, "Number of YARN container to allocate (=Number of Task Managers)");
    +		SLOTS = new Option(shortPrefix + "s", longPrefix + "slots", true, "Number of slots per TaskManager");
    +		DYNAMIC_PROPERTIES = new Option(shortPrefix + "D", true, "Dynamic properties");
    +	}
    +
    +	public AbstractFlinkYarnClient createFlinkYarnClient(CommandLine cmd) {
    +
    +		AbstractFlinkYarnClient flinkYarnClient = getFlinkYarnClient();
    +		if(flinkYarnClient == null) {
    +			return null;
    +		}
    +
    +		if(!cmd.hasOption(CONTAINER.getOpt())) { // number of containers is required option!
    +			LOG.error("Missing required argument " + CONTAINER.getOpt());
    +			printUsage();
    +			return null;
    +		}
    +		flinkYarnClient.setTaskManagerCount(Integer.valueOf(cmd.getOptionValue(CONTAINER.getOpt())));
    +
    +		// Jar Path
    +		Path localJarPath;
    +		if(cmd.hasOption(FLINK_JAR.getOpt())) {
    +			String userPath = cmd.getOptionValue(FLINK_JAR.getOpt());
    +			if(!userPath.startsWith("file://")) {
    +				userPath = "file://" + userPath;
    +			}
    +			localJarPath = new Path(userPath);
    +		} else {
    +			LOG.info("No path for the flink jar passed. Using the location of "+flinkYarnClient.getClass()+" to locate the jar");
    +			localJarPath = new Path("file://"+flinkYarnClient.getClass().getProtectionDomain().getCodeSource().getLocation().getPath());
    +			if(!localJarPath.toString().contains("uberjar")) {
    +				// we need to have a proper uberjar because otherwise we don't have the required classes available on the cluster.
    +				// most likely the user did try to start yarn in a regular hadoop2 flink build (not a yarn package) (using ./bin/flink -m yarn-cluster)
    +				LOG.error("The detected jar file '"+localJarPath+"' is not a uberjar.");
    +				return null;
    +			}
    +		}
    +
    +		flinkYarnClient.setLocalJarPath(localJarPath);
    +
    +		// Conf Path
    +		String confDirPath = CliFrontend.getConfigurationDirectoryFromEnv();
    +		GlobalConfiguration.loadConfiguration(confDirPath);
    +		flinkYarnClient.setConfigurationDirectory(confDirPath);
    +		File confFile = new File(confDirPath + File.separator + CONFIG_FILE_NAME);
    +		if(!confFile.exists()) {
    +			LOG.error("Unable to locate configuration file in "+confFile);
    +			return null;
    +		}
    +		Path confPath = new Path(confFile.getAbsolutePath());
    +
    +		flinkYarnClient.setConfigurationFilePath(confPath);
    +
    +		List<File> shipFiles = new ArrayList<File>();
    +		// path to directory to ship
    +		if(cmd.hasOption(SHIP_PATH.getOpt())) {
    +			String shipPath = cmd.getOptionValue(SHIP_PATH.getOpt());
    +			File shipDir = new File(shipPath);
    +			if(shipDir.isDirectory()) {
    +				shipFiles = new ArrayList<File>(Arrays.asList(shipDir.listFiles(new FilenameFilter() {
    +					@Override
    +					public boolean accept(File dir, String name) {
    +						return !(name.equals(".") || name.equals(".."));
    +					}
    +				})));
    +			} else {
    +				LOG.warn("Ship directory is not a directory. Ignoring it.");
    +			}
    +		}
    +
    +		//check if there is a logback or log4j file
    +		if(confDirPath.length() > 0) {
    +			File logback = new File(confDirPath + File.pathSeparator + CONFIG_FILE_LOGBACK_NAME);
    +			if(logback.exists()) {
    +				shipFiles.add(logback);
    +				flinkYarnClient.setConfigurationFilePath(new Path(logback.toURI()));
    +			}
    +			File log4j = new File(confDirPath + File.pathSeparator + CONFIG_FILE_LOG4J_NAME);
    +			if(log4j.exists()) {
    +				shipFiles.add(log4j);
    +				if(flinkYarnClient.getFlinkLoggingConfigurationPath() != null) {
    +					// this means there is already a logback configuration file --> fail
    +					LOG.error("The configuration directory ('"+confDirPath+"') contains both LOG4J and Logback configuration files." +
    +							"Please delete or rename one of them.");
    +					return null;
    +				} // else
    +				flinkYarnClient.setConfigurationFilePath(new Path(log4j.toURI()));
    +			}
    +		}
    --- End diff --
    
    Can we check that the corresponding logging backend will actually be used by Flink? Because the system should rather pick the config file whose backend is present.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172158
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    --- End diff --
    
    https://briankoberlein.com/wp-content/uploads/copypasta.jpg


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23289567
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    --- End diff --
    
    The Configuration field is a hadoop config field.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23169293
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -860,8 +910,50 @@ protected Properties getYarnProperties() throws IOException {
     		return yarnProperties;
     	}
     	
    -	protected Client getClient(CommandLine line, ClassLoader classLoader) throws IOException {
    -		return new Client(getJobManagerAddress(line), getGlobalConfiguration(), classLoader);
    +	protected Client getClient(CommandLine line, ClassLoader classLoader, String programName) throws IOException {
    +		String jmAddrString = getJobManagerAddressString(line);
    +		InetSocketAddress jobManagerAddress = null;
    +		if(jmAddrString.equals(YARN_DEPLOY_JOBMANAGER)) {
    +			System.out.println("YARN cluster mode detected. Switching Log4j output to console");
    +			LogManager.getRootLogger().addAppender(new ConsoleAppender(new PatternLayout(DEFAULT_LOG4J_PATTERN_LAYOUT)));
    --- End diff --
    
    Do we have to hardwire log4j into our code? That contradicts slf4j's purpose. Would be great if we could get rid of that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172541
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    +
    +	private final static PrintStream originalStdout = System.out;
    +	private final static PrintStream originalStderr = System.err;
    +
    +
    +	// Temp directory which is deleted after the unit test.
    +	private static TemporaryFolder tmp = new TemporaryFolder();
    +
    +	protected static MiniYARNCluster yarnCluster = null;
    +
    +	protected static File flinkUberjar;
    +
    +	protected static final Configuration yarnConfiguration;
    +	static {
    +		yarnConfiguration = new YarnConfiguration();
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 512);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 4096); // 4096 is the available memory anyways
    +		yarnConfiguration.setBoolean(YarnConfiguration.YARN_MINICLUSTER_FIXED_PORTS, true);
    +		yarnConfiguration.setBoolean(YarnConfiguration.RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME, true);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_MAX_COMPLETED_APPLICATIONS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
    +		yarnConfiguration.setBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED, true);
    +		yarnConfiguration.setInt(YarnConfiguration.NM_VCORES, 666); // memory is overwritten in the MiniYARNCluster.
    +		// so we have to change the number of cores for testing.
    +	}
    +
    +	// This code is taken from: http://stackoverflow.com/a/7201825/568695
    +	// it changes the environment variables of this JVM. Use only for testing purposes!
    +	private static void setEnv(Map<String, String> newenv) {
    +		try {
    +			Class<?> processEnvironmentClass = Class.forName("java.lang.ProcessEnvironment");
    +			Field theEnvironmentField = processEnvironmentClass.getDeclaredField("theEnvironment");
    +			theEnvironmentField.setAccessible(true);
    +			Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
    +			env.putAll(newenv);
    +			Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");
    +			theCaseInsensitiveEnvironmentField.setAccessible(true);
    +			Map<String, String> cienv = (Map<String, String>) theCaseInsensitiveEnvironmentField.get(null);
    +			cienv.putAll(newenv);
    +		} catch (NoSuchFieldException e) {
    +			try {
    +				Class[] classes = Collections.class.getDeclaredClasses();
    +				Map<String, String> env = System.getenv();
    +				for (Class cl : classes) {
    +					if ("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
    +						Field field = cl.getDeclaredField("m");
    +						field.setAccessible(true);
    +						Object obj = field.get(env);
    +						Map<String, String> map = (Map<String, String>) obj;
    +						map.clear();
    +						map.putAll(newenv);
    +					}
    +				}
    +			} catch (Exception e2) {
    +				throw new RuntimeException(e2);
    +			}
    +		} catch (Exception e1) {
    +			throw new RuntimeException(e1);
    +		}
    +	}
    +
    +	/**
    +	 * Sleep a bit between the tests (we are re-using the YARN cluster for the tests)
    +	 */
    +	@After
    +	public void sleep() {
    +		try {
    +			Thread.sleep(500);
    +		} catch (InterruptedException e) {
    +			Assert.fail("Should not happen");
    +		}
    +	}
    +
    +	@Before
    +	public void checkClusterEmpty() throws IOException, YarnException {
    +		YarnClient yarnClient = YarnClient.createYarnClient();
    +		yarnClient.init(yarnConfiguration);
    +		yarnClient.start();
    +		List<ApplicationReport> apps = yarnClient.getApplications();
    +		for(ApplicationReport app : apps) {
    +			if(app.getYarnApplicationState() != YarnApplicationState.FINISHED) {
    +				Assert.fail("There is at least one application on the cluster is not finished");
    +			}
    +		}
    +	}
    +
    +	/**
    +	 * Locate a file or diretory directory
    +	 */
    +	public static File findFile(String startAt, FilenameFilter fnf) {
    +		File root = new File(startAt);
    +		String[] files = root.list();
    +		if(files == null) {
    +			return null;
    +		}
    +		for(String file : files) {
    +
    +			File f = new File(startAt + File.separator + file);
    +			if(f.isDirectory()) {
    +				File r = findFile(f.getAbsolutePath(), fnf);
    +				if(r != null) {
    +					return r;
    +				}
    +			} else if (fnf.accept(f.getParentFile(), f.getName())) {
    +				return f;
    +			}
    +
    +		}
    +		return null;
    +	}
    +
    +	/**
    +	 * Filter to find root dir of the flink-yarn dist.
    +	 */
    +	public static class RootDirFilenameFilter implements FilenameFilter {
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			return name.endsWith("yarn-uberjar.jar") && dir.toString().contains("/lib");
    +		}
    +	}
    +	public static class ContainsName implements FilenameFilter {
    +		private String name;
    +		private String excludeInPath = null;
    +
    +		public ContainsName(String name) {
    +			this.name = name;
    +		}
    +
    +		public ContainsName(String name, String excludeInPath) {
    +			this.name = name;
    +			this.excludeInPath = excludeInPath;
    +		}
    +
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			if(excludeInPath == null) {
    +				return name.contains(this.name);
    +			} else {
    +				return name.contains(this.name) && !dir.toString().contains(excludeInPath);
    +			}
    +		}
    +	}
    +
    +	public static File writeYarnSiteConfigXML(Configuration yarnConf) throws IOException {
    +		tmp.create();
    +		File yarnSiteXML = new File(tmp.newFolder().getAbsolutePath() + "/yarn-site.xml");
    +
    +		FileWriter writer = new FileWriter(yarnSiteXML);
    +		yarnConf.writeXml(writer);
    +		writer.flush();
    +		writer.close();
    +		return yarnSiteXML;
    +	}
    +
    +	public static void startYARNWithConfig(Configuration conf) {
    +		flinkUberjar = findFile(".", new RootDirFilenameFilter());
    +		Assert.assertNotNull(flinkUberjar);
    +		String flinkDistRootDir = flinkUberjar.getParentFile().getParent();
    +
    +		if (!flinkUberjar.exists()) {
    +			Assert.fail("Unable to locate yarn-uberjar.jar");
    +		}
    +
    +		try {
    +			LOG.info("Starting up MiniYARN cluster");
    +			if (yarnCluster == null) {
    +				yarnCluster = new MiniYARNCluster(YARNSessionFIFOIT.class.getName(), 2, 1, 1);
    --- End diff --
    
    Finest italian copy pasta


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23290531
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    --- End diff --
    
    Ah ok. But can we get rid of the GlobalConfiguration anyways? Sooner or later the GlobalConfiguration will be removed completely and thus it would be helpful to already start with it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23134660
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerIT.java ---
    @@ -0,0 +1,72 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler;
    +import org.junit.BeforeClass;
    +import org.junit.Test;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +
    +/**
    + * This test starts a MiniYARNCluster with a CapacityScheduler.
    + * Is has, by default a queue called "default". The configuration here adds another queue: "qa-team".
    + */
    +public class YARNSessionCapacitySchedulerIT extends YarnTestBase {
    --- End diff --
    
    Should this be callse `YARNSessionCapacitySchedulerITCase` to make it a proper integration test case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/292#issuecomment-70559819
  
    Thank you for reviewing the code. I hope I have time tomorrow to address all remarks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/292#issuecomment-70416726
  
    Thank you for the feedback. I'll address the inline comments.
    
    Regarding the questions:
    1. The separation between `flink-yarn` and `flink-yarn-tests` exists because the `flink-yarn-tests` expect the `flink-dist` package to be build before the `flink-yarn-tests` package.
    The tests are really starting a YARN cluster and deploying a flink fat-jar to it. Therefore, we need to run `flink-dist` first. I asked on the maven mailing list if there is a way of simplifying this and it seems to be possible to store the archetype descriptors (used to build the fat-jar) in a separate maven module which is then accessed by `flink-dist` and a `prepare-tests` phase of `flink-yarn`. But for that approach, we would need to create an additional maven module (something like `flink-assemblies` for making the assembly descriptor independent of the `flink-dist` package). 
    
      *tl;dr* maven is not flexible enough for a better solution.
    2. Yep. That doesn't make much sense (I used a local yarn cluster for testing it .. thats why I didn't really stumble across it). I'll fix it.
    3. This is true for the old YARN client (before this pull request). As you can see here: https://github.com/apache/flink/pull/292/files#diff-37b2363833862d636afea47fab39a694L269 I removed the code that was computing the port. This new YARN client is allocating ALL ports dynamically (web frontend, RPC). I'm using YARN to transfer the RPC port of the AM to the client.
    4. Probably not hard. Its a matter of taste I guess. I think my approach is more flexible (imagine we want to have a `-m mesos-cluster` or a `-m flink-local` at some point). Also, we would need to throw an exception if a user is setting something like `-j -m myCluster:6123`. It would be good to get some more opinions here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r22783544
  
    --- Diff: docs/yarn_setup.md ---
    @@ -173,6 +171,24 @@ You can check the number of TaskManagers in the JobManager web interface. The ad
     If the TaskManagers do not show up after a minute, you should investigate the issue using the log files.
     
     
    +## Run a single Flink job on Hadoop YARN
    +
    +The documentation above describes how to start a Flink cluster within a Hadoop YARN environment.
    +It is also possible to launch Flink within YARN only for executing a single job.
    +
    +To deploy a job to a per-job YARN cluster, set the master name to `yarn-cluster`.
    +Please note that the client then expects the `-n` value to be set (number of TaskManagers).
    --- End diff --
    
    Shouldn't it be `-yn`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23170666
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
    @@ -860,8 +910,50 @@ protected Properties getYarnProperties() throws IOException {
     		return yarnProperties;
     	}
     	
    -	protected Client getClient(CommandLine line, ClassLoader classLoader) throws IOException {
    -		return new Client(getJobManagerAddress(line), getGlobalConfiguration(), classLoader);
    +	protected Client getClient(CommandLine line, ClassLoader classLoader, String programName) throws IOException {
    +		String jmAddrString = getJobManagerAddressString(line);
    +		InetSocketAddress jobManagerAddress = null;
    +		if(jmAddrString.equals(YARN_DEPLOY_JOBMANAGER)) {
    +			System.out.println("YARN cluster mode detected. Switching Log4j output to console");
    +			LogManager.getRootLogger().addAppender(new ConsoleAppender(new PatternLayout(DEFAULT_LOG4J_PATTERN_LAYOUT)));
    --- End diff --
    
    Okay. Let me see if I can come up with a workaround.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172386
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    +
    +	private final static PrintStream originalStdout = System.out;
    +	private final static PrintStream originalStderr = System.err;
    +
    +
    +	// Temp directory which is deleted after the unit test.
    +	private static TemporaryFolder tmp = new TemporaryFolder();
    +
    +	protected static MiniYARNCluster yarnCluster = null;
    +
    +	protected static File flinkUberjar;
    +
    +	protected static final Configuration yarnConfiguration;
    +	static {
    +		yarnConfiguration = new YarnConfiguration();
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 512);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 4096); // 4096 is the available memory anyways
    +		yarnConfiguration.setBoolean(YarnConfiguration.YARN_MINICLUSTER_FIXED_PORTS, true);
    +		yarnConfiguration.setBoolean(YarnConfiguration.RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME, true);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_MAX_COMPLETED_APPLICATIONS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
    +		yarnConfiguration.setBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED, true);
    +		yarnConfiguration.setInt(YarnConfiguration.NM_VCORES, 666); // memory is overwritten in the MiniYARNCluster.
    +		// so we have to change the number of cores for testing.
    +	}
    +
    +	// This code is taken from: http://stackoverflow.com/a/7201825/568695
    +	// it changes the environment variables of this JVM. Use only for testing purposes!
    +	private static void setEnv(Map<String, String> newenv) {
    +		try {
    +			Class<?> processEnvironmentClass = Class.forName("java.lang.ProcessEnvironment");
    +			Field theEnvironmentField = processEnvironmentClass.getDeclaredField("theEnvironment");
    +			theEnvironmentField.setAccessible(true);
    +			Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
    +			env.putAll(newenv);
    +			Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");
    +			theCaseInsensitiveEnvironmentField.setAccessible(true);
    +			Map<String, String> cienv = (Map<String, String>) theCaseInsensitiveEnvironmentField.get(null);
    +			cienv.putAll(newenv);
    +		} catch (NoSuchFieldException e) {
    +			try {
    +				Class[] classes = Collections.class.getDeclaredClasses();
    +				Map<String, String> env = System.getenv();
    +				for (Class cl : classes) {
    +					if ("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
    +						Field field = cl.getDeclaredField("m");
    +						field.setAccessible(true);
    +						Object obj = field.get(env);
    +						Map<String, String> map = (Map<String, String>) obj;
    +						map.clear();
    +						map.putAll(newenv);
    +					}
    +				}
    +			} catch (Exception e2) {
    +				throw new RuntimeException(e2);
    +			}
    +		} catch (Exception e1) {
    +			throw new RuntimeException(e1);
    +		}
    +	}
    +
    +	/**
    +	 * Sleep a bit between the tests (we are re-using the YARN cluster for the tests)
    +	 */
    +	@After
    +	public void sleep() {
    +		try {
    +			Thread.sleep(500);
    +		} catch (InterruptedException e) {
    +			Assert.fail("Should not happen");
    +		}
    +	}
    +
    +	@Before
    +	public void checkClusterEmpty() throws IOException, YarnException {
    +		YarnClient yarnClient = YarnClient.createYarnClient();
    +		yarnClient.init(yarnConfiguration);
    +		yarnClient.start();
    +		List<ApplicationReport> apps = yarnClient.getApplications();
    +		for(ApplicationReport app : apps) {
    +			if(app.getYarnApplicationState() != YarnApplicationState.FINISHED) {
    +				Assert.fail("There is at least one application on the cluster is not finished");
    +			}
    +		}
    +	}
    +
    +	/**
    +	 * Locate a file or diretory directory
    +	 */
    +	public static File findFile(String startAt, FilenameFilter fnf) {
    +		File root = new File(startAt);
    +		String[] files = root.list();
    +		if(files == null) {
    +			return null;
    +		}
    +		for(String file : files) {
    +
    +			File f = new File(startAt + File.separator + file);
    +			if(f.isDirectory()) {
    +				File r = findFile(f.getAbsolutePath(), fnf);
    +				if(r != null) {
    +					return r;
    +				}
    +			} else if (fnf.accept(f.getParentFile(), f.getName())) {
    +				return f;
    +			}
    +
    +		}
    +		return null;
    +	}
    +
    +	/**
    +	 * Filter to find root dir of the flink-yarn dist.
    +	 */
    +	public static class RootDirFilenameFilter implements FilenameFilter {
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			return name.endsWith("yarn-uberjar.jar") && dir.toString().contains("/lib");
    +		}
    +	}
    +	public static class ContainsName implements FilenameFilter {
    +		private String name;
    +		private String excludeInPath = null;
    +
    +		public ContainsName(String name) {
    +			this.name = name;
    +		}
    +
    +		public ContainsName(String name, String excludeInPath) {
    +			this.name = name;
    +			this.excludeInPath = excludeInPath;
    +		}
    +
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			if(excludeInPath == null) {
    +				return name.contains(this.name);
    +			} else {
    +				return name.contains(this.name) && !dir.toString().contains(excludeInPath);
    +			}
    +		}
    +	}
    +
    +	public static File writeYarnSiteConfigXML(Configuration yarnConf) throws IOException {
    +		tmp.create();
    +		File yarnSiteXML = new File(tmp.newFolder().getAbsolutePath() + "/yarn-site.xml");
    +
    +		FileWriter writer = new FileWriter(yarnSiteXML);
    +		yarnConf.writeXml(writer);
    +		writer.flush();
    +		writer.close();
    +		return yarnSiteXML;
    +	}
    +
    +	public static void startYARNWithConfig(Configuration conf) {
    +		flinkUberjar = findFile(".", new RootDirFilenameFilter());
    +		Assert.assertNotNull(flinkUberjar);
    +		String flinkDistRootDir = flinkUberjar.getParentFile().getParent();
    +
    +		if (!flinkUberjar.exists()) {
    +			Assert.fail("Unable to locate yarn-uberjar.jar");
    +		}
    +
    +		try {
    +			LOG.info("Starting up MiniYARN cluster");
    +			if (yarnCluster == null) {
    +				yarnCluster = new MiniYARNCluster(YARNSessionFIFOIT.class.getName(), 2, 1, 1);
    +
    +				yarnCluster.init(conf);
    +				yarnCluster.start();
    +			}
    +
    +			Map<String, String> map = new HashMap<String, String>(System.getenv());
    +			File flinkConfFilePath = findFile(flinkDistRootDir, new ContainsName("flink-conf.yaml"));
    +			Assert.assertNotNull(flinkConfFilePath);
    +			map.put("FLINK_CONF_DIR", flinkConfFilePath.getParent());
    +			File yarnConfFile = writeYarnSiteConfigXML(conf);
    +			map.put("YARN_CONF_DIR", yarnConfFile.getParentFile().getAbsolutePath());
    +			setEnv(map);
    +
    +			Assert.assertTrue(yarnCluster.getServiceState() == Service.STATE.STARTED);
    +		} catch (Exception ex) {
    +			ex.printStackTrace();
    +			LOG.error("setup failure", ex);
    +			Assert.fail();
    +		}
    +	}
    +
    +	/**
    +	 * Default @BeforeClass impl. Overwrite this for passing a different configuration
    +	 */
    +	@BeforeClass
    +	public static void setup() {
    +		startYARNWithConfig(yarnConfiguration);
    +	}
    +
    +	// -------------------------- Runner -------------------------- //
    +
    +	private static ByteArrayOutputStream outContent;
    +	private static ByteArrayOutputStream errContent;
    +	enum RunTypes {
    +		YARN_SESSION, CLI_FRONTEND
    +	}
    +
    +	protected void runWithArgs(String[] args, String expect, RunTypes type) {
    +		LOG.info("Running with args "+ Arrays.toString(args));
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23173266
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    +
    +		String logbackFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		boolean hasLogback = new File(logbackFile).exists();
    +		String log4jFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +
    +		boolean hasLog4j = new File(log4jFile).exists();
    +		if(hasLogback) {
    +			shipFiles.add(new File(logbackFile));
    +		}
    +		if(hasLog4j) {
    +			shipFiles.add(new File(log4jFile));
    +		}
    +
    +		// Set up the container launch context for the application master
    +		ContainerLaunchContext amContainer = Records
    +				.newRecord(ContainerLaunchContext.class);
    +
    +		String amCommand = "$JAVA_HOME/bin/java"
    +					+ " -Xmx"+Utils.calculateHeapSize(jobManagerMemoryMb)+"M " +javaOpts;
    +
    +		if(hasLogback || hasLog4j) {
    +			amCommand += " -Dlog.file=\""+ApplicationConstants.LOG_DIR_EXPANSION_VAR +"/jobmanager-main.log\"";
    +		}
    +
    +		if(hasLogback) {
    +			amCommand += " -Dlogback.configurationFile=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		}
    +		if(hasLog4j) {
    +			amCommand += " -Dlog4j.configuration=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +		}
    +
    +		amCommand 	+= " "+ApplicationMaster.class.getName()+" "
    +					+ " 1>"
    +					+ ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stdout.log"
    +					+ " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stderr.log";
    +		amContainer.setCommands(Collections.singletonList(amCommand));
    +
    +		LOG.debug("Application Master start command: "+amCommand);
    +
    +		// intialize HDFS
    +		// Copy the application master jar to the filesystem
    +		// Create a local resource to point to the destination jar path
    +		final FileSystem fs = FileSystem.get(conf);
    +
    +		// hard coded check for the GoogleHDFS client because its not overriding the getScheme() method.
    +		if( !fs.getClass().getSimpleName().equals("GoogleHadoopFileSystem") &&
    +				fs.getScheme().startsWith("file")) {
    +			LOG.warn("The file system scheme is '" + fs.getScheme() + "'. This indicates that the "
    +					+ "specified Hadoop configuration path is wrong and the sytem is using the default Hadoop configuration values."
    +					+ "The Flink YARN client needs to store its files in a distributed file system");
    +		}
    +
    +		// Set-up ApplicationSubmissionContext for the application
    +		ApplicationSubmissionContext appContext = yarnApplication.getApplicationSubmissionContext();
    +		final ApplicationId appId = appContext.getApplicationId();
    +
    +		// Setup jar for ApplicationMaster
    +		LocalResource appMasterJar = Records.newRecord(LocalResource.class);
    +		LocalResource flinkConf = Records.newRecord(LocalResource.class);
    +		Path remotePathJar = Utils.setupLocalResource(conf, fs, appId.toString(), flinkJarPath, appMasterJar, fs.getHomeDirectory());
    +		Path remotePathConf = Utils.setupLocalResource(conf, fs, appId.toString(), flinkConfigurationPath, flinkConf, fs.getHomeDirectory());
    +		Map<String, LocalResource> localResources = new HashMap<String, LocalResource>(2);
    +		localResources.put("flink.jar", appMasterJar);
    +		localResources.put("flink-conf.yaml", flinkConf);
    +
    +
    +		// setup security tokens (code from apache storm)
    +		final Path[] paths = new Path[3 + shipFiles.size()];
    +		StringBuffer envShipFileList = new StringBuffer();
    +		// upload ship files
    +		for (int i = 0; i < shipFiles.size(); i++) {
    +			File shipFile = shipFiles.get(i);
    +			LocalResource shipResources = Records.newRecord(LocalResource.class);
    +			Path shipLocalPath = new Path("file://" + shipFile.getAbsolutePath());
    +			paths[3 + i] = Utils.setupLocalResource(conf, fs, appId.toString(),
    +					shipLocalPath, shipResources, fs.getHomeDirectory());
    +			localResources.put(shipFile.getName(), shipResources);
    +
    +			envShipFileList.append(paths[3 + i]);
    +			if(i+1 < shipFiles.size()) {
    +				envShipFileList.append(',');
    +			}
    +		}
    +
    +		paths[0] = remotePathJar;
    +		paths[1] = remotePathConf;
    +		sessionFilesDir = new Path(fs.getHomeDirectory(), ".flink/" + appId.toString() + "/");
    +		FsPermission permission = new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL);
    +		fs.setPermission(sessionFilesDir, permission); // set permission for path.
    +		Utils.setTokensFor(amContainer, paths, this.conf);
    +
    +
    +		amContainer.setLocalResources(localResources);
    +		fs.close();
    +
    +		// Setup CLASSPATH for ApplicationMaster
    +		Map<String, String> appMasterEnv = new HashMap<String, String>();
    +		Utils.setupEnv(conf, appMasterEnv);
    +		// set configuration values
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_COUNT, String.valueOf(taskManagerCount));
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_MEMORY, String.valueOf(taskManagerMemoryMb));
    +		appMasterEnv.put(FlinkYarnClient.FLINK_JAR_PATH, remotePathJar.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_APP_ID, appId.toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_HOME_DIR, fs.getHomeDirectory().toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_SHIP_FILES, envShipFileList.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_USERNAME, UserGroupInformation.getCurrentUser().getShortUserName());
    +		appMasterEnv.put(FlinkYarnClient.ENV_SLOTS, String.valueOf(slots));
    +		if(dynamicPropertiesEncoded != null) {
    +			appMasterEnv.put(FlinkYarnClient.ENV_DYNAMIC_PROPERTIES, dynamicPropertiesEncoded);
    +		}
    +
    +		amContainer.setEnvironment(appMasterEnv);
    +
    +		// Set up resource type requirements for ApplicationMaster
    +		Resource capability = Records.newRecord(Resource.class);
    +		capability.setMemory(jobManagerMemoryMb);
    +		capability.setVirtualCores(1);
    +
    +		if(clusterName == null) {
    +			clusterName = "Flink session with "+taskManagerCount+" TaskManagers";
    +		}
    +
    +		appContext.setApplicationName(clusterName); // application name
    +		appContext.setApplicationType("Apache Flink");
    +		appContext.setAMContainerSpec(amContainer);
    +		appContext.setResource(capability);
    +		appContext.setQueue(yarnQueue);
    +
    +
    +		LOG.info("Submitting application master " + appId);
    +		yarnClient.submitApplication(appContext);
    +
    +		LOG.info("Waiting for the cluster to be allocated");
    +		int waittime = 0;
    +		loop: while( true ) {
    +			ApplicationReport report = yarnClient.getApplicationReport(appId);
    +			YarnApplicationState appState = report.getYarnApplicationState();
    +			switch(appState) {
    +				case FAILED:
    +				case FINISHED:
    +				case KILLED:
    +					throw new YarnDeploymentException("The YARN application unexpectedly switched to state "
    +							+ appState +" during deployment. \n" +
    +							"Diagnostics from YARN: "+report.getDiagnostics() + "\n" +
    +							"If log aggregation is enabled on your cluster, use this command to further invesitage the issue:\n" +
    +							"yarn logs -applicationId "+appId);
    +					//break ..
    +				case RUNNING:
    +					LOG.info("YARN application has been deployed successfully.");
    +					break loop;
    +				default:
    +					LOG.info("Deploying cluster, current state "+appState);
    +					if(waittime > 60000) {
    --- End diff --
    
    Should we make this threshold configurable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23171500
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/net/NetUtils.java ---
    @@ -86,7 +86,7 @@ public static InetAddress resolveAddress(InetSocketAddress jobManagerAddress) th
     						case SLOW_CONNECT:
     							boolean correct = tryToConnect(i, jobManagerAddress, strategy.getTimeout());
     							if (correct) {
    -								LOG.info("Determined " + i + " as the TaskTracker's own IP address");
    +								LOG.info("Determined " + i + " as the machine's own IP address");
    --- End diff --
    
    Same here with the logging string.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23176494
  
    --- Diff: flink-yarn/src/main/scala/org/apache/flink/yarn/YarnTaskManager.scala ---
    @@ -31,6 +31,7 @@ trait YarnTaskManager extends ActorLogMessages {
     
       def receiveYarnMessages: Receive = {
         case StopYarnSession(status) => {
    +      log.info(s"Stopping YARN TaskManager with final application status $status")
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23173179
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    +
    +		String logbackFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		boolean hasLogback = new File(logbackFile).exists();
    +		String log4jFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    --- End diff --
    
    Is it guaranteed that the configurationDirectory string never ends with a File.separator?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23175364
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnCluster.java ---
    @@ -0,0 +1,363 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import akka.actor.ActorRef;
    +import akka.actor.ActorSystem;
    +
    +import static akka.pattern.Patterns.ask;
    +
    +import akka.actor.Props;
    +import akka.util.Timeout;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.net.NetUtils;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import scala.None$;
    +import scala.Some;
    +import scala.concurrent.Await;
    +import scala.concurrent.Awaitable;
    +import scala.concurrent.Future;
    +import scala.concurrent.duration.Duration;
    +import scala.concurrent.duration.FiniteDuration;
    +
    +import java.io.IOException;
    +import java.net.InetAddress;
    +import java.net.InetSocketAddress;
    +import java.util.ArrayList;
    +import java.util.List;
    +import java.util.concurrent.TimeUnit;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +
    +public class FlinkYarnCluster extends AbstractFlinkYarnCluster {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnCluster.class);
    +
    +	private static final int POLLING_THREAD_INTERVAL_MS = 1000;
    +
    +	private YarnClient yarnClient;
    +	private Thread actorRunner;
    +	private Thread clientShutdownHook = new ClientShutdownHook();
    +	private PollingThread pollingRunner;
    +	private Configuration hadoopConfig;
    +	// (HDFS) location of the files required to run on YARN. Needed here to delete them on shutdown.
    +	private Path sessionFilesDir;
    +	private InetSocketAddress jobManagerAddress;
    +
    +	//---------- Class internal fields -------------------
    +
    +	private ActorSystem actorSystem;
    +	private ActorRef applicationClient;
    +	private ApplicationReport intialAppReport;
    +	private static FiniteDuration akkaDuration = Duration.apply(5, TimeUnit.SECONDS);
    +	private static Timeout akkaTimeout = Timeout.durationToTimeout(akkaDuration);
    +
    +	public FlinkYarnCluster(final YarnClient yarnClient, final ApplicationId appId,
    +							Configuration hadoopConfig, Path sessionFilesDir) throws IOException, YarnException {
    +		this.yarnClient = yarnClient;
    +		this.hadoopConfig = hadoopConfig;
    +		this.sessionFilesDir = sessionFilesDir;
    +
    +		// get one application report manually
    +		intialAppReport = yarnClient.getApplicationReport(appId);
    +		String jobManagerHost = intialAppReport.getHost();
    +		int jobManagerPort = intialAppReport.getRpcPort();
    +		this.jobManagerAddress = new InetSocketAddress(jobManagerHost, jobManagerPort);
    +
    +		// start actor system
    +		LOG.info("Start actor system.");
    +		InetAddress ownHostname = NetUtils.resolveAddress(jobManagerAddress); // find name of own public interface, able to connect to the JM
    +		actorSystem = YarnUtils.createActorSystem(ownHostname.getCanonicalHostName(), 0, GlobalConfiguration.getConfiguration()); // set port automatically.
    +
    +		// start application client
    +		LOG.info("Start application client.");
    +
    +		applicationClient = actorSystem.actorOf(Props.create(ApplicationClient.class));
    +
    +		// instruct ApplicationClient to start a periodical status polling
    +		applicationClient.tell(new Messages.LocalRegisterClient(jobManagerHost + ":" + jobManagerPort), applicationClient);
    +
    +
    +		// add hook to ensure proper shutdown
    +		Runtime.getRuntime().addShutdownHook(clientShutdownHook);
    +
    +		actorRunner = new Thread(new Runnable() {
    +			@Override
    +			public void run() {
    +				// blocks until ApplicationMaster has been stopped
    +				actorSystem.awaitTermination();
    +
    +				// get final application report
    +				try {
    +					ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    +
    +					LOG.info("Application " + appId + " finished with state " + appReport
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23170694
  
    --- Diff: flink-clients/src/main/java/org/apache/flink/client/FlinkYarnSessionCli.java ---
    @@ -0,0 +1,429 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.client;
    +
    +import org.apache.commons.cli.CommandLine;
    +import org.apache.commons.cli.CommandLineParser;
    +import org.apache.commons.cli.HelpFormatter;
    +import org.apache.commons.cli.Option;
    +import org.apache.commons.cli.Options;
    +import org.apache.commons.cli.PosixParser;
    +import org.apache.commons.lang3.StringUtils;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.flink.util.InstantiationUtil;
    +import org.apache.hadoop.fs.Path;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.BufferedReader;
    +import java.io.File;
    +import java.io.FileOutputStream;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.InputStreamReader;
    +import java.io.OutputStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * Class handling the command line interface to the YARN session.
    + */
    +public class FlinkYarnSessionCli {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnSessionCli.class);
    +
    +	//------------------------------------ Constants   -------------------------
    +
    +	private static final String CONFIG_FILE_NAME = "flink-conf.yaml";
    +	public static final String CONFIG_FILE_LOGBACK_NAME = "logback.xml";
    +	public static final String CONFIG_FILE_LOG4J_NAME = "log4j.properties";
    +
    +
    +	private static final int CLIENT_POLLING_INTERVALL = 3;
    +
    +
    +	//------------------------------------ Command Line argument options -------------------------
    +	// the prefix transformation is used by the CliFrontend static constructor.
    +	private final Option QUERY;
    +	// --- or ---
    +	private final Option QUEUE;
    +	private final Option SHIP_PATH;
    +	private final Option FLINK_JAR;
    +	private final Option JM_MEMORY;
    +	private final Option TM_MEMORY;
    +	private final Option CONTAINER;
    +	private final Option SLOTS;
    +
    +	/**
    +	 * Dynamic properties allow the user to specify additional configuration values with -D, such as
    +	 *  -Dfs.overwrite-files=true  -Dtaskmanager.network.numberOfBuffers=16368
    +	 */
    +	private final Option DYNAMIC_PROPERTIES;
    +
    +	private AbstractFlinkYarnCluster yarnCluster = null;
    +
    +	public FlinkYarnSessionCli(String shortPrefix, String longPrefix) {
    +		QUERY = new Option(shortPrefix + "q", longPrefix + "query", false, "Display available YARN resources (memory, cores)");
    +		QUEUE = new Option(shortPrefix + "qu", longPrefix + "queue", true, "Specify YARN queue.");
    +		SHIP_PATH = new Option(shortPrefix + "t", longPrefix + "ship", true, "Ship files in the specified directory (t for transfer)");
    +		FLINK_JAR = new Option(shortPrefix + "j", longPrefix + "jar", true, "Path to Flink jar file");
    +		JM_MEMORY = new Option(shortPrefix + "jm", longPrefix + "jobManagerMemory", true, "Memory for JobManager Container [in MB]");
    +		TM_MEMORY = new Option(shortPrefix + "tm", longPrefix + "taskManagerMemory", true, "Memory per TaskManager Container [in MB]");
    +		CONTAINER = new Option(shortPrefix + "n", longPrefix + "container", true, "Number of YARN container to allocate (=Number of Task Managers)");
    +		SLOTS = new Option(shortPrefix + "s", longPrefix + "slots", true, "Number of slots per TaskManager");
    +		DYNAMIC_PROPERTIES = new Option(shortPrefix + "D", true, "Dynamic properties");
    +	}
    +
    +	public AbstractFlinkYarnClient createFlinkYarnClient(CommandLine cmd) {
    +
    +		AbstractFlinkYarnClient flinkYarnClient = getFlinkYarnClient();
    +		if(flinkYarnClient == null) {
    +			return null;
    +		}
    +
    +		if(!cmd.hasOption(CONTAINER.getOpt())) { // number of containers is required option!
    +			LOG.error("Missing required argument " + CONTAINER.getOpt());
    +			printUsage();
    +			return null;
    +		}
    +		flinkYarnClient.setTaskManagerCount(Integer.valueOf(cmd.getOptionValue(CONTAINER.getOpt())));
    +
    +		// Jar Path
    +		Path localJarPath;
    +		if(cmd.hasOption(FLINK_JAR.getOpt())) {
    +			String userPath = cmd.getOptionValue(FLINK_JAR.getOpt());
    +			if(!userPath.startsWith("file://")) {
    +				userPath = "file://" + userPath;
    +			}
    +			localJarPath = new Path(userPath);
    +		} else {
    +			LOG.info("No path for the flink jar passed. Using the location of "+flinkYarnClient.getClass()+" to locate the jar");
    +			localJarPath = new Path("file://"+flinkYarnClient.getClass().getProtectionDomain().getCodeSource().getLocation().getPath());
    +			if(!localJarPath.toString().contains("uberjar")) {
    +				// we need to have a proper uberjar because otherwise we don't have the required classes available on the cluster.
    +				// most likely the user did try to start yarn in a regular hadoop2 flink build (not a yarn package) (using ./bin/flink -m yarn-cluster)
    +				LOG.error("The detected jar file '"+localJarPath+"' is not a uberjar.");
    +				return null;
    +			}
    +		}
    +
    +		flinkYarnClient.setLocalJarPath(localJarPath);
    +
    +		// Conf Path
    +		String confDirPath = CliFrontend.getConfigurationDirectoryFromEnv();
    +		GlobalConfiguration.loadConfiguration(confDirPath);
    +		flinkYarnClient.setConfigurationDirectory(confDirPath);
    +		File confFile = new File(confDirPath + File.separator + CONFIG_FILE_NAME);
    +		if(!confFile.exists()) {
    +			LOG.error("Unable to locate configuration file in "+confFile);
    +			return null;
    +		}
    +		Path confPath = new Path(confFile.getAbsolutePath());
    +
    +		flinkYarnClient.setConfigurationFilePath(confPath);
    +
    +		List<File> shipFiles = new ArrayList<File>();
    +		// path to directory to ship
    +		if(cmd.hasOption(SHIP_PATH.getOpt())) {
    +			String shipPath = cmd.getOptionValue(SHIP_PATH.getOpt());
    +			File shipDir = new File(shipPath);
    +			if(shipDir.isDirectory()) {
    +				shipFiles = new ArrayList<File>(Arrays.asList(shipDir.listFiles(new FilenameFilter() {
    +					@Override
    +					public boolean accept(File dir, String name) {
    +						return !(name.equals(".") || name.equals(".."));
    +					}
    +				})));
    +			} else {
    +				LOG.warn("Ship directory is not a directory. Ignoring it.");
    +			}
    +		}
    +
    +		//check if there is a logback or log4j file
    +		if(confDirPath.length() > 0) {
    +			File logback = new File(confDirPath + File.pathSeparator + CONFIG_FILE_LOGBACK_NAME);
    +			if(logback.exists()) {
    +				shipFiles.add(logback);
    +				flinkYarnClient.setConfigurationFilePath(new Path(logback.toURI()));
    +			}
    +			File log4j = new File(confDirPath + File.pathSeparator + CONFIG_FILE_LOG4J_NAME);
    +			if(log4j.exists()) {
    +				shipFiles.add(log4j);
    +				if(flinkYarnClient.getFlinkLoggingConfigurationPath() != null) {
    +					// this means there is already a logback configuration file --> fail
    +					LOG.error("The configuration directory ('"+confDirPath+"') contains both LOG4J and Logback configuration files." +
    +							"Please delete or rename one of them.");
    +					return null;
    +				} // else
    +				flinkYarnClient.setConfigurationFilePath(new Path(log4j.toURI()));
    +			}
    +		}
    --- End diff --
    
    I agree that logging an error and cancelling the deployment is probably a bit too strict.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172012
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    --- End diff --
    
    Why retrieving a logger for YARNSessionFIFOIT.class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172490
  
    --- Diff: flink-yarn-tests/src/test/resources/log4j-test.properties ---
    @@ -0,0 +1,25 @@
    +################################################################################
    +#  Licensed to the Apache Software Foundation (ASF) under one
    +#  or more contributor license agreements.  See the NOTICE file
    +#  distributed with this work for additional information
    +#  regarding copyright ownership.  The ASF licenses this file
    +#  to you under the Apache License, Version 2.0 (the
    +#  "License"); you may not use this file except in compliance
    +#  with the License.  You may obtain a copy of the License at
    +#
    +#      http://www.apache.org/licenses/LICENSE-2.0
    +#
    +#  Unless required by applicable law or agreed to in writing, software
    +#  distributed under the License is distributed on an "AS IS" BASIS,
    +#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +#  See the License for the specific language governing permissions and
    +# limitations under the License.
    +################################################################################
    +
    +log4j.rootLogger=INFO, file
    --- End diff --
    
    Do we want to have INFO log output in the yarn tests? Isn't it a little bit too verbose?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172243
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    +
    +	private final static PrintStream originalStdout = System.out;
    +	private final static PrintStream originalStderr = System.err;
    +
    +
    +	// Temp directory which is deleted after the unit test.
    +	private static TemporaryFolder tmp = new TemporaryFolder();
    +
    +	protected static MiniYARNCluster yarnCluster = null;
    +
    +	protected static File flinkUberjar;
    +
    +	protected static final Configuration yarnConfiguration;
    +	static {
    +		yarnConfiguration = new YarnConfiguration();
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 512);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 4096); // 4096 is the available memory anyways
    +		yarnConfiguration.setBoolean(YarnConfiguration.YARN_MINICLUSTER_FIXED_PORTS, true);
    +		yarnConfiguration.setBoolean(YarnConfiguration.RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME, true);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_MAX_COMPLETED_APPLICATIONS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
    +		yarnConfiguration.setBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED, true);
    +		yarnConfiguration.setInt(YarnConfiguration.NM_VCORES, 666); // memory is overwritten in the MiniYARNCluster.
    +		// so we have to change the number of cores for testing.
    +	}
    +
    +	// This code is taken from: http://stackoverflow.com/a/7201825/568695
    +	// it changes the environment variables of this JVM. Use only for testing purposes!
    +	private static void setEnv(Map<String, String> newenv) {
    +		try {
    +			Class<?> processEnvironmentClass = Class.forName("java.lang.ProcessEnvironment");
    +			Field theEnvironmentField = processEnvironmentClass.getDeclaredField("theEnvironment");
    +			theEnvironmentField.setAccessible(true);
    +			Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
    +			env.putAll(newenv);
    +			Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");
    +			theCaseInsensitiveEnvironmentField.setAccessible(true);
    +			Map<String, String> cienv = (Map<String, String>) theCaseInsensitiveEnvironmentField.get(null);
    +			cienv.putAll(newenv);
    +		} catch (NoSuchFieldException e) {
    +			try {
    +				Class[] classes = Collections.class.getDeclaredClasses();
    +				Map<String, String> env = System.getenv();
    +				for (Class cl : classes) {
    +					if ("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
    +						Field field = cl.getDeclaredField("m");
    +						field.setAccessible(true);
    +						Object obj = field.get(env);
    +						Map<String, String> map = (Map<String, String>) obj;
    +						map.clear();
    +						map.putAll(newenv);
    +					}
    +				}
    +			} catch (Exception e2) {
    +				throw new RuntimeException(e2);
    +			}
    +		} catch (Exception e1) {
    +			throw new RuntimeException(e1);
    +		}
    +	}
    +
    +	/**
    +	 * Sleep a bit between the tests (we are re-using the YARN cluster for the tests)
    +	 */
    +	@After
    +	public void sleep() {
    +		try {
    +			Thread.sleep(500);
    +		} catch (InterruptedException e) {
    +			Assert.fail("Should not happen");
    +		}
    +	}
    +
    +	@Before
    +	public void checkClusterEmpty() throws IOException, YarnException {
    +		YarnClient yarnClient = YarnClient.createYarnClient();
    +		yarnClient.init(yarnConfiguration);
    +		yarnClient.start();
    +		List<ApplicationReport> apps = yarnClient.getApplications();
    +		for(ApplicationReport app : apps) {
    +			if(app.getYarnApplicationState() != YarnApplicationState.FINISHED) {
    +				Assert.fail("There is at least one application on the cluster is not finished");
    +			}
    +		}
    +	}
    +
    +	/**
    +	 * Locate a file or diretory directory
    +	 */
    +	public static File findFile(String startAt, FilenameFilter fnf) {
    +		File root = new File(startAt);
    +		String[] files = root.list();
    +		if(files == null) {
    +			return null;
    +		}
    +		for(String file : files) {
    +
    +			File f = new File(startAt + File.separator + file);
    +			if(f.isDirectory()) {
    +				File r = findFile(f.getAbsolutePath(), fnf);
    +				if(r != null) {
    +					return r;
    +				}
    +			} else if (fnf.accept(f.getParentFile(), f.getName())) {
    +				return f;
    +			}
    +
    +		}
    +		return null;
    +	}
    +
    +	/**
    +	 * Filter to find root dir of the flink-yarn dist.
    +	 */
    +	public static class RootDirFilenameFilter implements FilenameFilter {
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			return name.endsWith("yarn-uberjar.jar") && dir.toString().contains("/lib");
    +		}
    +	}
    +	public static class ContainsName implements FilenameFilter {
    +		private String name;
    +		private String excludeInPath = null;
    +
    +		public ContainsName(String name) {
    +			this.name = name;
    +		}
    +
    +		public ContainsName(String name, String excludeInPath) {
    +			this.name = name;
    +			this.excludeInPath = excludeInPath;
    +		}
    +
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			if(excludeInPath == null) {
    +				return name.contains(this.name);
    +			} else {
    +				return name.contains(this.name) && !dir.toString().contains(excludeInPath);
    +			}
    +		}
    +	}
    +
    +	public static File writeYarnSiteConfigXML(Configuration yarnConf) throws IOException {
    +		tmp.create();
    +		File yarnSiteXML = new File(tmp.newFolder().getAbsolutePath() + "/yarn-site.xml");
    +
    +		FileWriter writer = new FileWriter(yarnSiteXML);
    +		yarnConf.writeXml(writer);
    +		writer.flush();
    +		writer.close();
    +		return yarnSiteXML;
    +	}
    +
    +	public static void startYARNWithConfig(Configuration conf) {
    +		flinkUberjar = findFile(".", new RootDirFilenameFilter());
    +		Assert.assertNotNull(flinkUberjar);
    +		String flinkDistRootDir = flinkUberjar.getParentFile().getParent();
    +
    +		if (!flinkUberjar.exists()) {
    +			Assert.fail("Unable to locate yarn-uberjar.jar");
    +		}
    +
    +		try {
    +			LOG.info("Starting up MiniYARN cluster");
    +			if (yarnCluster == null) {
    +				yarnCluster = new MiniYARNCluster(YARNSessionFIFOIT.class.getName(), 2, 1, 1);
    --- End diff --
    
    Why YARNSessionFIFOIT.class.getName()?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23290607
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    +
    +		String logbackFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		boolean hasLogback = new File(logbackFile).exists();
    +		String log4jFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +
    +		boolean hasLog4j = new File(log4jFile).exists();
    +		if(hasLogback) {
    +			shipFiles.add(new File(logbackFile));
    +		}
    +		if(hasLog4j) {
    +			shipFiles.add(new File(log4jFile));
    +		}
    +
    +		// Set up the container launch context for the application master
    +		ContainerLaunchContext amContainer = Records
    +				.newRecord(ContainerLaunchContext.class);
    +
    +		String amCommand = "$JAVA_HOME/bin/java"
    +					+ " -Xmx"+Utils.calculateHeapSize(jobManagerMemoryMb)+"M " +javaOpts;
    +
    +		if(hasLogback || hasLog4j) {
    +			amCommand += " -Dlog.file=\""+ApplicationConstants.LOG_DIR_EXPANSION_VAR +"/jobmanager-main.log\"";
    +		}
    +
    +		if(hasLogback) {
    +			amCommand += " -Dlogback.configurationFile=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		}
    +		if(hasLog4j) {
    +			amCommand += " -Dlog4j.configuration=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +		}
    +
    +		amCommand 	+= " "+ApplicationMaster.class.getName()+" "
    +					+ " 1>"
    +					+ ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stdout.log"
    +					+ " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stderr.log";
    +		amContainer.setCommands(Collections.singletonList(amCommand));
    +
    +		LOG.debug("Application Master start command: "+amCommand);
    +
    +		// intialize HDFS
    +		// Copy the application master jar to the filesystem
    +		// Create a local resource to point to the destination jar path
    +		final FileSystem fs = FileSystem.get(conf);
    +
    +		// hard coded check for the GoogleHDFS client because its not overriding the getScheme() method.
    +		if( !fs.getClass().getSimpleName().equals("GoogleHadoopFileSystem") &&
    +				fs.getScheme().startsWith("file")) {
    +			LOG.warn("The file system scheme is '" + fs.getScheme() + "'. This indicates that the "
    +					+ "specified Hadoop configuration path is wrong and the sytem is using the default Hadoop configuration values."
    +					+ "The Flink YARN client needs to store its files in a distributed file system");
    +		}
    +
    +		// Set-up ApplicationSubmissionContext for the application
    +		ApplicationSubmissionContext appContext = yarnApplication.getApplicationSubmissionContext();
    +		final ApplicationId appId = appContext.getApplicationId();
    +
    +		// Setup jar for ApplicationMaster
    +		LocalResource appMasterJar = Records.newRecord(LocalResource.class);
    +		LocalResource flinkConf = Records.newRecord(LocalResource.class);
    +		Path remotePathJar = Utils.setupLocalResource(conf, fs, appId.toString(), flinkJarPath, appMasterJar, fs.getHomeDirectory());
    +		Path remotePathConf = Utils.setupLocalResource(conf, fs, appId.toString(), flinkConfigurationPath, flinkConf, fs.getHomeDirectory());
    +		Map<String, LocalResource> localResources = new HashMap<String, LocalResource>(2);
    +		localResources.put("flink.jar", appMasterJar);
    +		localResources.put("flink-conf.yaml", flinkConf);
    +
    +
    +		// setup security tokens (code from apache storm)
    +		final Path[] paths = new Path[3 + shipFiles.size()];
    +		StringBuffer envShipFileList = new StringBuffer();
    +		// upload ship files
    +		for (int i = 0; i < shipFiles.size(); i++) {
    +			File shipFile = shipFiles.get(i);
    +			LocalResource shipResources = Records.newRecord(LocalResource.class);
    +			Path shipLocalPath = new Path("file://" + shipFile.getAbsolutePath());
    +			paths[3 + i] = Utils.setupLocalResource(conf, fs, appId.toString(),
    +					shipLocalPath, shipResources, fs.getHomeDirectory());
    +			localResources.put(shipFile.getName(), shipResources);
    +
    +			envShipFileList.append(paths[3 + i]);
    +			if(i+1 < shipFiles.size()) {
    +				envShipFileList.append(',');
    +			}
    +		}
    +
    +		paths[0] = remotePathJar;
    +		paths[1] = remotePathConf;
    +		sessionFilesDir = new Path(fs.getHomeDirectory(), ".flink/" + appId.toString() + "/");
    +		FsPermission permission = new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL);
    +		fs.setPermission(sessionFilesDir, permission); // set permission for path.
    +		Utils.setTokensFor(amContainer, paths, this.conf);
    +
    +
    +		amContainer.setLocalResources(localResources);
    +		fs.close();
    +
    +		// Setup CLASSPATH for ApplicationMaster
    +		Map<String, String> appMasterEnv = new HashMap<String, String>();
    +		Utils.setupEnv(conf, appMasterEnv);
    +		// set configuration values
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_COUNT, String.valueOf(taskManagerCount));
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_MEMORY, String.valueOf(taskManagerMemoryMb));
    +		appMasterEnv.put(FlinkYarnClient.FLINK_JAR_PATH, remotePathJar.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_APP_ID, appId.toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_HOME_DIR, fs.getHomeDirectory().toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_SHIP_FILES, envShipFileList.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_USERNAME, UserGroupInformation.getCurrentUser().getShortUserName());
    +		appMasterEnv.put(FlinkYarnClient.ENV_SLOTS, String.valueOf(slots));
    +		if(dynamicPropertiesEncoded != null) {
    +			appMasterEnv.put(FlinkYarnClient.ENV_DYNAMIC_PROPERTIES, dynamicPropertiesEncoded);
    +		}
    +
    +		amContainer.setEnvironment(appMasterEnv);
    +
    +		// Set up resource type requirements for ApplicationMaster
    +		Resource capability = Records.newRecord(Resource.class);
    +		capability.setMemory(jobManagerMemoryMb);
    +		capability.setVirtualCores(1);
    +
    +		if(clusterName == null) {
    +			clusterName = "Flink session with "+taskManagerCount+" TaskManagers";
    +		}
    +
    +		appContext.setApplicationName(clusterName); // application name
    +		appContext.setApplicationType("Apache Flink");
    +		appContext.setAMContainerSpec(amContainer);
    +		appContext.setResource(capability);
    +		appContext.setQueue(yarnQueue);
    +
    +
    +		LOG.info("Submitting application master " + appId);
    +		yarnClient.submitApplication(appContext);
    +
    +		LOG.info("Waiting for the cluster to be allocated");
    +		int waittime = 0;
    +		loop: while( true ) {
    +			ApplicationReport report = yarnClient.getApplicationReport(appId);
    +			YarnApplicationState appState = report.getYarnApplicationState();
    +			switch(appState) {
    +				case FAILED:
    +				case FINISHED:
    +				case KILLED:
    +					throw new YarnDeploymentException("The YARN application unexpectedly switched to state "
    +							+ appState +" during deployment. \n" +
    +							"Diagnostics from YARN: "+report.getDiagnostics() + "\n" +
    +							"If log aggregation is enabled on your cluster, use this command to further invesitage the issue:\n" +
    +							"yarn logs -applicationId "+appId);
    +					//break ..
    +				case RUNNING:
    +					LOG.info("YARN application has been deployed successfully.");
    +					break loop;
    +				default:
    +					LOG.info("Deploying cluster, current state "+appState);
    +					if(waittime > 60000) {
    --- End diff --
    
    Yeah you're right. Somehow thought that something important would depend on this threshold.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger closed the pull request at:

    https://github.com/apache/flink/pull/292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23110179
  
    --- Diff: docs/yarn_setup.md ---
    @@ -173,6 +171,24 @@ You can check the number of TaskManagers in the JobManager web interface. The ad
     If the TaskManagers do not show up after a minute, you should investigate the issue using the log files.
     
     
    +## Run a single Flink job on Hadoop YARN
    +
    +The documentation above describes how to start a Flink cluster within a Hadoop YARN environment.
    +It is also possible to launch Flink within YARN only for executing a single job.
    +
    +To deploy a job to a per-job YARN cluster, set the master name to `yarn-cluster`.
    +Please note that the client then expects the `-n` value to be set (number of TaskManagers).
    --- End diff --
    
    Thank you for having a look at the PR.
    Yes, you are right. I will update the pull request and add a fix for FLINK-1385 to it as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23290018
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    +		// Create application via yarnClient
    +		yarnApplication = yarnClient.createApplication();
    +		GetNewApplicationResponse appResponse = yarnApplication.getNewApplicationResponse();
    +
    +		// ------------------ Check if the specified queue exists --------------
    +
    +		List<QueueInfo> queues = yarnClient.getAllQueues();
    +		if(queues.size() > 0) { // check only if there are queues configured.
    +			boolean queueFound = false;
    +			for (QueueInfo queue : queues) {
    +				if (queue.getQueueName().equals(this.yarnQueue)) {
    +					queueFound = true;
    +					break;
    +				}
    +			}
    +			if (!queueFound) {
    +				String queueNames = "";
    +				for (QueueInfo queue : queues) {
    +					queueNames += queue.getQueueName() + ", ";
    +				}
    +				throw new YarnDeploymentException("The specified queue '" + this.yarnQueue + "' does not exist. " +
    +						"Available queues: " + queueNames);
    +			}
    +		} else {
    +			LOG.debug("The YARN cluster does not have any queues configured");
    +		}
    +
    +		// ------------------ Check if the YARN Cluster has the requested resources --------------
    +
    +		// the yarnMinAllocationMB specifies the smallest possible container allocation size.
    +		// all allocations below this value are automatically set to this value.
    +		final int yarnMinAllocationMB = conf.getInt("yarn.scheduler.minimum-allocation-mb", 0);
    +		if(jobManagerMemoryMb < yarnMinAllocationMB || taskManagerMemoryMb < yarnMinAllocationMB) {
    +			LOG.warn("The JobManager or TaskManager memory is below the smallest possible YARN Container size. "
    +					+ "The value of 'yarn.scheduler.minimum-allocation-mb' is '"+yarnMinAllocationMB+"'. Please increase the memory size." +
    +					"YARN will allocate the smaller containers but the scheduler will account for the minimum-allocation-mb, maybe not all instances " +
    +					"you requested will start.");
    +		}
    +
    +		// set the memory to minAllocationMB to do the next checks correctly
    +		if(jobManagerMemoryMb < yarnMinAllocationMB) {
    +			jobManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +		if(taskManagerMemoryMb < yarnMinAllocationMB) {
    +			taskManagerMemoryMb =  yarnMinAllocationMB;
    +		}
    +
    +		Resource maxRes = appResponse.getMaximumResourceCapability();
    +		final String NOTE = "Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values\n";
    +		if(jobManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the JobManager available!\n"
    +					+ "Maximum Memory: "+maxRes.getMemory() + "MB Requested: "+jobManagerMemoryMb+"MB. " + NOTE);
    +		}
    +
    +		if(taskManagerMemoryMb > maxRes.getMemory() ) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The cluster does not have the requested resources for the TaskManagers available!\n"
    +					+ "Maximum Memory: " + maxRes.getMemory() + " Requested: "+taskManagerMemoryMb + "MB. " + NOTE);
    +		}
    +
    +
    +		int totalMemoryRequired = jobManagerMemoryMb + taskManagerMemoryMb * taskManagerCount;
    +		ClusterResourceDescription freeClusterMem = getCurrentFreeClusterResources(yarnClient);
    +		if(freeClusterMem.totalFreeMemory < totalMemoryRequired) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("This YARN session requires " + totalMemoryRequired + "MB of memory in the cluster. "
    +					+ "There are currently only " + freeClusterMem.totalFreeMemory+"MB available.");
    +
    +		}
    +		if( taskManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the TaskManagers ("+taskManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +		if( jobManagerMemoryMb > freeClusterMem.containerLimit) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("The requested amount of memory for the JobManager ("+jobManagerMemoryMb+"MB) is more than "
    +					+ "the largest possible YARN container: "+freeClusterMem.containerLimit);
    +		}
    +
    +		// ----------------- check if the requested containers fit into the cluster.
    +
    +		int[] nmFree = Arrays.copyOf(freeClusterMem.nodeManagersFree, freeClusterMem.nodeManagersFree.length);
    +		// first, allocate the jobManager somewhere.
    +		if(!allocateResource(nmFree, jobManagerMemoryMb)) {
    +			failSessionDuringDeployment();
    +			throw new YarnDeploymentException("Unable to find a NodeManager that can fit the JobManager/Application master. " +
    +					"The JobManager requires " + jobManagerMemoryMb + "MB. NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree));
    +		}
    +		// allocate TaskManagers
    +		for(int i = 0; i < taskManagerCount; i++) {
    +			if(!allocateResource(nmFree, taskManagerMemoryMb)) {
    +				failSessionDuringDeployment();
    +				throw new YarnDeploymentException("There is not enough memory available in the YARN cluster. " +
    +						"The TaskManager(s) require " + taskManagerMemoryMb + "MB each. " +
    +						"NodeManagers available: "+Arrays.toString(freeClusterMem.nodeManagersFree) + "\n" +
    +						"After allocating the JobManager (" + jobManagerMemoryMb + "MB) and (" + i + "/" + taskManagerCount + ") TaskManagers, " +
    +						"the following NodeManagers are available: " + Arrays.toString(nmFree) );
    +			}
    +		}
    +
    +		// ------------------ Prepare Application Master Container  ------------------------------
    +
    +		// respect custom JVM options in the YAML file
    +		final String javaOpts = GlobalConfiguration.getString(ConfigConstants.FLINK_JVM_OPTIONS, "");
    +
    +		String logbackFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		boolean hasLogback = new File(logbackFile).exists();
    +		String log4jFile = configurationDirectory + File.separator + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +
    +		boolean hasLog4j = new File(log4jFile).exists();
    +		if(hasLogback) {
    +			shipFiles.add(new File(logbackFile));
    +		}
    +		if(hasLog4j) {
    +			shipFiles.add(new File(log4jFile));
    +		}
    +
    +		// Set up the container launch context for the application master
    +		ContainerLaunchContext amContainer = Records
    +				.newRecord(ContainerLaunchContext.class);
    +
    +		String amCommand = "$JAVA_HOME/bin/java"
    +					+ " -Xmx"+Utils.calculateHeapSize(jobManagerMemoryMb)+"M " +javaOpts;
    +
    +		if(hasLogback || hasLog4j) {
    +			amCommand += " -Dlog.file=\""+ApplicationConstants.LOG_DIR_EXPANSION_VAR +"/jobmanager-main.log\"";
    +		}
    +
    +		if(hasLogback) {
    +			amCommand += " -Dlogback.configurationFile=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOGBACK_NAME;
    +		}
    +		if(hasLog4j) {
    +			amCommand += " -Dlog4j.configuration=file:" + FlinkYarnSessionCli.CONFIG_FILE_LOG4J_NAME;
    +		}
    +
    +		amCommand 	+= " "+ApplicationMaster.class.getName()+" "
    +					+ " 1>"
    +					+ ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stdout.log"
    +					+ " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager-stderr.log";
    +		amContainer.setCommands(Collections.singletonList(amCommand));
    +
    +		LOG.debug("Application Master start command: "+amCommand);
    +
    +		// intialize HDFS
    +		// Copy the application master jar to the filesystem
    +		// Create a local resource to point to the destination jar path
    +		final FileSystem fs = FileSystem.get(conf);
    +
    +		// hard coded check for the GoogleHDFS client because its not overriding the getScheme() method.
    +		if( !fs.getClass().getSimpleName().equals("GoogleHadoopFileSystem") &&
    +				fs.getScheme().startsWith("file")) {
    +			LOG.warn("The file system scheme is '" + fs.getScheme() + "'. This indicates that the "
    +					+ "specified Hadoop configuration path is wrong and the sytem is using the default Hadoop configuration values."
    +					+ "The Flink YARN client needs to store its files in a distributed file system");
    +		}
    +
    +		// Set-up ApplicationSubmissionContext for the application
    +		ApplicationSubmissionContext appContext = yarnApplication.getApplicationSubmissionContext();
    +		final ApplicationId appId = appContext.getApplicationId();
    +
    +		// Setup jar for ApplicationMaster
    +		LocalResource appMasterJar = Records.newRecord(LocalResource.class);
    +		LocalResource flinkConf = Records.newRecord(LocalResource.class);
    +		Path remotePathJar = Utils.setupLocalResource(conf, fs, appId.toString(), flinkJarPath, appMasterJar, fs.getHomeDirectory());
    +		Path remotePathConf = Utils.setupLocalResource(conf, fs, appId.toString(), flinkConfigurationPath, flinkConf, fs.getHomeDirectory());
    +		Map<String, LocalResource> localResources = new HashMap<String, LocalResource>(2);
    +		localResources.put("flink.jar", appMasterJar);
    +		localResources.put("flink-conf.yaml", flinkConf);
    +
    +
    +		// setup security tokens (code from apache storm)
    +		final Path[] paths = new Path[3 + shipFiles.size()];
    +		StringBuffer envShipFileList = new StringBuffer();
    +		// upload ship files
    +		for (int i = 0; i < shipFiles.size(); i++) {
    +			File shipFile = shipFiles.get(i);
    +			LocalResource shipResources = Records.newRecord(LocalResource.class);
    +			Path shipLocalPath = new Path("file://" + shipFile.getAbsolutePath());
    +			paths[3 + i] = Utils.setupLocalResource(conf, fs, appId.toString(),
    +					shipLocalPath, shipResources, fs.getHomeDirectory());
    +			localResources.put(shipFile.getName(), shipResources);
    +
    +			envShipFileList.append(paths[3 + i]);
    +			if(i+1 < shipFiles.size()) {
    +				envShipFileList.append(',');
    +			}
    +		}
    +
    +		paths[0] = remotePathJar;
    +		paths[1] = remotePathConf;
    +		sessionFilesDir = new Path(fs.getHomeDirectory(), ".flink/" + appId.toString() + "/");
    +		FsPermission permission = new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL);
    +		fs.setPermission(sessionFilesDir, permission); // set permission for path.
    +		Utils.setTokensFor(amContainer, paths, this.conf);
    +
    +
    +		amContainer.setLocalResources(localResources);
    +		fs.close();
    +
    +		// Setup CLASSPATH for ApplicationMaster
    +		Map<String, String> appMasterEnv = new HashMap<String, String>();
    +		Utils.setupEnv(conf, appMasterEnv);
    +		// set configuration values
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_COUNT, String.valueOf(taskManagerCount));
    +		appMasterEnv.put(FlinkYarnClient.ENV_TM_MEMORY, String.valueOf(taskManagerMemoryMb));
    +		appMasterEnv.put(FlinkYarnClient.FLINK_JAR_PATH, remotePathJar.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_APP_ID, appId.toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_HOME_DIR, fs.getHomeDirectory().toString());
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_SHIP_FILES, envShipFileList.toString() );
    +		appMasterEnv.put(FlinkYarnClient.ENV_CLIENT_USERNAME, UserGroupInformation.getCurrentUser().getShortUserName());
    +		appMasterEnv.put(FlinkYarnClient.ENV_SLOTS, String.valueOf(slots));
    +		if(dynamicPropertiesEncoded != null) {
    +			appMasterEnv.put(FlinkYarnClient.ENV_DYNAMIC_PROPERTIES, dynamicPropertiesEncoded);
    +		}
    +
    +		amContainer.setEnvironment(appMasterEnv);
    +
    +		// Set up resource type requirements for ApplicationMaster
    +		Resource capability = Records.newRecord(Resource.class);
    +		capability.setMemory(jobManagerMemoryMb);
    +		capability.setVirtualCores(1);
    +
    +		if(clusterName == null) {
    +			clusterName = "Flink session with "+taskManagerCount+" TaskManagers";
    +		}
    +
    +		appContext.setApplicationName(clusterName); // application name
    +		appContext.setApplicationType("Apache Flink");
    +		appContext.setAMContainerSpec(amContainer);
    +		appContext.setResource(capability);
    +		appContext.setQueue(yarnQueue);
    +
    +
    +		LOG.info("Submitting application master " + appId);
    +		yarnClient.submitApplication(appContext);
    +
    +		LOG.info("Waiting for the cluster to be allocated");
    +		int waittime = 0;
    +		loop: while( true ) {
    +			ApplicationReport report = yarnClient.getApplicationReport(appId);
    +			YarnApplicationState appState = report.getYarnApplicationState();
    +			switch(appState) {
    +				case FAILED:
    +				case FINISHED:
    +				case KILLED:
    +					throw new YarnDeploymentException("The YARN application unexpectedly switched to state "
    +							+ appState +" during deployment. \n" +
    +							"Diagnostics from YARN: "+report.getDiagnostics() + "\n" +
    +							"If log aggregation is enabled on your cluster, use this command to further invesitage the issue:\n" +
    +							"yarn logs -applicationId "+appId);
    +					//break ..
    +				case RUNNING:
    +					LOG.info("YARN application has been deployed successfully.");
    +					break loop;
    +				default:
    +					LOG.info("Deploying cluster, current state "+appState);
    +					if(waittime > 60000) {
    --- End diff --
    
    This threshold doesn't change the behavior of the system. its an information for users so that they can understand why deployment takes so long.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23292258
  
    --- Diff: flink-yarn/src/main/scala/org/apache/flink/yarn/ApplicationMaster.scala ---
    @@ -41,9 +41,9 @@ object ApplicationMaster{
       val MODIFIED_CONF_FILE = "flink-conf-modified.yaml"
     
       def main(args: Array[String]): Unit ={
    -    val yarnClientUsername = System.getenv(Client.ENV_CLIENT_USERNAME)
    -    LOG.info(s"YARN daemon runs as ${UserGroupInformation.getCurrentUser.getShortUserName} " +
    -      s"' setting user to execute Flink ApplicationMaster/JobManager to ${yarnClientUsername}'")
    +    val yarnClientUsername = System.getenv(FlinkYarnClient.ENV_CLIENT_USERNAME)
    +    LOG.info(s"YARN daemon runs as ${UserGroupInformation.getCurrentUser.getShortUserName}" +
    --- End diff --
    
    I leave the scala string interpolation here. The message is only logged once, we have the info level on by default and it helps improving the clearness to distinguish between the two usernames.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23176437
  
    --- Diff: flink-yarn/src/main/scala/org/apache/flink/yarn/ApplicationMaster.scala ---
    @@ -153,15 +159,24 @@ object ApplicationMaster{
         output.close()
       }
     
    -  def startJobManager(currDir: String): (ActorSystem, ActorRef) = {
    +  def startJobManager(currDir: String, hostname: String, dynamicPropertiesEncodedString: String):
    +    (ActorSystem, ActorRef) = {
         LOG.info("Start job manager for yarn")
    -    val pathToConfig = s"$currDir/$MODIFIED_CONF_FILE"
    -    val args = Array[String]("--configDir", pathToConfig)
    +    val args = Array[String]("--configDir", currDir)
     
    -    LOG.info(s"Config path: ${pathToConfig}.")
    -    val (hostname, port, configuration, _) = JobManager.parseArgs(args)
    +    LOG.info(s"Config path: ${currDir}.")
    +    val (_, _, configuration, _) = JobManager.parseArgs(args)
    +
    +    // add dynamic properties to JobManager configuration.
    +    val dynamicProperties = CliFrontend.getDynamicProperties(dynamicPropertiesEncodedString)
    +    import scala.collection.JavaConverters._
    +    for(property <- dynamicProperties.asScala){
    +      configuration.setString(property.f0, property.f1)
    +    }
    +    GlobalConfiguration.getConfiguration.addAll(configuration) // make part of globalConf.
    --- End diff --
    
    GlobalConfiguration is bad.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172407
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
    @@ -0,0 +1,379 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.CliFrontend;
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.server.MiniYARNCluster;
    +import org.junit.After;
    +import org.junit.AfterClass;
    +import org.junit.Assert;
    +import org.junit.Before;
    +import org.junit.BeforeClass;
    +import org.junit.rules.TemporaryFolder;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.FilenameFilter;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.lang.reflect.Field;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +
    +/**
    + * This base class allows to use the MiniYARNCluster.
    + * The cluster is re-used for all tests.
    + *
    + * This class is located in a different package which is build after flink-dist. This way,
    + * we can use the YARN uberjar of flink to start a Flink YARN session.
    + */
    +public abstract class YarnTestBase {
    +	private static final Logger LOG = LoggerFactory.getLogger(YARNSessionFIFOIT.class);
    +
    +	private final static PrintStream originalStdout = System.out;
    +	private final static PrintStream originalStderr = System.err;
    +
    +
    +	// Temp directory which is deleted after the unit test.
    +	private static TemporaryFolder tmp = new TemporaryFolder();
    +
    +	protected static MiniYARNCluster yarnCluster = null;
    +
    +	protected static File flinkUberjar;
    +
    +	protected static final Configuration yarnConfiguration;
    +	static {
    +		yarnConfiguration = new YarnConfiguration();
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 512);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 4096); // 4096 is the available memory anyways
    +		yarnConfiguration.setBoolean(YarnConfiguration.YARN_MINICLUSTER_FIXED_PORTS, true);
    +		yarnConfiguration.setBoolean(YarnConfiguration.RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME, true);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.RM_MAX_COMPLETED_APPLICATIONS, 2);
    +		yarnConfiguration.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
    +		yarnConfiguration.setBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED, true);
    +		yarnConfiguration.setInt(YarnConfiguration.NM_VCORES, 666); // memory is overwritten in the MiniYARNCluster.
    +		// so we have to change the number of cores for testing.
    +	}
    +
    +	// This code is taken from: http://stackoverflow.com/a/7201825/568695
    +	// it changes the environment variables of this JVM. Use only for testing purposes!
    +	private static void setEnv(Map<String, String> newenv) {
    +		try {
    +			Class<?> processEnvironmentClass = Class.forName("java.lang.ProcessEnvironment");
    +			Field theEnvironmentField = processEnvironmentClass.getDeclaredField("theEnvironment");
    +			theEnvironmentField.setAccessible(true);
    +			Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
    +			env.putAll(newenv);
    +			Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");
    +			theCaseInsensitiveEnvironmentField.setAccessible(true);
    +			Map<String, String> cienv = (Map<String, String>) theCaseInsensitiveEnvironmentField.get(null);
    +			cienv.putAll(newenv);
    +		} catch (NoSuchFieldException e) {
    +			try {
    +				Class[] classes = Collections.class.getDeclaredClasses();
    +				Map<String, String> env = System.getenv();
    +				for (Class cl : classes) {
    +					if ("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
    +						Field field = cl.getDeclaredField("m");
    +						field.setAccessible(true);
    +						Object obj = field.get(env);
    +						Map<String, String> map = (Map<String, String>) obj;
    +						map.clear();
    +						map.putAll(newenv);
    +					}
    +				}
    +			} catch (Exception e2) {
    +				throw new RuntimeException(e2);
    +			}
    +		} catch (Exception e1) {
    +			throw new RuntimeException(e1);
    +		}
    +	}
    +
    +	/**
    +	 * Sleep a bit between the tests (we are re-using the YARN cluster for the tests)
    +	 */
    +	@After
    +	public void sleep() {
    +		try {
    +			Thread.sleep(500);
    +		} catch (InterruptedException e) {
    +			Assert.fail("Should not happen");
    +		}
    +	}
    +
    +	@Before
    +	public void checkClusterEmpty() throws IOException, YarnException {
    +		YarnClient yarnClient = YarnClient.createYarnClient();
    +		yarnClient.init(yarnConfiguration);
    +		yarnClient.start();
    +		List<ApplicationReport> apps = yarnClient.getApplications();
    +		for(ApplicationReport app : apps) {
    +			if(app.getYarnApplicationState() != YarnApplicationState.FINISHED) {
    +				Assert.fail("There is at least one application on the cluster is not finished");
    +			}
    +		}
    +	}
    +
    +	/**
    +	 * Locate a file or diretory directory
    +	 */
    +	public static File findFile(String startAt, FilenameFilter fnf) {
    +		File root = new File(startAt);
    +		String[] files = root.list();
    +		if(files == null) {
    +			return null;
    +		}
    +		for(String file : files) {
    +
    +			File f = new File(startAt + File.separator + file);
    +			if(f.isDirectory()) {
    +				File r = findFile(f.getAbsolutePath(), fnf);
    +				if(r != null) {
    +					return r;
    +				}
    +			} else if (fnf.accept(f.getParentFile(), f.getName())) {
    +				return f;
    +			}
    +
    +		}
    +		return null;
    +	}
    +
    +	/**
    +	 * Filter to find root dir of the flink-yarn dist.
    +	 */
    +	public static class RootDirFilenameFilter implements FilenameFilter {
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			return name.endsWith("yarn-uberjar.jar") && dir.toString().contains("/lib");
    +		}
    +	}
    +	public static class ContainsName implements FilenameFilter {
    +		private String name;
    +		private String excludeInPath = null;
    +
    +		public ContainsName(String name) {
    +			this.name = name;
    +		}
    +
    +		public ContainsName(String name, String excludeInPath) {
    +			this.name = name;
    +			this.excludeInPath = excludeInPath;
    +		}
    +
    +		@Override
    +		public boolean accept(File dir, String name) {
    +			if(excludeInPath == null) {
    +				return name.contains(this.name);
    +			} else {
    +				return name.contains(this.name) && !dir.toString().contains(excludeInPath);
    +			}
    +		}
    +	}
    +
    +	public static File writeYarnSiteConfigXML(Configuration yarnConf) throws IOException {
    +		tmp.create();
    +		File yarnSiteXML = new File(tmp.newFolder().getAbsolutePath() + "/yarn-site.xml");
    +
    +		FileWriter writer = new FileWriter(yarnSiteXML);
    +		yarnConf.writeXml(writer);
    +		writer.flush();
    +		writer.close();
    +		return yarnSiteXML;
    +	}
    +
    +	public static void startYARNWithConfig(Configuration conf) {
    +		flinkUberjar = findFile(".", new RootDirFilenameFilter());
    +		Assert.assertNotNull(flinkUberjar);
    +		String flinkDistRootDir = flinkUberjar.getParentFile().getParent();
    +
    +		if (!flinkUberjar.exists()) {
    +			Assert.fail("Unable to locate yarn-uberjar.jar");
    +		}
    +
    +		try {
    +			LOG.info("Starting up MiniYARN cluster");
    +			if (yarnCluster == null) {
    +				yarnCluster = new MiniYARNCluster(YARNSessionFIFOIT.class.getName(), 2, 1, 1);
    +
    +				yarnCluster.init(conf);
    +				yarnCluster.start();
    +			}
    +
    +			Map<String, String> map = new HashMap<String, String>(System.getenv());
    +			File flinkConfFilePath = findFile(flinkDistRootDir, new ContainsName("flink-conf.yaml"));
    +			Assert.assertNotNull(flinkConfFilePath);
    +			map.put("FLINK_CONF_DIR", flinkConfFilePath.getParent());
    +			File yarnConfFile = writeYarnSiteConfigXML(conf);
    +			map.put("YARN_CONF_DIR", yarnConfFile.getParentFile().getAbsolutePath());
    +			setEnv(map);
    +
    +			Assert.assertTrue(yarnCluster.getServiceState() == Service.STATE.STARTED);
    +		} catch (Exception ex) {
    +			ex.printStackTrace();
    +			LOG.error("setup failure", ex);
    +			Assert.fail();
    +		}
    +	}
    +
    +	/**
    +	 * Default @BeforeClass impl. Overwrite this for passing a different configuration
    +	 */
    +	@BeforeClass
    +	public static void setup() {
    +		startYARNWithConfig(yarnConfiguration);
    +	}
    +
    +	// -------------------------- Runner -------------------------- //
    +
    +	private static ByteArrayOutputStream outContent;
    +	private static ByteArrayOutputStream errContent;
    +	enum RunTypes {
    +		YARN_SESSION, CLI_FRONTEND
    +	}
    +
    +	protected void runWithArgs(String[] args, String expect, RunTypes type) {
    +		LOG.info("Running with args "+ Arrays.toString(args));
    +
    +		outContent = new ByteArrayOutputStream();
    +		errContent = new ByteArrayOutputStream();
    +		System.setOut(new PrintStream(outContent));
    +		System.setErr(new PrintStream(errContent));
    +
    +
    +		final int START_TIMEOUT_SECONDS = 60;
    +
    +		Runner runner = new Runner(args, type);
    +		runner.start();
    +
    +		boolean expectedStringSeen = false;
    +		for(int second = 0; second <  START_TIMEOUT_SECONDS; second++) {
    +			try {
    +				Thread.sleep(1000);
    +			} catch (InterruptedException e) {
    +				Assert.fail("Interruption not expected");
    +			}
    +			// check output for correct TaskManager startup.
    +			if(outContent.toString().contains(expect)
    +					|| errContent.toString().contains(expect) ) {
    +				expectedStringSeen = true;
    +				LOG.info("Found expected output in redirected streams");
    +				// send "stop" command to command line interface
    +				runner.sendStop();
    +				// wait for the thread to stop
    +				try {
    +					runner.join(1000);
    +				} catch (InterruptedException e) {
    +					LOG.warn("Interrupted while stopping runner", e);
    +				}
    +				LOG.warn("stopped");
    +				break;
    +			}
    +			// check if thread died
    +			if(!runner.isAlive()) {
    +				sendOutput();
    +				Assert.fail("Runner thread died before the test was finished. Return value = "+runner.getReturnValue());
    +			}
    +		}
    +
    +		sendOutput();
    +		Assert.assertTrue("During the timeout period of " + START_TIMEOUT_SECONDS + " seconds the " +
    +				"expected string did not show up", expectedStringSeen);
    +		LOG.info("Test was successful");
    +	}
    +
    +	private static void sendOutput() {
    +		System.setOut(originalStdout);
    +		System.setErr(originalStderr);
    +
    +		LOG.info("Sending stdout content through logger: \n\n"+outContent.toString()+"\n\n");
    +		LOG.info("Sending stderr content through logger: \n\n"+errContent.toString()+"\n\n");
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/292#issuecomment-70536571
  
    As far as I can tell without running tests on the cluster it looks good to me. Only some minor remarks as in-code comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/292#issuecomment-70924255
  
    Thank you for reviewing the pull request. I'll merge it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172930
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    +		}
    +		conf = Utils.initializeYarnConfiguration();
    +		if(this.yarnClient == null) {
    +			// Create yarnClient
    +			yarnClient = YarnClient.createYarnClient();
    +			yarnClient.init(conf);
    +			yarnClient.start();
    +		}
    +	}
    +
    +	@Override
    +	public void setJobManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_JM_MEMORY) {
    +			throw new IllegalArgumentException("The JobManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_JM_MEMORY+" MB");
    +		}
    +		this.jobManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerMemory(int memoryMb) {
    +		if(memoryMb < MIN_TM_MEMORY) {
    +			throw new IllegalArgumentException("The TaskManager memory is below the minimum required memory amount "
    +					+ "of "+MIN_TM_MEMORY+" MB");
    +		}
    +		this.taskManagerMemoryMb = memoryMb;
    +	}
    +
    +	@Override
    +	public void setTaskManagerSlots(int slots) {
    +		if(slots <= 0) {
    +			throw new IllegalArgumentException("Number of TaskManager slots must be positive");
    +		}
    +		this.slots = slots;
    +	}
    +
    +	@Override
    +	public int getTaskManagerSlots() {
    +		return this.slots;
    +	}
    +
    +	@Override
    +	public void setQueue(String queue) {
    +		this.yarnQueue = queue;
    +	}
    +
    +	@Override
    +	public void setLocalJarPath(Path localJarPath) {
    +		if(!localJarPath.toString().endsWith("jar")) {
    +			throw new IllegalArgumentException("The passed jar path ('"+localJarPath+"') does not end with the 'jar' extension");
    +		}
    +		this.flinkJarPath = localJarPath;
    +	}
    +
    +	@Override
    +	public void setConfigurationFilePath(Path confPath) {
    +		flinkConfigurationPath = confPath;
    +	}
    +
    +	public void setConfigurationDirectory(String configurationDirectory) {
    +		this.configurationDirectory = configurationDirectory;
    +	}
    +
    +	@Override
    +	public void setFlinkLoggingConfigurationPath(Path logConfPath) {
    +		flinkLoggingConfigurationPath = logConfPath;
    +	}
    +
    +	@Override
    +	public Path getFlinkLoggingConfigurationPath() {
    +		return flinkLoggingConfigurationPath;
    +	}
    +
    +	@Override
    +	public void setTaskManagerCount(int tmCount) {
    +		if(tmCount < 1) {
    +			throw new IllegalArgumentException("The TaskManager count has to be at least 1.");
    +		}
    +		this.taskManagerCount = tmCount;
    +	}
    +
    +	@Override
    +	public int getTaskManagerCount() {
    +		return this.taskManagerCount;
    +	}
    +
    +	@Override
    +	public void setShipFiles(List<File> shipFiles) {
    +		this.shipFiles.addAll(shipFiles);
    +	}
    +
    +	public void setDynamicPropertiesEncoded(String dynamicPropertiesEncoded) {
    +		this.dynamicPropertiesEncoded = dynamicPropertiesEncoded;
    +	}
    +
    +	@Override
    +	public String getDynamicPropertiesEncoded() {
    +		return this.dynamicPropertiesEncoded;
    +	}
    +
    +
    +	public void isReadyForDepoyment() throws YarnDeploymentException {
    +		if(taskManagerCount <= 0) {
    +			throw new YarnDeploymentException("Taskmanager count must be positive");
    +		}
    +		if(this.flinkJarPath == null) {
    +			throw new YarnDeploymentException("The Flink jar path is null");
    +		}
    +		if(this.configurationDirectory == null) {
    +			throw new YarnDeploymentException("Configuration directory not set");
    +		}
    +		if(this.flinkConfigurationPath == null) {
    +			throw new YarnDeploymentException("Configuration path not set");
    +		}
    +
    +	}
    +
    +	public static boolean allocateResource(int[] nodeManagers, int toAllocate) {
    +		for(int i = 0; i < nodeManagers.length; i++) {
    +			if(nodeManagers[i] >= toAllocate) {
    +				nodeManagers[i] -= toAllocate;
    +				return true;
    +			}
    +		}
    +		return false;
    +	}
    +
    +	/**
    +	 * This method will block until the ApplicationMaster/JobManager have been
    +	 * deployed on YARN.
    +	 */
    +	@Override
    +	public AbstractFlinkYarnCluster deploy(String clusterName) throws Exception {
    +		isReadyForDepoyment();
    +
    +		LOG.info("Using values:");
    +		LOG.info("\tTaskManager count = " + taskManagerCount);
    +		LOG.info("\tJobManager memory = " + jobManagerMemoryMb);
    +		LOG.info("\tTaskManager memory = " + taskManagerMemoryMb);
    +
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23176016
  
    --- Diff: flink-yarn/src/main/scala/org/apache/flink/yarn/ApplicationMaster.scala ---
    @@ -41,9 +41,9 @@ object ApplicationMaster{
       val MODIFIED_CONF_FILE = "flink-conf-modified.yaml"
     
       def main(args: Array[String]): Unit ={
    -    val yarnClientUsername = System.getenv(Client.ENV_CLIENT_USERNAME)
    -    LOG.info(s"YARN daemon runs as ${UserGroupInformation.getCurrentUser.getShortUserName} " +
    -      s"' setting user to execute Flink ApplicationMaster/JobManager to ${yarnClientUsername}'")
    +    val yarnClientUsername = System.getenv(FlinkYarnClient.ENV_CLIENT_USERNAME)
    +    LOG.info(s"YARN daemon runs as ${UserGroupInformation.getCurrentUser.getShortUserName}" +
    --- End diff --
    
    {}-placeholder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23172849
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnClient.java ---
    @@ -0,0 +1,653 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import java.io.ByteArrayOutputStream;
    +import java.io.File;
    +import java.io.IOException;
    +import java.io.PrintStream;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import org.apache.flink.configuration.ConfigConstants;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.fs.permission.FsAction;
    +import org.apache.hadoop.fs.permission.FsPermission;
    +import org.apache.hadoop.security.UserGroupInformation;
    +import org.apache.hadoop.yarn.api.ApplicationConstants;
    +import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
    +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
    +import org.apache.hadoop.yarn.api.records.LocalResource;
    +import org.apache.hadoop.yarn.api.records.NodeReport;
    +import org.apache.hadoop.yarn.api.records.NodeState;
    +import org.apache.hadoop.yarn.api.records.QueueInfo;
    +import org.apache.hadoop.yarn.api.records.Resource;
    +import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.client.api.YarnClientApplication;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.apache.hadoop.yarn.util.Records;
    +
    +/**
    + * All classes in this package contain code taken from
    + * https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java?source=cc
    + * and
    + * https://github.com/hortonworks/simple-yarn-app
    + * and
    + * https://github.com/yahoo/storm-yarn/blob/master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
    + *
    + * The Flink jar is uploaded to HDFS by this client.
    + * The application master and all the TaskManager containers get the jar file downloaded
    + * by YARN into their local fs.
    + *
    + */
    +public class FlinkYarnClient extends AbstractFlinkYarnClient {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnClient.class);
    +
    +	/**
    +	 * Constants,
    +	 * all starting with ENV_ are used as environment variables to pass values from the Client
    +	 * to the Application Master.
    +	 */
    +	public final static String ENV_TM_MEMORY = "_CLIENT_TM_MEMORY";
    +	public final static String ENV_TM_COUNT = "_CLIENT_TM_COUNT";
    +	public final static String ENV_APP_ID = "_APP_ID";
    +	public final static String FLINK_JAR_PATH = "_FLINK_JAR_PATH"; // the Flink jar resource location (in HDFS).
    +	public static final String ENV_CLIENT_HOME_DIR = "_CLIENT_HOME_DIR";
    +	public static final String ENV_CLIENT_SHIP_FILES = "_CLIENT_SHIP_FILES";
    +	public static final String ENV_CLIENT_USERNAME = "_CLIENT_USERNAME";
    +	public static final String ENV_SLOTS = "_SLOTS";
    +	public static final String ENV_DYNAMIC_PROPERTIES = "_DYNAMIC_PROPERTIES";
    +
    +	private static final String DEFAULT_QUEUE_NAME = "default";
    +
    +
    +	/**
    +	 * Minimum memory requirements, checked by the Client.
    +	 */
    +	private static final int MIN_JM_MEMORY = 128;
    +	private static final int MIN_TM_MEMORY = 128;
    +
    +	private Configuration conf;
    +	private YarnClient yarnClient;
    +	private YarnClientApplication yarnApplication;
    +
    +
    +	/**
    +	 * Files (usually in a distributed file system) used for the YARN session of Flink.
    +	 * Contains configuration files and jar files.
    +	 */
    +	private Path sessionFilesDir;
    +
    +	/**
    +	 * If the user has specified a different number of slots, we store them here
    +	 */
    +	private int slots = -1;
    +
    +	private int jobManagerMemoryMb = 512;
    +
    +	private int taskManagerMemoryMb = 512;
    +
    +	private int taskManagerCount = 1;
    +
    +	private String yarnQueue = DEFAULT_QUEUE_NAME;
    +
    +	private String configurationDirectory;
    +
    +	private Path flinkConfigurationPath;
    +
    +	private Path flinkLoggingConfigurationPath; // optional
    +
    +	private Path flinkJarPath;
    +
    +	private String dynamicPropertiesEncoded;
    +
    +	private List<File> shipFiles = new ArrayList<File>();
    +
    +
    +	public FlinkYarnClient() {
    +		// Check if security is enabled
    +		if(UserGroupInformation.isSecurityEnabled()) {
    +			throw new RuntimeException("Flink YARN client does not have security support right now."
    +					+ "File a bug, we will fix it asap");
    --- End diff --
    
    You think so? ;-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23175486
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnCluster.java ---
    @@ -0,0 +1,363 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import akka.actor.ActorRef;
    +import akka.actor.ActorSystem;
    +
    +import static akka.pattern.Patterns.ask;
    +
    +import akka.actor.Props;
    +import akka.util.Timeout;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.net.NetUtils;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import scala.None$;
    +import scala.Some;
    +import scala.concurrent.Await;
    +import scala.concurrent.Awaitable;
    +import scala.concurrent.Future;
    +import scala.concurrent.duration.Duration;
    +import scala.concurrent.duration.FiniteDuration;
    +
    +import java.io.IOException;
    +import java.net.InetAddress;
    +import java.net.InetSocketAddress;
    +import java.util.ArrayList;
    +import java.util.List;
    +import java.util.concurrent.TimeUnit;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +
    +public class FlinkYarnCluster extends AbstractFlinkYarnCluster {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnCluster.class);
    +
    +	private static final int POLLING_THREAD_INTERVAL_MS = 1000;
    +
    +	private YarnClient yarnClient;
    +	private Thread actorRunner;
    +	private Thread clientShutdownHook = new ClientShutdownHook();
    +	private PollingThread pollingRunner;
    +	private Configuration hadoopConfig;
    +	// (HDFS) location of the files required to run on YARN. Needed here to delete them on shutdown.
    +	private Path sessionFilesDir;
    +	private InetSocketAddress jobManagerAddress;
    +
    +	//---------- Class internal fields -------------------
    +
    +	private ActorSystem actorSystem;
    +	private ActorRef applicationClient;
    +	private ApplicationReport intialAppReport;
    +	private static FiniteDuration akkaDuration = Duration.apply(5, TimeUnit.SECONDS);
    +	private static Timeout akkaTimeout = Timeout.durationToTimeout(akkaDuration);
    +
    +	public FlinkYarnCluster(final YarnClient yarnClient, final ApplicationId appId,
    +							Configuration hadoopConfig, Path sessionFilesDir) throws IOException, YarnException {
    +		this.yarnClient = yarnClient;
    +		this.hadoopConfig = hadoopConfig;
    +		this.sessionFilesDir = sessionFilesDir;
    +
    +		// get one application report manually
    +		intialAppReport = yarnClient.getApplicationReport(appId);
    +		String jobManagerHost = intialAppReport.getHost();
    +		int jobManagerPort = intialAppReport.getRpcPort();
    +		this.jobManagerAddress = new InetSocketAddress(jobManagerHost, jobManagerPort);
    +
    +		// start actor system
    +		LOG.info("Start actor system.");
    +		InetAddress ownHostname = NetUtils.resolveAddress(jobManagerAddress); // find name of own public interface, able to connect to the JM
    +		actorSystem = YarnUtils.createActorSystem(ownHostname.getCanonicalHostName(), 0, GlobalConfiguration.getConfiguration()); // set port automatically.
    +
    +		// start application client
    +		LOG.info("Start application client.");
    +
    +		applicationClient = actorSystem.actorOf(Props.create(ApplicationClient.class));
    +
    +		// instruct ApplicationClient to start a periodical status polling
    +		applicationClient.tell(new Messages.LocalRegisterClient(jobManagerHost + ":" + jobManagerPort), applicationClient);
    +
    +
    +		// add hook to ensure proper shutdown
    +		Runtime.getRuntime().addShutdownHook(clientShutdownHook);
    +
    +		actorRunner = new Thread(new Runnable() {
    +			@Override
    +			public void run() {
    +				// blocks until ApplicationMaster has been stopped
    +				actorSystem.awaitTermination();
    +
    +				// get final application report
    +				try {
    +					ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    +
    +					LOG.info("Application " + appId + " finished with state " + appReport
    +							.getYarnApplicationState() + " and final state " + appReport
    +							.getFinalApplicationStatus() + " at " + appReport.getFinishTime());
    +
    +					if(appReport.getYarnApplicationState() == YarnApplicationState.FAILED || appReport.getYarnApplicationState()
    +							== YarnApplicationState.KILLED	) {
    +						LOG.warn("Application failed. Diagnostics "+appReport.getDiagnostics());
    +						LOG.warn("If log aggregation is activated in the Hadoop cluster, we recommend to retrieve "
    +								+ "the full application log using this command:\n"
    +								+ "\tyarn logs -applicationId "+appReport.getApplicationId()+"\n"
    +								+ "(It sometimes takes a few seconds until the logs are aggregated)");
    +					}
    +				} catch(Exception e) {
    +					LOG.warn("Error while getting final application report", e);
    +				}
    +			}
    +		});
    +		actorRunner.setDaemon(true);
    +		actorRunner.start();
    +
    +		pollingRunner = new PollingThread(yarnClient, appId);
    +		pollingRunner.setDaemon(true);
    +		pollingRunner.start();
    +	}
    +
    +	// -------------------------- Interaction with the cluster ------------------------
    +
    +	@Override
    +	public InetSocketAddress getJobManagerAddress() {
    +		return jobManagerAddress;
    +	}
    +
    +	@Override
    +	public String getWebInterfaceURL() {
    +		return this.intialAppReport.getTrackingUrl();
    +	}
    +
    +
    +	@Override
    +	public FlinkYarnClusterStatus getClusterStatus() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		Future<Object> clusterStatusOption = ask(applicationClient, Messages.LocalGetYarnClusterStatus$.MODULE$, akkaTimeout);
    +		Object clusterStatus = awaitUtil(clusterStatusOption, "Unable to get Cluster status from Application Client");
    +		if(clusterStatus instanceof None$) {
    +			return null;
    +		} else if(clusterStatus instanceof Some) {
    +			return (FlinkYarnClusterStatus) (((Some) clusterStatus).get());
    +		} else {
    +			throw new RuntimeException("Unexpected type: "+clusterStatus.getClass().getCanonicalName());
    +		}
    +	}
    +
    +	@Override
    +	public boolean hasFailed() {
    +		if(pollingRunner == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on an uninitialized cluster." +
    +					"The system might be in an erroneous state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if(lastReport == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on a cluster. that didn't receive a status so far." +
    +					"The system might be in an erroneous state");
    +			return false;
    +		} else {
    +			return (lastReport.getYarnApplicationState() == YarnApplicationState.FAILED ||
    +					lastReport.getYarnApplicationState() == YarnApplicationState.KILLED);
    +		}
    +	}
    +
    +	@Override
    +	public String getDiagnostics() {
    +		if (!hasFailed()) {
    +			LOG.warn("getDiagnostics() called for cluster which is not in failed state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if (lastReport == null) {
    +			LOG.warn("Last report is null");
    +			return null;
    +		} else {
    +			return lastReport.getDiagnostics();
    +		}
    +	}
    +
    +	@Override
    +	public List<String> getNewMessages() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		List<String> ret = new ArrayList<String>();
    +		// get messages from ApplicationClient (locally)
    +		while(true) {
    +			Future<Object> messageOptionFuture = ask(applicationClient, Messages.LocalGetYarnMessage$.MODULE$, akkaTimeout);
    --- End diff --
    
    For Eclipse user it is better to define a method which returns the case object.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23134668
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOIT.java ---
    @@ -0,0 +1,225 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import org.apache.flink.client.FlinkYarnSessionCli;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnClient;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.yarn.conf.YarnConfiguration;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
    +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler;
    +import org.junit.Assert;
    +import org.junit.BeforeClass;
    +import org.junit.Test;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.File;
    +
    +
    +/**
    + * This test starts a MiniYARNCluster with a FIFO scheudler.
    + * There are no queues for that scheduler.
    + */
    +public class YARNSessionFIFOIT extends YarnTestBase {
    --- End diff --
    
    `YARNSessionFIFOITCase` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23171462
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/net/NetUtils.java ---
    @@ -76,7 +76,7 @@ public static InetAddress resolveAddress(InetSocketAddress jobManagerAddress) th
     						case ADDRESS:
     							if (hasCommonPrefix(jobManagerAddress.getAddress().getAddress(), i.getAddress())) {
     								if (tryToConnect(i, jobManagerAddress, strategy.getTimeout())) {
    -									LOG.info("Determined " + i + " as the TaskTracker's own IP address");
    +									LOG.info("Determined " + i + " as the machine's own IP address");
    --- End diff --
    
    Better to use Log.info("Determined {} as the machine's own IP address.", i). That way, the string will only be created if the log level is info or higher.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1295][FLINK-883] Allow to deploy 'job o...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/292#discussion_r23175580
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/FlinkYarnCluster.java ---
    @@ -0,0 +1,363 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.yarn;
    +
    +import akka.actor.ActorRef;
    +import akka.actor.ActorSystem;
    +
    +import static akka.pattern.Patterns.ask;
    +
    +import akka.actor.Props;
    +import akka.util.Timeout;
    +import org.apache.flink.configuration.GlobalConfiguration;
    +import org.apache.flink.runtime.net.NetUtils;
    +import org.apache.flink.runtime.yarn.AbstractFlinkYarnCluster;
    +import org.apache.flink.runtime.yarn.FlinkYarnClusterStatus;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.fs.FileSystem;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.hadoop.service.Service;
    +import org.apache.hadoop.yarn.api.records.ApplicationId;
    +import org.apache.hadoop.yarn.api.records.ApplicationReport;
    +import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
    +import org.apache.hadoop.yarn.api.records.YarnApplicationState;
    +import org.apache.hadoop.yarn.client.api.YarnClient;
    +import org.apache.hadoop.yarn.exceptions.YarnException;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import scala.None$;
    +import scala.Some;
    +import scala.concurrent.Await;
    +import scala.concurrent.Awaitable;
    +import scala.concurrent.Future;
    +import scala.concurrent.duration.Duration;
    +import scala.concurrent.duration.FiniteDuration;
    +
    +import java.io.IOException;
    +import java.net.InetAddress;
    +import java.net.InetSocketAddress;
    +import java.util.ArrayList;
    +import java.util.List;
    +import java.util.concurrent.TimeUnit;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +
    +public class FlinkYarnCluster extends AbstractFlinkYarnCluster {
    +	private static final Logger LOG = LoggerFactory.getLogger(FlinkYarnCluster.class);
    +
    +	private static final int POLLING_THREAD_INTERVAL_MS = 1000;
    +
    +	private YarnClient yarnClient;
    +	private Thread actorRunner;
    +	private Thread clientShutdownHook = new ClientShutdownHook();
    +	private PollingThread pollingRunner;
    +	private Configuration hadoopConfig;
    +	// (HDFS) location of the files required to run on YARN. Needed here to delete them on shutdown.
    +	private Path sessionFilesDir;
    +	private InetSocketAddress jobManagerAddress;
    +
    +	//---------- Class internal fields -------------------
    +
    +	private ActorSystem actorSystem;
    +	private ActorRef applicationClient;
    +	private ApplicationReport intialAppReport;
    +	private static FiniteDuration akkaDuration = Duration.apply(5, TimeUnit.SECONDS);
    +	private static Timeout akkaTimeout = Timeout.durationToTimeout(akkaDuration);
    +
    +	public FlinkYarnCluster(final YarnClient yarnClient, final ApplicationId appId,
    +							Configuration hadoopConfig, Path sessionFilesDir) throws IOException, YarnException {
    +		this.yarnClient = yarnClient;
    +		this.hadoopConfig = hadoopConfig;
    +		this.sessionFilesDir = sessionFilesDir;
    +
    +		// get one application report manually
    +		intialAppReport = yarnClient.getApplicationReport(appId);
    +		String jobManagerHost = intialAppReport.getHost();
    +		int jobManagerPort = intialAppReport.getRpcPort();
    +		this.jobManagerAddress = new InetSocketAddress(jobManagerHost, jobManagerPort);
    +
    +		// start actor system
    +		LOG.info("Start actor system.");
    +		InetAddress ownHostname = NetUtils.resolveAddress(jobManagerAddress); // find name of own public interface, able to connect to the JM
    +		actorSystem = YarnUtils.createActorSystem(ownHostname.getCanonicalHostName(), 0, GlobalConfiguration.getConfiguration()); // set port automatically.
    +
    +		// start application client
    +		LOG.info("Start application client.");
    +
    +		applicationClient = actorSystem.actorOf(Props.create(ApplicationClient.class));
    +
    +		// instruct ApplicationClient to start a periodical status polling
    +		applicationClient.tell(new Messages.LocalRegisterClient(jobManagerHost + ":" + jobManagerPort), applicationClient);
    +
    +
    +		// add hook to ensure proper shutdown
    +		Runtime.getRuntime().addShutdownHook(clientShutdownHook);
    +
    +		actorRunner = new Thread(new Runnable() {
    +			@Override
    +			public void run() {
    +				// blocks until ApplicationMaster has been stopped
    +				actorSystem.awaitTermination();
    +
    +				// get final application report
    +				try {
    +					ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    +
    +					LOG.info("Application " + appId + " finished with state " + appReport
    +							.getYarnApplicationState() + " and final state " + appReport
    +							.getFinalApplicationStatus() + " at " + appReport.getFinishTime());
    +
    +					if(appReport.getYarnApplicationState() == YarnApplicationState.FAILED || appReport.getYarnApplicationState()
    +							== YarnApplicationState.KILLED	) {
    +						LOG.warn("Application failed. Diagnostics "+appReport.getDiagnostics());
    +						LOG.warn("If log aggregation is activated in the Hadoop cluster, we recommend to retrieve "
    +								+ "the full application log using this command:\n"
    +								+ "\tyarn logs -applicationId "+appReport.getApplicationId()+"\n"
    +								+ "(It sometimes takes a few seconds until the logs are aggregated)");
    +					}
    +				} catch(Exception e) {
    +					LOG.warn("Error while getting final application report", e);
    +				}
    +			}
    +		});
    +		actorRunner.setDaemon(true);
    +		actorRunner.start();
    +
    +		pollingRunner = new PollingThread(yarnClient, appId);
    +		pollingRunner.setDaemon(true);
    +		pollingRunner.start();
    +	}
    +
    +	// -------------------------- Interaction with the cluster ------------------------
    +
    +	@Override
    +	public InetSocketAddress getJobManagerAddress() {
    +		return jobManagerAddress;
    +	}
    +
    +	@Override
    +	public String getWebInterfaceURL() {
    +		return this.intialAppReport.getTrackingUrl();
    +	}
    +
    +
    +	@Override
    +	public FlinkYarnClusterStatus getClusterStatus() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		Future<Object> clusterStatusOption = ask(applicationClient, Messages.LocalGetYarnClusterStatus$.MODULE$, akkaTimeout);
    +		Object clusterStatus = awaitUtil(clusterStatusOption, "Unable to get Cluster status from Application Client");
    +		if(clusterStatus instanceof None$) {
    +			return null;
    +		} else if(clusterStatus instanceof Some) {
    +			return (FlinkYarnClusterStatus) (((Some) clusterStatus).get());
    +		} else {
    +			throw new RuntimeException("Unexpected type: "+clusterStatus.getClass().getCanonicalName());
    +		}
    +	}
    +
    +	@Override
    +	public boolean hasFailed() {
    +		if(pollingRunner == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on an uninitialized cluster." +
    +					"The system might be in an erroneous state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if(lastReport == null) {
    +			LOG.warn("FlinkYarnCluster.hasFailed() has been called on a cluster. that didn't receive a status so far." +
    +					"The system might be in an erroneous state");
    +			return false;
    +		} else {
    +			return (lastReport.getYarnApplicationState() == YarnApplicationState.FAILED ||
    +					lastReport.getYarnApplicationState() == YarnApplicationState.KILLED);
    +		}
    +	}
    +
    +	@Override
    +	public String getDiagnostics() {
    +		if (!hasFailed()) {
    +			LOG.warn("getDiagnostics() called for cluster which is not in failed state");
    +		}
    +		ApplicationReport lastReport = pollingRunner.getLastReport();
    +		if (lastReport == null) {
    +			LOG.warn("Last report is null");
    +			return null;
    +		} else {
    +			return lastReport.getDiagnostics();
    +		}
    +	}
    +
    +	@Override
    +	public List<String> getNewMessages() {
    +		if(hasBeenStopped()) {
    +			throw new RuntimeException("The FlinkYarnCluster has alread been stopped");
    +		}
    +		List<String> ret = new ArrayList<String>();
    +		// get messages from ApplicationClient (locally)
    +		while(true) {
    +			Future<Object> messageOptionFuture = ask(applicationClient, Messages.LocalGetYarnMessage$.MODULE$, akkaTimeout);
    +			Object messageOption = awaitUtil(messageOptionFuture, "Error getting new messages from Appliation Client");
    +			if(messageOption instanceof None$) {
    +				break;
    +			} else if(messageOption instanceof org.apache.flink.yarn.Messages.YarnMessage) {
    +				Messages.YarnMessage msg = (Messages.YarnMessage) messageOption;
    +				ret.add("["+msg.date()+"] "+msg.message());
    +			} else {
    +				LOG.warn("LocalGetYarnMessage returned unexpected type: "+messageOption);
    +			}
    +		}
    +		return ret;
    +	}
    +
    +	private static <T> T awaitUtil(Awaitable<T> awaitable, String message) {
    --- End diff --
    
    That is exactly what AkkaUtils.ask does. We could reduce code redundancy here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---