You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/25 08:29:39 UTC
[GitHub] [flink] kl0u opened a new pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
kl0u opened a new pull request #12317:
URL: https://github.com/apache/flink/pull/12317
## What is the purpose of the change
This PR simply changes the error message presented to the user when he/she is trying to submit to Yarn but hadoop is not in the classpath. The new message will be:
```
No Executor found. Please make sure to have hadoop in your classpath. For more information on how to do so refer to the "Deployment & Operations" section of the official Apache Flink documentation.
```
## Brief change log
The changes are:
1. a `FallbackYarnSessionCli` that will be loaded by the `CliFrontend` in case there is a loading error when the frontend tries to load the actual `FlinkYarnSessionCli`. This cli will simply throw an exception with the above message when it detects that the `FlinkYarnSessionCli` would be used but it failed to be loaded (the `isActive` method).
2. change of the error message in the `YarnJobClusterExecutorFactory` and the `YarnSessionClusterExecutorFactory` when the executor cannot be instantiated. This is to cover for the case where the user tries to access a Yarn cluster through the `ExecutorCLI`.
## Verifying this change
Was verified manually and also added the `FallbackYarnSessionCliTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #12317:
URL: https://github.com/apache/flink/pull/12317#discussion_r429811417
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/DefaultClusterClientServiceLoader.java
##########
@@ -70,6 +70,14 @@
throw new IllegalStateException("Multiple compatible client factories found for:\n" + String.join("\n", configStr) + ".");
}
- return compatibleFactories.isEmpty() ? null : (ClusterClientFactory<ClusterID>) compatibleFactories.get(0);
+ if (compatibleFactories.isEmpty()) {
+ throw new IllegalStateException(
+ "No ClusterClientFactory found to execute the application." +
+ " If you were targeting a Yarn cluster, please make sure to have " +
+ "hadoop in your classpath. For more information refer to the " +
Review comment:
```suggestion
"Hadoop in your classpath. For more information refer to the " +
```
same as above, also mention `HADOOP_CLASSPATH`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633451337
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 284e50d0f89d2ad4647a2f22ab68ec10459cf612 (Mon May 25 08:32:42 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u closed pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
kl0u closed pull request #12317:
URL: https://github.com/apache/flink/pull/12317
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #12317:
URL: https://github.com/apache/flink/pull/12317#discussion_r429830852
##########
File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##########
@@ -1067,15 +1085,19 @@ static void setJobManagerAddressInConfig(Configuration config, InetSocketAddress
* @param commandLine The input to the command-line.
* @return custom command-line which is active (may only be one at a time)
*/
- public CustomCommandLine getActiveCustomCommandLine(CommandLine commandLine) {
+ public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
LOG.debug("Custom commandlines: {}", customCommandLines);
for (CustomCommandLine cli : customCommandLines) {
LOG.debug("Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
if (cli.isActive(commandLine)) {
return cli;
}
}
- throw new IllegalStateException("No command-line ran.");
+
+ throw new IllegalStateException(
+ "No valid command-line found. If you were targeting a Yarn cluster, please make sure to have " +
+ "hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " +
Review comment:
Actually this is wrong now because we load the `FallbackYarnSessionCLI` so if we reach here, it is not simply because of the classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128",
"triggerID" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"triggerType" : "PUSH"
}, {
"hash" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2134",
"triggerID" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284e50d0f89d2ad4647a2f22ab68ec10459cf612 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128)
* 8da52f34e59df9d77911ee498ea76a79e3034b49 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2134)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-634380791
The changes looks really good to me. +1 for merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284e50d0f89d2ad4647a2f22ab68ec10459cf612 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #12317:
URL: https://github.com/apache/flink/pull/12317#discussion_r430133100
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/executors/YarnJobClusterExecutorFactory.java
##########
@@ -38,12 +39,16 @@ public String getName() {
}
@Override
- public boolean isCompatibleWith(@Nonnull final Configuration configuration) {
+ public boolean isCompatibleWith(@ Nonnull final Configuration configuration) {
Review comment:
Maybe this is a typo here. We should not have unnecessary space here.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FallbackYarnSessionCli.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn.cli;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.client.cli.AbstractCustomCommandLine;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+import org.apache.flink.yarn.configuration.YarnDeploymentTarget;
+import org.apache.flink.yarn.executors.YarnJobClusterExecutor;
+import org.apache.flink.yarn.executors.YarnSessionClusterExecutor;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * A stub Yarn Command Line to throw an exception with the correct
+ * message when the {@code HADOOP_CLASSPATH} is not set.
+ */
+@Internal
+public class FallbackYarnSessionCli extends AbstractCustomCommandLine {
+
+ public static final String ID = "yarn-cluster";
+
+ private final Option applicationId;
+
+ public FallbackYarnSessionCli(Configuration configuration) {
+ super(configuration);
+ applicationId = new Option("yid", "yarnapplicationId", true, "Attach to running YARN session");
+ }
+
+ @Override
+ public void addGeneralOptions(Options baseOptions) {
+ super.addGeneralOptions(baseOptions);
+ baseOptions.addOption(applicationId);
+ }
+
+ @Override
+ public boolean isActive(CommandLine commandLine) {
+ if (originalIsActive(commandLine)) {
+ throw new IllegalStateException(YarnDeploymentTarget.ERROR_MESSAGE);
+ }
+ return false;
+ }
+
+ private boolean originalIsActive(CommandLine commandLine) {
Review comment:
So we are intentional to ignore the Yarn properties file here. Right?
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FallbackYarnSessionCli.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn.cli;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.client.cli.AbstractCustomCommandLine;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+import org.apache.flink.yarn.configuration.YarnDeploymentTarget;
+import org.apache.flink.yarn.executors.YarnJobClusterExecutor;
+import org.apache.flink.yarn.executors.YarnSessionClusterExecutor;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * A stub Yarn Command Line to throw an exception with the correct
+ * message when the {@code HADOOP_CLASSPATH} is not set.
+ */
+@Internal
+public class FallbackYarnSessionCli extends AbstractCustomCommandLine {
+
+ public static final String ID = "yarn-cluster";
+
+ private final Option applicationId;
+
+ public FallbackYarnSessionCli(Configuration configuration) {
+ super(configuration);
+ applicationId = new Option("yid", "yarnapplicationId", true, "Attach to running YARN session");
+ }
+
+ @Override
+ public void addGeneralOptions(Options baseOptions) {
+ super.addGeneralOptions(baseOptions);
+ baseOptions.addOption(applicationId);
+ }
+
+ @Override
+ public boolean isActive(CommandLine commandLine) {
+ if (originalIsActive(commandLine)) {
+ throw new IllegalStateException(YarnDeploymentTarget.ERROR_MESSAGE);
+ }
+ return false;
+ }
+
+ private boolean originalIsActive(CommandLine commandLine) {
Review comment:
Yeah, i agree with you that we do not need to add the logics for Yarn magic properties file here.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FallbackYarnSessionCli.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn.cli;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.client.cli.AbstractCustomCommandLine;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+import org.apache.flink.yarn.configuration.YarnDeploymentTarget;
+import org.apache.flink.yarn.executors.YarnJobClusterExecutor;
+import org.apache.flink.yarn.executors.YarnSessionClusterExecutor;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * A stub Yarn Command Line to throw an exception with the correct
+ * message when the {@code HADOOP_CLASSPATH} is not set.
+ */
+@Internal
+public class FallbackYarnSessionCli extends AbstractCustomCommandLine {
+
+ public static final String ID = "yarn-cluster";
+
+ private final Option applicationId;
+
+ public FallbackYarnSessionCli(Configuration configuration) {
+ super(configuration);
+ applicationId = new Option("yid", "yarnapplicationId", true, "Attach to running YARN session");
+ }
+
+ @Override
+ public void addGeneralOptions(Options baseOptions) {
+ super.addGeneralOptions(baseOptions);
+ baseOptions.addOption(applicationId);
+ }
+
+ @Override
+ public boolean isActive(CommandLine commandLine) {
+ if (originalIsActive(commandLine)) {
+ throw new IllegalStateException(YarnDeploymentTarget.ERROR_MESSAGE);
+ }
+ return false;
+ }
+
+ private boolean originalIsActive(CommandLine commandLine) {
Review comment:
Yeah, i agree with you that we do not need to add the logics for Yarn magic properties file here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128",
"triggerID" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"triggerType" : "PUSH"
}, {
"hash" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2134",
"triggerID" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284e50d0f89d2ad4647a2f22ab68ec10459cf612 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128)
* 8da52f34e59df9d77911ee498ea76a79e3034b49 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2134)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #12317:
URL: https://github.com/apache/flink/pull/12317#discussion_r429811204
##########
File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##########
@@ -1067,15 +1085,19 @@ static void setJobManagerAddressInConfig(Configuration config, InetSocketAddress
* @param commandLine The input to the command-line.
* @return custom command-line which is active (may only be one at a time)
*/
- public CustomCommandLine getActiveCustomCommandLine(CommandLine commandLine) {
+ public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
LOG.debug("Custom commandlines: {}", customCommandLines);
for (CustomCommandLine cli : customCommandLines) {
LOG.debug("Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
if (cli.isActive(commandLine)) {
return cli;
}
}
- throw new IllegalStateException("No command-line ran.");
+
+ throw new IllegalStateException(
+ "No valid command-line found. If you were targeting a Yarn cluster, please make sure to have " +
+ "hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " +
Review comment:
```suggestion
"Hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " +
```
maybe also say sth about`HADOOP_CLASSPATH` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
kl0u commented on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633474399
Thanks for the comments @aljoscha. I updated the PR accordingly. Let me know what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #12317:
URL: https://github.com/apache/flink/pull/12317#discussion_r430387701
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FallbackYarnSessionCli.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn.cli;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.client.cli.AbstractCustomCommandLine;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+import org.apache.flink.yarn.configuration.YarnDeploymentTarget;
+import org.apache.flink.yarn.executors.YarnJobClusterExecutor;
+import org.apache.flink.yarn.executors.YarnSessionClusterExecutor;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * A stub Yarn Command Line to throw an exception with the correct
+ * message when the {@code HADOOP_CLASSPATH} is not set.
+ */
+@Internal
+public class FallbackYarnSessionCli extends AbstractCustomCommandLine {
+
+ public static final String ID = "yarn-cluster";
+
+ private final Option applicationId;
+
+ public FallbackYarnSessionCli(Configuration configuration) {
+ super(configuration);
+ applicationId = new Option("yid", "yarnapplicationId", true, "Attach to running YARN session");
+ }
+
+ @Override
+ public void addGeneralOptions(Options baseOptions) {
+ super.addGeneralOptions(baseOptions);
+ baseOptions.addOption(applicationId);
+ }
+
+ @Override
+ public boolean isActive(CommandLine commandLine) {
+ if (originalIsActive(commandLine)) {
+ throw new IllegalStateException(YarnDeploymentTarget.ERROR_MESSAGE);
+ }
+ return false;
+ }
+
+ private boolean originalIsActive(CommandLine commandLine) {
Review comment:
Yes, the reason is that:
1) we would have to copy more code
2) I would assume that if we have the properties (hidden) file, it means that we created the Yarn cluster using the session CLI, so we already have hadoop in the classpath.
(In fact the solution here is already fragile, as whoever touches the `isActive()` should know to keep both methods in sync)
Do you agree @wangyang0918 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128",
"triggerID" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"triggerType" : "PUSH"
}, {
"hash" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8da52f34e59df9d77911ee498ea76a79e3034b49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284e50d0f89d2ad4647a2f22ab68ec10459cf612 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128)
* 8da52f34e59df9d77911ee498ea76a79e3034b49 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12317: [FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12317:
URL: https://github.com/apache/flink/pull/12317#issuecomment-633460641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128",
"triggerID" : "284e50d0f89d2ad4647a2f22ab68ec10459cf612",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284e50d0f89d2ad4647a2f22ab68ec10459cf612 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2128)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org