You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/08/19 13:56:50 UTC
[GitHub] [flink] kl0u commented on a change in pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
kl0u commented on a change in pull request #13024:
URL: https://github.com/apache/flink/pull/13024#discussion_r473046293
##########
File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##########
@@ -219,34 +219,49 @@ protected void run(String[] args) throws Exception {
final ProgramOptions programOptions = ProgramOptions.create(commandLine);
- final PackagedProgram program =
- getPackagedProgram(programOptions);
+ final List<URL> jobJars = getJobJarAndDependencies(programOptions);
- final List<URL> jobJars = program.getJobJarAndDependencies();
final Configuration effectiveConfiguration = getEffectiveConfiguration(
activeCommandLine, commandLine, programOptions, jobJars);
LOG.debug("Effective executor configuration: {}", effectiveConfiguration);
+ final PackagedProgram program = getPackagedProgram(programOptions, effectiveConfiguration);
+
try {
executeProgram(effectiveConfiguration, program);
} finally {
program.deleteExtractedLibraries();
}
}
- private PackagedProgram getPackagedProgram(ProgramOptions programOptions) throws ProgramInvocationException, CliArgsException {
+ /**
+ * Get all provided libraries needed to run the program from the ProgramOptions.
+ */
+ public List<URL> getJobJarAndDependencies(ProgramOptions programOptions) throws FileNotFoundException, ProgramInvocationException {
Review comment:
Here this can be package-private and not public. Also we should annotate it as `@VisibleForTesting`.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##########
@@ -219,34 +219,49 @@ protected void run(String[] args) throws Exception {
final ProgramOptions programOptions = ProgramOptions.create(commandLine);
- final PackagedProgram program =
- getPackagedProgram(programOptions);
+ final List<URL> jobJars = getJobJarAndDependencies(programOptions);
- final List<URL> jobJars = program.getJobJarAndDependencies();
final Configuration effectiveConfiguration = getEffectiveConfiguration(
activeCommandLine, commandLine, programOptions, jobJars);
LOG.debug("Effective executor configuration: {}", effectiveConfiguration);
+ final PackagedProgram program = getPackagedProgram(programOptions, effectiveConfiguration);
+
try {
executeProgram(effectiveConfiguration, program);
} finally {
program.deleteExtractedLibraries();
}
}
- private PackagedProgram getPackagedProgram(ProgramOptions programOptions) throws ProgramInvocationException, CliArgsException {
+ /**
+ * Get all provided libraries needed to run the program from the ProgramOptions.
+ */
+ public List<URL> getJobJarAndDependencies(ProgramOptions programOptions) throws FileNotFoundException, ProgramInvocationException {
+ String entryPointClass = programOptions.getEntryPointClassName();
+ String jarFilePath = programOptions.getJarFilePath();
+
+ File jarFile = jarFilePath != null ? getJarFile(jarFilePath) : null;
+
+ return PackagedProgram
+ .getJobJarAndDependencies(jarFile != null ? jarFile : null, entryPointClass);
Review comment:
This can become `return PackagedProgram.getJobJarAndDependencies(jarFile, entryPointClass);`
##########
File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##########
@@ -219,34 +219,49 @@ protected void run(String[] args) throws Exception {
final ProgramOptions programOptions = ProgramOptions.create(commandLine);
- final PackagedProgram program =
- getPackagedProgram(programOptions);
+ final List<URL> jobJars = getJobJarAndDependencies(programOptions);
- final List<URL> jobJars = program.getJobJarAndDependencies();
final Configuration effectiveConfiguration = getEffectiveConfiguration(
activeCommandLine, commandLine, programOptions, jobJars);
LOG.debug("Effective executor configuration: {}", effectiveConfiguration);
+ final PackagedProgram program = getPackagedProgram(programOptions, effectiveConfiguration);
+
try {
executeProgram(effectiveConfiguration, program);
} finally {
program.deleteExtractedLibraries();
}
}
- private PackagedProgram getPackagedProgram(ProgramOptions programOptions) throws ProgramInvocationException, CliArgsException {
+ /**
+ * Get all provided libraries needed to run the program from the ProgramOptions.
+ */
+ public List<URL> getJobJarAndDependencies(ProgramOptions programOptions) throws FileNotFoundException, ProgramInvocationException {
+ String entryPointClass = programOptions.getEntryPointClassName();
+ String jarFilePath = programOptions.getJarFilePath();
+
+ File jarFile = jarFilePath != null ? getJarFile(jarFilePath) : null;
+
+ return PackagedProgram
+ .getJobJarAndDependencies(jarFile != null ? jarFile : null, entryPointClass);
+ }
+
+ private PackagedProgram getPackagedProgram(
+ ProgramOptions programOptions,
+ Configuration effectiveConfiguration) throws ProgramInvocationException, CliArgsException {
PackagedProgram program;
try {
LOG.info("Building program from JAR file");
- program = buildProgram(programOptions);
+ program = buildProgram(programOptions, effectiveConfiguration);
} catch (FileNotFoundException e) {
throw new CliArgsException("Could not build the program from JAR file: " + e.getMessage(), e);
}
return program;
}
- private <T> Configuration getEffectiveConfiguration(
+ public <T> Configuration getEffectiveConfiguration(
Review comment:
Here this can be package-private and not public. Also we should annotate it as @VisibleForTesting.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
##########
@@ -220,12 +220,47 @@ public ClassLoader getUserCodeClassLoader() {
* Returns all provided libraries needed to run the program.
*/
public List<URL> getJobJarAndDependencies() {
- List<URL> libs = new ArrayList<URL>(this.extractedTempLibraries.size() + 1);
+ return getJobJarAndDependencies(jarFile, extractedTempLibraries, isPython);
+ }
+
+ /**
+ * Returns all provided libraries needed to run the program.
+ */
+ public static List<URL> getJobJarAndDependencies(File jarFile, @Nullable String entryPointClassName) throws ProgramInvocationException {
Review comment:
In this class you add this method only, so the rest of the changes can be reverted, right. Essentially the `public List<URL> getJobJarAndDependencies()` can stay as is currently on the master and you can remove the `List<URL> getJobJarAndDependencies(URL jarFileUrl, List<File> extractedTempLibraries, boolean isPython)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org