You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/06/25 06:23:02 UTC
[GitHub] [flink] XComp opened a new pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
XComp opened a new pull request #16286:
URL: https://github.com/apache/flink/pull/16286
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
*(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)*
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-870142926
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663865457
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
Since now we always have a non-null `jarFile`, do we need to check whether the specified jar contains the `jobClassName` and then fail fast?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-874020720
Thanks @zentol . I addressed your JavaDoc comments and took the opportunity to make the condition on the `FromJarEntryClassInformationProvider` stricter. It's only called with a jar archive being specified. That resolved also some of your JavaDoc-related comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663781616
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
Review comment:
The condition when it returns an empty optional is unnecessary and just creates confusion. The jar _and_ classname not being specified is a case that a consumer of this APIn ever has to be concerned with because the constructor rejects it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r660043411
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
Review comment:
Where do you see the benefits of using inheritance here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662100188
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.commons.io.FilenameUtils;
+import org.hamcrest.CoreMatchers;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsCollectionContaining;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.Collection;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/**
+ * {@code FromClasspathEntryClassInformationProviderTest} tests {@link
+ * FromClasspathEntryClassInformationProvider}.
+ */
+public class FromClasspathEntryClassInformationProviderTest extends TestLogger {
+
+ @Rule public final ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testJobClassOnUserClasspathWithExplicitJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.create(
+ classpathProvider.getJobClassName(),
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testJobClassOnUserClasspathWithOnlyTestFileOnClasspath()
+ throws MalformedURLException, FlinkException {
+ // we want to check that the right exception is thrown if the user classpath is empty
+ FromClasspathEntryClassInformationProvider.create(
+ "SomeJobClassName", classpathProvider.getURLUserClasspathWithOnlyTextFile());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider.create(
+ null, classpathProvider.getURLUserClasspathWithEntryClass());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingUserClasspath() throws FlinkException {
+ FromClasspathEntryClassInformationProvider.create("jobClassName", null);
+ }
+
+ @Test
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithoutEntryClass());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnUserClasspath() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassNameAndMissingUserClasspath()
+ throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(null);
+ }
+
+ @Test
+ public void testJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithoutEntryClass();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test
+ public void testJarFromSystemClasspathSanityCheck() {
Review comment:
We can keep it because it already exists, but what I objected to is searching for the junit jar on a conceptual level. It is not sound to couple the usage of some testing library to the correctness of production code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662104426
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithoutUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ assertThat(jobGraph.getClasspaths().isEmpty(), is(true));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testChildFirstDefaultConfiguration() throws FlinkException {
Review comment:
The actual bug which we want to fix in `FLINK-21445` is that we missed setting the configuration in the `ClassPathPackagedProgramRetriever` which is now replaced by `PackagedProgramRetrieverImpl`. That's why I added the tests here: To cover this behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662086061
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithoutUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ assertThat(jobGraph.getClasspaths().isEmpty(), is(true));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testChildFirstDefaultConfiguration() throws FlinkException {
Review comment:
Shouldn't this then belong into a `PackagedProgramTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 7fe6639f640124b804662177ea510334834f761f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
* 5e03a511663c7cadda1e3f037c30ba3dfda1b05e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662103266
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -162,6 +164,7 @@ public void before() throws IOException {
directory = temporaryFolder.newFolder(directoryNameSuffix);
directoryContentCreator.accept(directory);
+ resetSystemClasspath();
Review comment:
it shouldn't be necessary to reset it in before()
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -590,4 +567,25 @@ private JobGraph retrieveJobGraph(
packagedProgram, configuration, defaultParallelism, false);
return PipelineExecutorUtils.getJobGraph(pipeline, configuration);
}
+
+ private static List<String> extractRelativizedURLsForJarsFromDirectory(File directory)
+ throws MalformedURLException {
+ Preconditions.checkArgument(
+ directory.listFiles() != null,
+ "The passed File does not seem to be a directory: " + directory.getAbsolutePath());
Review comment:
Why not use `File.isDirectory`?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -384,12 +384,22 @@ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
}
@Test(expected = FlinkException.class)
- public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ public void testWithoutJobClassAndMultipleEntryClassesOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ multipleEntryClassesClasspathProvider.getDirectory(),
+ null,
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testWithoutJobClassAndMultipleEntryClassesOnSystemClasspath()
Review comment:
add a comment for why multiple entry classes are on the system classpath.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -126,10 +126,9 @@ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws Fl
}
@Test
- public void testDeriveEntryClassInformationForCustomJar()
- throws FlinkException, MalformedURLException {
- // make loading from system classpath fail to make sure that it's not triggered
- multipleEntryClassesClasspathProvider.setSystemClasspath();
+ public void testDeriveEntryClassInformationForCustomJar() throws FlinkException {
+ // clearing the system classpath to make sure that no data is collected from there
+ System.setProperty("java.class.path", "");
Review comment:
Could we not use the `noEntryClassClasspathProvider`? As is I'd argue this test should explicitly restore the classpath property afterwards, and not rely on the ClasspathProvider usages by other tests.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -197,8 +197,8 @@ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
// the user directory must be specified
multipleEntryClassesClasspathProvider.getDirectory(),
- // the user classpath is derived from the user directory outside of the
- // method
+ // the user classpath is derived from the user directory outside of
+ // createEntryClassInformationProvider
Review comment:
I still don't see the value of this comment. Isn't it obvious the classpath is derived outside of createEntryClassInformationProvider if we pass in the classpath?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658541693
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
Review comment:
Is this a right concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-873055234
I rebased the PR to make the `JobMasterITCase` succeed in the AzureCI builds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
* 44c00395058d15d3fdee19f54c7c5261db213234 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663818788
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
Review comment:
Fair enough, the information is also given in `@return` docs. So, I guess, that's good enough.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
Review comment:
I agree, but that would not reflect the original code. That's why I didn't add it. We also have to consider the fact that the entry class is not specified in the jar archive. In that case setting the job class name explicitly makes sense
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
Review comment:
I thought about it once more. I got the precondition from [PackagedProgram:687](https://github.com/apache/flink/blob/c08bef1f7913eb1c416c278354fd62b82b172549/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java#L687) not realizing that the `FromJarEntryClassInformationProvider` is actually on used for the PyFlink (where the Jar and the job class name are explicitly given) or [if a jarFile is specified in DefaultPackagedProgramRetriever](https://github.com/XComp/flink/blob/FLINK-21445/flink-clients/src/main/java/org/apache/flink/client/program/DefaultPackagedProgramRetriever.java#L128). Hence, we can make the implementation more stricter which makes the JavaDoc valid again.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
I rather don't want to change the failure logic as part of this PR. Just by looking at the code, it would be possible, I guess. But I'm hesitant to do this because it might open other refactoring efforts which I want to avoid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-870142926
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661745110
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnApplicationClusterEntryPoint.java
##########
@@ -129,16 +129,12 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws FlinkException {
final File userLibDir = YarnEntrypointUtils.getUsrLibDir(configuration).orElse(null);
final File userApplicationJar = getUserApplicationJar(userLibDir, configuration);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJarFile(userApplicationJar)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(
+ userLibDir, userApplicationJar, jobClassName, programArguments, configuration);
Review comment:
I switched to actual factory methods because we didn't use the builder functionality. `@Nullable` values are passed. The production code boiled down to two different signatures reflected in the `PackagedProgramRetrieverImpl.create` factory methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* e3b0831a1476a562452576d854dc2d43a3ab3e21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* d5aa380c6fbe941651d2b095a6d80e92fee04ab3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642)
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662079061
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -215,20 +215,6 @@ public void testCreateWithUserLibDir() throws FlinkException {
is(classpathProvider.getJobClassName()));
}
- @Test
- public void testCreateWithoutUserLibDir() throws IOException, FlinkException {
Review comment:
Not setting the user classpath makes the code fallback to the system classpath. We're not able to modify the system classpath after JVM is started already. Modifying `java.class.path` doesn't help here. The default system classpath contains multiple classes with main methods. That's why, the test would fail. The code path is, instead, covered by PackagedProgramRetrieverImplTest.testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663940924
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
I rather don't want to change the failure logic as part of this PR. Just by looking at the code, it would be possible, I guess. But I'm hesitant to do this because it might open other refactoring efforts which I want to avoid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662128008
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -162,6 +164,7 @@ public void before() throws IOException {
directory = temporaryFolder.newFolder(directoryNameSuffix);
directoryContentCreator.accept(directory);
+ resetSystemClasspath();
Review comment:
You're right. I just felt that it would be another safety net. But ok, I'm gonna remove it again since I have no actual reason to keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r659659300
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
Review comment:
I left the test here for now for transparency reasons. I think we should remove it but make the job fail in case the job class is not even provided on the system class path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658521964
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir, @Nullable String jobClassName, String[] programArgs)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
Review comment:
Thanks for being so responsive looking into the PR. You're right. The draft is still a draft. I'm gonna add the fixes for configuration and savepoint settings soon. I needed to create the draft first because I wanted to link it to [FLINK-23154](https://issues.apache.org/jira/browse/FLINK-23154).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
}, {
"hash" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19957",
"triggerID" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 878a107e0364b415c288f58eea4a4a716797e77d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920)
* 74c80baae62fa14f9f62f8707fd95093a86fd302 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19957)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* e3b0831a1476a562452576d854dc2d43a3ab3e21 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523)
* 349d6855c21fed4d4368c56646551c3524437384 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661754312
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
Review comment:
I'm not 100% sure whether I understand correctly. You're referring to `jarFile`? My goal was to have the `createEntryClassInformationProvider` method contain all the different code paths to select the right `EntryClassInformationProvider`. That enabled us to test this method specifically and be as close as possible to the original code. Making one of the fields `@Nonnull` would have meant to move some if condition out of this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868257439
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 74c80baae62fa14f9f62f8707fd95093a86fd302 (Thu Sep 23 17:56:44 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* e3b0831a1476a562452576d854dc2d43a3ab3e21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523)
* 349d6855c21fed4d4368c56646551c3524437384 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 7fe6639f640124b804662177ea510334834f761f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662057935
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProviderTest.java
##########
@@ -72,9 +73,13 @@ public void testEitherJobClassNameOrJarHasToBeSet() {
FromJarEntryClassInformationProvider.createFromCustomJar(null, null);
}
- @Ignore // FLINK_OPT_DIR has to be set
@Test
public void testPythonJarFile() {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
Review comment:
Yeah, I thought of building a classpath that includes a mocked `flink-python.jar`. But adapting the enviroment variables is not advisable. That's were I stopped investigating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662085256
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -328,18 +329,15 @@ public void testFailIfJobDirDoesNotHaveEntryClass() {
}
}
- @Test
- public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException, IOException {
- classpathProvider.setSystemClasspathWithoutEntryClass();
- final PackagedProgramRetrieverImpl retriever =
- PackagedProgramRetrieverImpl.create(
- null,
- classpathProvider.getTestJobClassName(),
- new String[0],
- new Configuration());
- assertThat(
- retriever.getPackagedProgram().getMainClassName(),
- is(classpathProvider.getTestJobClassName()));
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
Review comment:
Correct, but I would keep it in as an additional test in case we decide to make the code more strict (as mentioned in [this comment](https://github.com/apache/flink/pull/16286#discussion_r658542629)).
Right now, I lean towards keeping it like it was implemented in the old implementation to not open any hidden problems which might make the PR become even bigger. Instead, I'd create a Flink issue to resolve that. But I wanted to get your opinions on that one. I wasn't 100% sure whether it's actually a desired behavior not checking the class existence on the system classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661397201
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
Review comment:
I refactored the class to remove the "hodgepodge". Now, one instance serves as a utility method provider for a specific case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 2a812aa5c826e278b0679572a5b37025aecbb24b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895)
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 878a107e0364b415c288f58eea4a4a716797e77d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 349d6855c21fed4d4368c56646551c3524437384 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661990749
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
Review comment:
But I updated the condition to use `FileUtils` and added a more meaningful comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 44c00395058d15d3fdee19f54c7c5261db213234 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777)
* 7fe6639f640124b804662177ea510334834f761f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-869623576
@XComp , thanks for your efforts to update the pull request again me. I have reviewed the changes of the refactors. I have confusion about why not validate whether to retrever job graph in tests, which originally validate through `retrieveJobGraph`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
}, {
"hash" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19957",
"triggerID" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 74c80baae62fa14f9f62f8707fd95093a86fd302 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19957)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661991647
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithoutUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ assertThat(jobGraph.getClasspaths().isEmpty(), is(true));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testChildFirstDefaultConfiguration() throws FlinkException {
Review comment:
Their actual purpose is to check whether the configuration is acknowledged. The classloader is only the vehicle to test that. That's why, I would object renaming them. Or do you have some proposal for a better name? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663785588
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
Review comment:
```suggestion
* system classpath. It can return an empty {@code Optional} despite the respective {@code jarFile}
* being set if the job class is the entry class of the jar.
```
The descriptions are bit difficult to understand :/
Note also that this behavior is super weird and we should just return the entry class imo.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
+ * {@code jarFile}.
+ *
+ * @return Returns the job class that can either be found in the respective {@code jarFile} or
+ * on the system classpath. It can also return an empty {@code Optional} if the {@code
+ * jarFile} is specified. In that case the caller should assume the job class being provided
+ * by the {@code jarFile} as the entry class.
Review comment:
similar suggestion to above
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
Review comment:
```suggestion
* @param jobClassName The job class to use; if null the system classpath will be scanned for possible main classes.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
Review comment:
Well this is quite a non-comment isn't it. It could say something like "default X implementation that can retrieve a packaged program from a specific jar or the system classpath."
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
Review comment:
Recently we used `Default*` for the default interface implementations.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
Review comment:
```suggestion
* @param jarFile The jar archive expected to contain the job class; null if the job class is on the system classpath.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610)
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663865457
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
Since now we always have a non-null `jarFile`, do we need to check whether the specified jar contains the `jobClassName` and then fail fast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662117121
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -590,4 +567,25 @@ private JobGraph retrieveJobGraph(
packagedProgram, configuration, defaultParallelism, false);
return PipelineExecutorUtils.getJobGraph(pipeline, configuration);
}
+
+ private static List<String> extractRelativizedURLsForJarsFromDirectory(File directory)
+ throws MalformedURLException {
+ Preconditions.checkArgument(
+ directory.listFiles() != null,
+ "The passed File does not seem to be a directory: " + directory.getAbsolutePath());
Review comment:
I initially thought the same. But objected it because in the end we're using `listFiles` in the code. And `listFiles` does not only return `null` in case of it not being a directory. But I udpated the error message to reflect that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661985331
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
Review comment:
No, it also fails if there are too many entrypoints on the classpath but no job class name is specified. But you're right. The test method's name is not specific enough. I updated the method name and additional tests. I realized that I wasn't precise here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661394188
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
Review comment:
Right now, we can specify a class and don't check whether it's actual present on the classpath. I was wondering whether that's something we want to change. We're already checking whether the class is available in the user classpath. We could do the same for the system classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662059148
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.File;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+/**
+ * {@code FromJarEntryClassInformationProviderTest} tests {@link
+ * FromJarEntryClassInformationProvider}.
+ */
+public class FromJarEntryClassInformationProviderTest extends TestLogger {
+
+ @Test
+ public void testCustomJarFile() {
+ final File jarFile = new File("some/path/to/jar");
+ final String jobClassName = "JobClassName";
+ final FromJarEntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+
+ assertThat(testInstance.getJarFile().isPresent(), is(true));
+ assertThat(testInstance.getJarFile().get(), is(jarFile));
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(jobClassName));
+ }
+
+ @Test
+ public void testMissingJar() {
+ final String jobClassName = "JobClassName";
+ final EntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(null, jobClassName);
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(jobClassName));
+ }
+
+ @Test
+ public void testMissingJobClassName() {
+ final File jarFile = new File("some/path/to/jar");
+ final EntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, null);
+ assertThat(testInstance.getJarFile().isPresent(), is(true));
+ assertThat(testInstance.getJarFile().get(), is(jarFile));
+ assertThat(testInstance.getJobClassName().isPresent(), is(false));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testEitherJobClassNameOrJarHasToBeSet() {
+ FromJarEntryClassInformationProvider.createFromCustomJar(null, null);
+ }
+
+ @Test
+ public void testPythonJarFile() {
Review comment:
I guess, you're right. It was just that I thought it's an obvious test and just later on realized that it's not that easy to fix. Reconsidering this, I have to admit that my approach is not the right one. I'm gonna revert the tests and comment on `FLINK-23154` accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610)
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r659716223
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProvider.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/**
+ * {@code FromClasspathEntryClassInformationProvider} assumes the passed job class being available
+ * on some classpath.
+ */
+public class FromClasspathEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} based on the passed job class
+ * and classpath.
+ *
+ * @param jobClassName The job's class name.
+ * @param classpath The classpath the job class should be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} instances collecting the
+ * necessary information.
+ * @throws FlinkException If the passed job class is not present on the passed classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider create(
+ String jobClassName, Iterable<URL> classpath) throws FlinkException {
+ Preconditions.checkNotNull(jobClassName, "No job class name passed.");
+ Preconditions.checkNotNull(classpath, "No classpath passed.");
+ if (!userClasspathContainsJobClass(jobClassName, classpath)) {
+ throw new FlinkException(
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ jobClassName));
+ }
+
+ return new FromClasspathEntryClassInformationProvider(jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} looking for the job class on the
+ * passed classpath.
+ *
+ * @param classpath The classpath the job class is expected to be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} providing the job class found
+ * on the passed classpath.
+ * @throws IOException If some Jar listed on the classpath wasn't accessible.
+ * @throws NoSuchElementException No main method was found on the classpath.
+ * @throws IllegalArgumentException To many main methods were found on the classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider createFromClasspath(
+ Iterable<URL> classpath)
+ throws IOException, NoSuchElementException, IllegalArgumentException {
+ return new FromClasspathEntryClassInformationProvider(
+ extractJobClassFromUrlClasspath(classpath));
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} looking for the job class on the
+ * system classpath.
+ *
+ * @return The {@code FromClasspathEntryClassInformationProvider} providing the job class found
+ * on the system classpath.
+ * @throws IOException If some Jar listed on the system classpath wasn't accessible.
+ * @throws NoSuchElementException No main method was found on the system classpath.
+ * @throws IllegalArgumentException To many main methods were found on the system classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider createFromSystemClasspath()
+ throws IOException, NoSuchElementException, IllegalArgumentException {
+ return new FromClasspathEntryClassInformationProvider(extractJobClassFromSystemClasspath());
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} assuming that the passed job
+ * class is available on the system classpath. This method is kind of legacy and might be
+ * subject to removal.
+ *
+ * @param jobClassName The job class name working as the entry point.
+ * @return The {@code FromClasspathEntryClassInformationProvider} providing the job class found.
+ */
+ public static FromClasspathEntryClassInformationProvider
+ createWithJobClassAssumingOnSystemClasspath(String jobClassName) {
+ return new FromClasspathEntryClassInformationProvider(jobClassName);
+ }
+
+ private FromClasspathEntryClassInformationProvider(String jobClassName) {
+ this.jobClassName = Preconditions.checkNotNull(jobClassName, "No job class name set.");
+ }
+
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.empty();
+ }
+
+ @Override
+ public String getJobClassName() {
+ return jobClassName;
+ }
+
+ @VisibleForTesting
+ static Iterable<File> extractSystemClasspath() {
+ final String classpathPropertyValue = System.getProperty("java.class.path", "");
+ final String pathSeparator = System.getProperty("path.separator", ":");
+
+ return Arrays.stream(classpathPropertyValue.split(pathSeparator))
+ .filter(entry -> !StringUtils.isNullOrWhitespaceOnly(entry))
+ .map(File::new)
+ .filter(File::isFile)
+ .filter(f -> isJarFilename(f.getName()))
+ .collect(Collectors.toList());
+ }
+
+ private static String extractJobClassFromSystemClasspath()
+ throws IOException, NoSuchElementException, IllegalArgumentException {
+ return extractJobClassNameFromFileClasspath(extractSystemClasspath());
+ }
+
+ private static String extractJobClassFromUrlClasspath(Iterable<URL> classpath)
+ throws IOException, NoSuchElementException, IllegalArgumentException {
+ final List<File> jarFilesFromClasspath =
+ StreamSupport.stream(classpath.spliterator(), false)
+ .map(url -> new File(url.getFile()))
+ .filter(f -> isJarFilename(f.getName()))
+ .collect(Collectors.toList());
+ return extractJobClassNameFromFileClasspath(jarFilesFromClasspath);
+ }
+
+ private static String extractJobClassNameFromFileClasspath(Iterable<File> classpath)
+ throws IOException, NoSuchElementException, IllegalArgumentException {
+ return JarManifestParser.findOnlyEntryClass(classpath).getEntryClass();
+ }
+
+ private static boolean userClasspathContainsJobClass(
+ String jobClassName, Iterable<URL> classpath) {
+ for (URL url : classpath) {
+ if (!isJarFile(url)) {
+ continue;
+ }
+
+ try (final JarFile jarFile = new JarFile(url.getFile())) {
+ if (jarContainsJobClass(jobClassName, jarFile)) {
+ return true;
+ }
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e,
+ String.format(
+ "Failed to open user class path %s. Make sure that all files on the user classpath can be accessed.",
Review comment:
```suggestion
"Failed to open user classpath %s. Make sure that all files on the user classpath can be accessed.",
```
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/JarManifestParser.java
##########
@@ -118,7 +118,7 @@ static JarFileWithEntryClass findOnlyEntryClass(Iterable<File> jarFiles) throws
* @return Optional holding value of first found attribute
* @throws IOException If there is an error accessing the JAR
*/
- private static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
+ public static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
Review comment:
Why not `findFirstManifestAttribute` define as `protected`?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code EntryClassInformationProvider} provides information about the entry class that shall be
+ * used by the {@link org.apache.flink.client.program.PackagedProgramRetrieverImpl}.
Review comment:
Could `{@link org.apache.flink.client.program.PackagedProgramRetrieverImpl}` update to `{@link PackagedProgramRetrieverImpl}`? In my pull request, this is recommended to update like this.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
Review comment:
Does this need to inherit `FromClasspathEntryClassInformationProvider`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658542629
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws IOException, FlinkException {
+ // setting the system classpath to include two main methods makes sure that we can verify
+ // that the user classpath information is picked up from the user directory - otherwise, the
+ // call would fail
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testCreateWithoutUserLibDir() throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ // setting no jar, user lib dir and jobClassName makes sure that everything's picked up from
+ // the system classpath
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null, null, null, new String[0], new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(), is(classpathProvider.getJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: Shouldn't the related jar be on the classpath?
+ // classpathProvider.setSystemClasspathWithTestJob();
Review comment:
This test was migrated from the old `ClassPathPackagedProgramRetrieverTest`. But it feels to me that we don't want that behavior, do we?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663959462
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
Review comment:
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663839349
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
Review comment:
I thought about it once more. I got the precondition from [PackagedProgram:687](https://github.com/apache/flink/blob/c08bef1f7913eb1c416c278354fd62b82b172549/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java#L687) not realizing that the `FromJarEntryClassInformationProvider` is actually on used for the PyFlink (where the Jar and the job class name are explicitly given) or [if a jarFile is specified in DefaultPackagedProgramRetriever](https://github.com/XComp/flink/blob/FLINK-21445/flink-clients/src/main/java/org/apache/flink/client/program/DefaultPackagedProgramRetriever.java#L128). Hence, we can make the implementation more stricter which makes the JavaDoc valid again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658518114
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnApplicationClusterEntryPoint.java
##########
@@ -129,16 +129,12 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = YarnEntrypointUtils.getUsrLibDir(configuration).orElse(null);
final File userApplicationJar = getUserApplicationJar(userLibDir, configuration);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJarFile(userApplicationJar)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterEntryPoint.java
##########
@@ -111,13 +111,9 @@ private static PackagedProgram getPackagedProgram(
private static PackagedProgramRetriever getPackagedProgramRetriever(
final String[] programArguments, @Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(userLibDir, jobClassName, programArguments);
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesApplicationClusterEntrypoint.java
##########
@@ -105,22 +105,20 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
// No need to do pipelineJars validation if it is a PyFlink job.
if (!(PackagedProgramUtils.isPython(jobClassName)
|| PackagedProgramUtils.isPython(programArguments))) {
final List<File> pipelineJars =
KubernetesUtils.checkJarFileForApplicationMode(configuration);
Preconditions.checkArgument(pipelineJars.size() == 1, "Should only have one jar");
- retrieverBuilder.setJarFile(pipelineJars.get(0));
+ return PackagedProgramRetrieverImpl.create(
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesApplicationClusterEntrypoint.java
##########
@@ -105,22 +105,20 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
// No need to do pipelineJars validation if it is a PyFlink job.
if (!(PackagedProgramUtils.isPython(jobClassName)
|| PackagedProgramUtils.isPython(programArguments))) {
final List<File> pipelineJars =
KubernetesUtils.checkJarFileForApplicationMode(configuration);
Preconditions.checkArgument(pipelineJars.size() == 1, "Should only have one jar");
- retrieverBuilder.setJarFile(pipelineJars.get(0));
+ return PackagedProgramRetrieverImpl.create(
+ userLibDir, pipelineJars.get(0), jobClassName, programArguments);
}
- return retrieverBuilder.build();
+
+ return PackagedProgramRetrieverImpl.create(userLibDir, jobClassName, programArguments);
Review comment:
This doesn't pass the `Configuration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 878a107e0364b415c288f58eea4a4a716797e77d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663822386
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
Review comment:
I agree, but that would not reflect the original code. That's why I didn't add it. We also have to consider the fact that the entry class is not specified in the jar archive. In that case setting the job class name explicitly makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* d5aa380c6fbe941651d2b095a6d80e92fee04ab3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642)
* 58ae82a70a8915e4cae5b661ce4def47151d92af UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658517140
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir, @Nullable String jobClassName, String[] programArgs)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
Review comment:
Does this need to add `Configuration` parameter?
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnApplicationClusterEntryPoint.java
##########
@@ -129,16 +129,12 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = YarnEntrypointUtils.getUsrLibDir(configuration).orElse(null);
final File userApplicationJar = getUserApplicationJar(userLibDir, configuration);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJarFile(userApplicationJar)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesApplicationClusterEntrypoint.java
##########
@@ -105,22 +105,20 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
// No need to do pipelineJars validation if it is a PyFlink job.
if (!(PackagedProgramUtils.isPython(jobClassName)
|| PackagedProgramUtils.isPython(programArguments))) {
final List<File> pipelineJars =
KubernetesUtils.checkJarFileForApplicationMode(configuration);
Preconditions.checkArgument(pipelineJars.size() == 1, "Should only have one jar");
- retrieverBuilder.setJarFile(pipelineJars.get(0));
+ return PackagedProgramRetrieverImpl.create(
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesApplicationClusterEntrypoint.java
##########
@@ -105,22 +105,20 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
// No need to do pipelineJars validation if it is a PyFlink job.
if (!(PackagedProgramUtils.isPython(jobClassName)
|| PackagedProgramUtils.isPython(programArguments))) {
final List<File> pipelineJars =
KubernetesUtils.checkJarFileForApplicationMode(configuration);
Preconditions.checkArgument(pipelineJars.size() == 1, "Should only have one jar");
- retrieverBuilder.setJarFile(pipelineJars.get(0));
+ return PackagedProgramRetrieverImpl.create(
+ userLibDir, pipelineJars.get(0), jobClassName, programArguments);
}
- return retrieverBuilder.build();
+
+ return PackagedProgramRetrieverImpl.create(userLibDir, jobClassName, programArguments);
Review comment:
This doesn't pass the `Configuration`.
##########
File path: flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterEntryPoint.java
##########
@@ -111,13 +111,9 @@ private static PackagedProgram getPackagedProgram(
private static PackagedProgramRetriever getPackagedProgramRetriever(
final String[] programArguments, @Nullable final String jobClassName)
- throws IOException {
+ throws IOException, FlinkException {
final File userLibDir = ClusterEntrypointUtils.tryFindUserLibDirectory().orElse(null);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(userLibDir, jobClassName, programArguments);
Review comment:
This doesn't pass the `Configuration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661232763
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/JarManifestParserTest.java
##########
@@ -157,6 +158,24 @@ public void testFindOnlyEntryClassMultipleJarsWithSingleManifestEntry() throws I
is(equalTo(TestJob.class.getCanonicalName())));
}
+ @Test
+ public void testFindFirstManifestAttributeWithNoAttribute() throws IOException {
+ assertThat(
+ JarManifestParser.findFirstManifestAttribute(TestJob.getTestJobJar()).isPresent(),
+ is(false));
+ }
+
+ @Test
+ public void testFindFirstManifestAttributeWithAttributes() throws IOException {
+ Optional<String> optionalValue =
+ JarManifestParser.findFirstManifestAttribute(
+ TestJob.getTestJobJar(),
+ "some-attr",
Review comment:
what's this for?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProvider.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/**
+ * {@code FromClasspathEntryClassInformationProvider} assumes the passed job class being available
+ * on some classpath.
+ */
+public class FromClasspathEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} based on the passed job class
+ * and classpath.
+ *
+ * @param jobClassName The job's class name.
+ * @param classpath The classpath the job class should be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} instances collecting the
+ * necessary information.
+ * @throws FlinkException If the passed job class is not present on the passed classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider create(
+ String jobClassName, Iterable<URL> classpath) throws FlinkException {
+ Preconditions.checkNotNull(jobClassName, "No job class name passed.");
+ Preconditions.checkNotNull(classpath, "No classpath passed.");
+ if (!userClasspathContainsJobClass(jobClassName, classpath)) {
+ throw new FlinkException(
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ jobClassName));
+ }
+
+ return new FromClasspathEntryClassInformationProvider(jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} looking for the job class on the
+ * passed classpath.
Review comment:
Would be good to clarify _which_ job class it is looking for.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
+ temporaryFolder.newFolder("_test_user_dir_has_two_entry_classes");
+ copyJar(JOB_JAR_PATH, userDirHasTwoEntryClasses);
Review comment:
Why do we need these copy operations? Couldn't we construct the classpath from the original locations?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir, userClasspath, jarFile, jobClassName, programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable File userLibDir,
+ Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userLibDir != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
+ throws FlinkException {
+ if (jobClassName != null) {
+ return FromClasspathEntryClassInformationProvider
+ .createWithJobClassAssumingOnSystemClasspath(jobClassName);
+ }
+
+ try {
+ return FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ } catch (Throwable t) {
Review comment:
Why do you catch Throwables? Why is catching FlinkExceptions not sufficient?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithoutUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ assertThat(jobGraph.getClasspaths().isEmpty(), is(true));
+ }
+
+ @Test
+ public void testRetrieveFromJarFileWithUserLib()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getTestJobJar(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getUserJars(),
+ containsInAnyOrder(
+ new org.apache.flink.core.fs.Path(
+ classpathProvider.getTestJobJar().toURI())));
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
+
+ assertThat(
+ actualClasspath,
+ IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
+ }
+
+ @Test
+ public void testChildFirstDefaultConfiguration() throws FlinkException {
Review comment:
both this test and the one below should be renamed to clarify that they are about the class loaders
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
Review comment:
the comments are also weird; they claim the user directory is required, but testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName uses null for it and does the same assertions.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
Review comment:
Yeah its a bit weird; why are we referencing a deleted class?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
Review comment:
I don't understand why this would make it fail, nor what "it" that is not triggered actually is. Wouldn't it be more obvious if you did something like clearing the classpath?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
Review comment:
Why do we allow this to be nullable if there is a dedicated factory method where this argument does not exist?
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnApplicationClusterEntryPoint.java
##########
@@ -129,16 +129,12 @@ private static PackagedProgramRetriever getPackagedProgramRetriever(
final Configuration configuration,
final String[] programArguments,
@Nullable final String jobClassName)
- throws IOException {
+ throws FlinkException {
final File userLibDir = YarnEntrypointUtils.getUsrLibDir(configuration).orElse(null);
final File userApplicationJar = getUserApplicationJar(userLibDir, configuration);
- final ClassPathPackagedProgramRetriever.Builder retrieverBuilder =
- ClassPathPackagedProgramRetriever.newBuilder(programArguments)
- .setUserLibDirectory(userLibDir)
- .setJarFile(userApplicationJar)
- .setJobClassName(jobClassName);
- return retrieverBuilder.build();
+ return PackagedProgramRetrieverImpl.create(
+ userLibDir, userApplicationJar, jobClassName, programArguments, configuration);
Review comment:
Why is this not using the builder approach any longer? Wasn't that way more flexible?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
+ temporaryFolder.newFolder("_test_user_dir_has_two_entry_classes");
+ copyJar(JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(TEST_JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasTwoEntryClasses);
+ createTestFile(userDirHasTwoEntryClasses);
+
+ userDirHasEntryClass = temporaryFolder.newFolder("_test_user_dir_has_entry_class");
+ copyJar(JOB_JAR_PATH, userDirHasEntryClass);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasEntryClass);
+ createTestFile(userDirHasEntryClass);
+
+ userDirHasNotEntryClass = temporaryFolder.newFolder("_test_user_dir_has_not_entry_class");
+ copyJar(JOB_LIB_JAR_PATH, userDirHasNotEntryClass);
+ createTestFile(userDirHasNotEntryClass);
+
+ userDirWithTestJob = temporaryFolder.newFolder("_with_test_job");
+ copyJar(TEST_JOB_JAR_PATH, userDirWithTestJob);
+
+ userDirWithNoJarButTextFile = temporaryFolder.newFolder("_with_no_jar");
+ createTestFile(userDirWithNoJarButTextFile);
+ }
+
+ private static void copyJar(Path sourcePath, File targetDir) throws IOException {
+ Files.copy(sourcePath, targetDir.toPath().resolve(sourcePath.toFile().getName()));
+ }
+
+ private static void createTestFile(File targetDir) throws IOException {
+ Files.createFile(targetDir.toPath().resolve("test.txt"));
+ }
+
+ @Override
+ protected void after() {
+ temporaryFolder.delete();
+ resetSystemClasspath();
+ }
+
+ public String getJobClassName() {
+ return extractEntryClassNameFromJar(JOB_JAR_PATH.toFile());
+ }
+
+ public String getTestJobClassName() {
+ return extractEntryClassNameFromJar(getTestJobJar());
+ }
Review comment:
these really need better names.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramRetrieverImpl;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code EntryClassInformationProvider} provides information about the entry class that shall be
+ * used by the {@link PackagedProgramRetrieverImpl}.
+ */
+public interface EntryClassInformationProvider {
+
+ /**
+ * Returns the {@link File} referring to the Jar file that contains the job class or no {@code
+ * File} if the job class is located on some classpath.
+ *
+ * @return The {@code File} referring to the job's Jar archive or an empty {@code Optional} in
+ * case the job shall be extracted from the some classpath.
Review comment:
```suggestion
* case the job shall be extracted from the classpath.
```
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.commons.io.FilenameUtils;
+import org.hamcrest.CoreMatchers;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsCollectionContaining;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.Collection;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/**
+ * {@code FromClasspathEntryClassInformationProviderTest} tests {@link
+ * FromClasspathEntryClassInformationProvider}.
+ */
+public class FromClasspathEntryClassInformationProviderTest extends TestLogger {
+
+ @Rule public final ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testJobClassOnUserClasspathWithExplicitJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.create(
+ classpathProvider.getJobClassName(),
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testJobClassOnUserClasspathWithOnlyTestFileOnClasspath()
+ throws MalformedURLException, FlinkException {
+ // we want to check that the right exception is thrown if the user classpath is empty
+ FromClasspathEntryClassInformationProvider.create(
+ "SomeJobClassName", classpathProvider.getURLUserClasspathWithOnlyTextFile());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider.create(
+ null, classpathProvider.getURLUserClasspathWithEntryClass());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingUserClasspath() throws FlinkException {
+ FromClasspathEntryClassInformationProvider.create("jobClassName", null);
+ }
+
+ @Test
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithoutEntryClass());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnUserClasspath() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassNameAndMissingUserClasspath()
+ throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(null);
+ }
+
+ @Test
+ public void testJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithoutEntryClass();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test
+ public void testJarFromSystemClasspathSanityCheck() {
Review comment:
what do we need this test for? If you want to check that we can properly load something from the system CP without any modifications to it, then why not put a jar on the test classpath and try loading a job from that.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
Review comment:
nit: seems weird that a more complex case is handled before the plain case of 1 entry class
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramRetrieverImpl;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code EntryClassInformationProvider} provides information about the entry class that shall be
+ * used by the {@link PackagedProgramRetrieverImpl}.
Review comment:
Seems weird that this interface has a reference to an implementation of another interface.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.File;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+/**
+ * {@code FromJarEntryClassInformationProviderTest} tests {@link
+ * FromJarEntryClassInformationProvider}.
+ */
+public class FromJarEntryClassInformationProviderTest extends TestLogger {
+
+ @Test
+ public void testCustomJarFile() {
+ final File jarFile = new File("some/path/to/jar");
+ final String jobClassName = "JobClassName";
+ final FromJarEntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+
+ assertThat(testInstance.getJarFile().isPresent(), is(true));
+ assertThat(testInstance.getJarFile().get(), is(jarFile));
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(jobClassName));
+ }
+
+ @Test
+ public void testMissingJar() {
+ final String jobClassName = "JobClassName";
+ final EntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(null, jobClassName);
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(jobClassName));
+ }
+
+ @Test
+ public void testMissingJobClassName() {
+ final File jarFile = new File("some/path/to/jar");
+ final EntryClassInformationProvider testInstance =
+ FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, null);
+ assertThat(testInstance.getJarFile().isPresent(), is(true));
+ assertThat(testInstance.getJarFile().get(), is(jarFile));
+ assertThat(testInstance.getJobClassName().isPresent(), is(false));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testEitherJobClassNameOrJarHasToBeSet() {
+ FromJarEntryClassInformationProvider.createFromCustomJar(null, null);
+ }
+
+ @Test
+ public void testPythonJarFile() {
Review comment:
I think it's a highly questionable to add a test, then effectively disable it, and then file a ticket for fixing it before this PR is even merged.
Instead, I would not have added this test, and opened a ticket _afterwards_ for adding a test for the python side of things.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
+ temporaryFolder.newFolder("_test_user_dir_has_two_entry_classes");
+ copyJar(JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(TEST_JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasTwoEntryClasses);
+ createTestFile(userDirHasTwoEntryClasses);
+
+ userDirHasEntryClass = temporaryFolder.newFolder("_test_user_dir_has_entry_class");
+ copyJar(JOB_JAR_PATH, userDirHasEntryClass);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasEntryClass);
+ createTestFile(userDirHasEntryClass);
+
+ userDirHasNotEntryClass = temporaryFolder.newFolder("_test_user_dir_has_not_entry_class");
+ copyJar(JOB_LIB_JAR_PATH, userDirHasNotEntryClass);
+ createTestFile(userDirHasNotEntryClass);
+
+ userDirWithTestJob = temporaryFolder.newFolder("_with_test_job");
+ copyJar(TEST_JOB_JAR_PATH, userDirWithTestJob);
+
+ userDirWithNoJarButTextFile = temporaryFolder.newFolder("_with_no_jar");
+ createTestFile(userDirWithNoJarButTextFile);
+ }
+
+ private static void copyJar(Path sourcePath, File targetDir) throws IOException {
+ Files.copy(sourcePath, targetDir.toPath().resolve(sourcePath.toFile().getName()));
+ }
+
+ private static void createTestFile(File targetDir) throws IOException {
+ Files.createFile(targetDir.toPath().resolve("test.txt"));
+ }
+
+ @Override
+ protected void after() {
+ temporaryFolder.delete();
+ resetSystemClasspath();
+ }
+
+ public String getJobClassName() {
+ return extractEntryClassNameFromJar(JOB_JAR_PATH.toFile());
+ }
+
+ public String getTestJobClassName() {
+ return extractEntryClassNameFromJar(getTestJobJar());
+ }
+
+ public File getTestJobJar() {
+ return TEST_JOB_JAR_PATH.toFile();
+ }
+
+ private static String extractEntryClassNameFromJar(File f) {
+ try {
+ return JarManifestParser.findFirstManifestAttribute(
+ f, PackagedProgram.MANIFEST_ATTRIBUTE_MAIN_CLASS)
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "The passed file does not contain a main class: "
+ + f.getAbsolutePath()));
+ } catch (Throwable t) {
+ throw new AssertionError(
+ "Something went wrong with retrieving the main class from "
+ + f.getAbsolutePath(),
+ t);
+ }
+ }
+
+ public File getUserDirectoryWithNoEntryClass() {
+ return userDirHasNotEntryClass;
+ }
+
+ public File getUserDirectoryWithOneEntryClass() {
+ return userDirHasEntryClass;
+ }
+
+ public File getUserDirectoryWithTwoEntryClasses() {
+ return userDirHasTwoEntryClasses;
+ }
+
+ public File getUserDirectoryWithTestJob() {
+ return userDirWithTestJob;
+ }
+
+ public String[] getTestJobArgs(String expectedSuffix) {
+ return new String[] {"--arg", expectedSuffix};
+ }
+
+ public Iterable<URL> getURLUserClasspathWithTwoEntryClasses() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasTwoEntryClasses);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithEntryClass() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasEntryClass);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithoutEntryClass() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasNotEntryClass);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithTestJob() throws MalformedURLException {
+ return getURLUserClasspath(userDirWithTestJob);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithOnlyTextFile() throws MalformedURLException {
+ return getURLUserClasspath(userDirWithNoJarButTextFile);
+ }
+
+ public void setSystemClasspathWithTwoEntryClasses() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ public void setSystemClasspathWithEntryClass() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithEntryClass());
+ }
+
+ public void setSystemClasspathWithoutEntryClass() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithoutEntryClass());
+ }
+
+ public void setSystemClasspathWithTestJob() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithTestJob());
+ }
+
+ private void setSystemClasspath(Iterable<URL> classpath) {
Review comment:
If this is called twice the original classpath will not be properly restored; capture the original value in the constructor instead
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
Review comment:
So if 2 entry points are on the user CP we don't fail?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
Review comment:
The name of the class was a bit surprising; I expected it at first sight to be some `Iterable<URL>/String` provider, not a hodgepodge of getters and setters.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
Review comment:
I don't understand the comment, please elaborate.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
Review comment:
which method are you referencing?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
Review comment:
Aren't these irrelevant since we explicitly provide the job classname anyway?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
Review comment:
It doesn't matter where the setter is used in production or not, the purpose of the test is to ensure behavior despite _changes_ in production code (i.e., if the setter were used).
Where else do you test that, if 2 entrypoints are on the CP and the user specifies the entrypoint class, that the user argument is honored?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
Review comment:
What is actually being derived from anything? It seems like all we're testing here is that the stuff passed via the method is eventually assigned.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
Review comment:
Why should we silently ignore this? Wouldn't it be a bug in the ClassPathProvider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662113132
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.commons.io.FilenameUtils;
+import org.hamcrest.CoreMatchers;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsCollectionContaining;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.Collection;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/**
+ * {@code FromClasspathEntryClassInformationProviderTest} tests {@link
+ * FromClasspathEntryClassInformationProvider}.
+ */
+public class FromClasspathEntryClassInformationProviderTest extends TestLogger {
+
+ @Rule public final ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testJobClassOnUserClasspathWithExplicitJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.create(
+ classpathProvider.getJobClassName(),
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testJobClassOnUserClasspathWithOnlyTestFileOnClasspath()
+ throws MalformedURLException, FlinkException {
+ // we want to check that the right exception is thrown if the user classpath is empty
+ FromClasspathEntryClassInformationProvider.create(
+ "SomeJobClassName", classpathProvider.getURLUserClasspathWithOnlyTextFile());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider.create(
+ null, classpathProvider.getURLUserClasspathWithEntryClass());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingUserClasspath() throws FlinkException {
+ FromClasspathEntryClassInformationProvider.create("jobClassName", null);
+ }
+
+ @Test
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithoutEntryClass());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnUserClasspath() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassNameAndMissingUserClasspath()
+ throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(null);
+ }
+
+ @Test
+ public void testJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithoutEntryClass();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test
+ public void testJarFromSystemClasspathSanityCheck() {
Review comment:
You have a point. I agree. But I would also vote for keeping it like this for now as I want to avoid opening another refactoring sub-project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-869949832
> @XComp , thanks for your efforts to update the pull request again me. I have reviewed the changes of the refactors. I have confusion about why not validate whether to retrever job graph in tests, which originally validate through retrieveJobGraph.
Thanks for your review. I addressed two of the changes and rebased the branch. About your questions with not using the retriever job graph. Building the `PackagedProgram` will trigger the class loading internally. This is problematic for testing the system classpath functionality.
I tried to use legacy test code as much as possible. New tests, in contrast, avoid loading the class from classpath but rather check whether the correct class is selected. That's why the test implementation focuses on the `EntryClassInformationProvider`. It's more of a unit test rather than a test testing multiple classes interacting with each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868257439
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 4cbaee738f213985ae8d6850b36005e0ebafd5bf (Fri Jun 25 06:26:39 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-874583048
Backport PRs for `release-1.11` (here I only applied the configuration fix excluding the refactoring), `release-1.12` and `release-1.13` are created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662124087
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -126,10 +126,9 @@ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws Fl
}
@Test
- public void testDeriveEntryClassInformationForCustomJar()
- throws FlinkException, MalformedURLException {
- // make loading from system classpath fail to make sure that it's not triggered
- multipleEntryClassesClasspathProvider.setSystemClasspath();
+ public void testDeriveEntryClassInformationForCustomJar() throws FlinkException {
+ // clearing the system classpath to make sure that no data is collected from there
+ System.setProperty("java.class.path", "");
Review comment:
Good point. I will update it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-874020720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 349d6855c21fed4d4368c56646551c3524437384 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531)
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661989890
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
+ throws IOException, FlinkException, ProgramInvocationException {
+ // TODO: I don't get this test: It feels like it does not add any extra value - it does the
+ // same like testTooManyEntryClassesOnSystemClasspath
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ // Both a class name is specified and a JAR "is" on the class path
+ // The class name should have precedence.
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(jobGraph.getName(), is(classpathProvider.getTestJobClassName() + "-suffix"));
+ }
+
+ @Test
+ public void testSavepointRestoreSettings()
+ throws FlinkException, IOException, ProgramInvocationException {
+ final Configuration configuration = new Configuration();
+ final SavepointRestoreSettings savepointRestoreSettings =
+ SavepointRestoreSettings.forPath("foobar", true);
+ final JobID jobId = new JobID();
+
+ configuration.setString(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+ SavepointRestoreSettings.toConfiguration(savepointRestoreSettings, configuration);
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, configuration);
+
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(savepointRestoreSettings));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
+ e,
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ classpathProvider.getTestJobClassName()))
+ .isPresent(),
+ is(true));
+ }
+ }
+
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ null, "NotExistingClass", new String[0], new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnUserClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testTooManyEntryClassesOnSystemClasspath() throws FlinkException {
+ PackagedProgramRetrieverImpl.create(null, null, new String[0], new Configuration());
+ }
+
+ @Test
+ public void testTooManyEntryClassesOnUserClasspath() throws FlinkException, IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ final PackagedProgramRetrieverImpl retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ classpathProvider.getTestJobClassName(),
+ new String[0],
+ new Configuration());
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getTestJobClassName()));
+ }
+
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getTestJobArgs("suffix"),
+ new Configuration());
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
Review comment:
The `ClasspathProvider` provides all the resources that are on the classpath (including a text file in most of the cases). The production code filters them out in [PackagedProgramRetrieverImpl.extractUserClasspath](https://github.com/XComp/flink/blob/e9a2e57adf4f45958e6fa0e9e9f0f7afac50d01e/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java#L208). The filtering here is just the corresponding test code checking this filtering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661994274
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/JarManifestParserTest.java
##########
@@ -157,6 +158,24 @@ public void testFindOnlyEntryClassMultipleJarsWithSingleManifestEntry() throws I
is(equalTo(TestJob.class.getCanonicalName())));
}
+ @Test
+ public void testFindFirstManifestAttributeWithNoAttribute() throws IOException {
+ assertThat(
+ JarManifestParser.findFirstManifestAttribute(TestJob.getTestJobJar()).isPresent(),
+ is(false));
+ }
+
+ @Test
+ public void testFindFirstManifestAttributeWithAttributes() throws IOException {
+ Optional<String> optionalValue =
+ JarManifestParser.findFirstManifestAttribute(
+ TestJob.getTestJobJar(),
+ "some-attr",
Review comment:
Fair enough - it doesn't add any value but might confuse the reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662999747
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProvider.java
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/**
+ * {@code FromClasspathEntryClassInformationProvider} assumes the passed job class being available
+ * on some classpath.
+ */
+public class FromClasspathEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} based on the passed job class
+ * and classpath.
+ *
+ * @param jobClassName The job's class name.
+ * @param classpath The classpath the job class should be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} instances collecting the
+ * necessary information.
+ * @throws FlinkException If the passed job class is not present on the passed classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider create(
+ String jobClassName, Iterable<URL> classpath) throws FlinkException {
+ Preconditions.checkNotNull(jobClassName, "No job class name passed.");
+ Preconditions.checkNotNull(classpath, "No classpath passed.");
+ if (!userClasspathContainsJobClass(jobClassName, classpath)) {
+ throw new FlinkException(
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ jobClassName));
+ }
+
+ return new FromClasspathEntryClassInformationProvider(jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} looking for the entry class
+ * providing the main method on the passed classpath.
+ *
+ * @param classpath The classpath the job class is expected to be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} providing the job class found
+ * on the passed classpath.
+ * @throws IOException If some Jar listed on the classpath wasn't accessible.
Review comment:
Good catch. I moved the IOException handling out of the private class into `PackagedProgramRetrieverImpl` analogous to how it's done for the other classpath accesses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662082340
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir, userClasspath, jarFile, jobClassName, programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable File userLibDir,
+ Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userLibDir != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
+ throws FlinkException {
+ if (jobClassName != null) {
+ return FromClasspathEntryClassInformationProvider
+ .createWithJobClassAssumingOnSystemClasspath(jobClassName);
+ }
+
+ try {
+ return FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ } catch (Throwable t) {
Review comment:
You could've just catched `Exception` to cover these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663818788
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
Review comment:
Fair enough, the information is also given in `@return` docs. So, I guess, that's good enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662085256
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -328,18 +329,15 @@ public void testFailIfJobDirDoesNotHaveEntryClass() {
}
}
- @Test
- public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException, IOException {
- classpathProvider.setSystemClasspathWithoutEntryClass();
- final PackagedProgramRetrieverImpl retriever =
- PackagedProgramRetrieverImpl.create(
- null,
- classpathProvider.getTestJobClassName(),
- new String[0],
- new Configuration());
- assertThat(
- retriever.getPackagedProgram().getMainClassName(),
- is(classpathProvider.getTestJobClassName()));
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
Review comment:
Correct, but I would keep it in as an additional test in case we decide to make the code more strict (as mentioned in [this comment](https://github.com/apache/flink/pull/16286/files#r661394188)).
Right now, I lean towards keeping it like it was implemented in the old implementation to not open any hidden problems which might make the PR become even bigger. Instead, I'd create a Flink issue to resolve that. But I wanted to get your opinions on that one. I wasn't 100% sure whether it's actually a desired behavior not checking the class existence on the system classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661412119
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/JarManifestParserTest.java
##########
@@ -157,6 +158,24 @@ public void testFindOnlyEntryClassMultipleJarsWithSingleManifestEntry() throws I
is(equalTo(TestJob.class.getCanonicalName())));
}
+ @Test
+ public void testFindFirstManifestAttributeWithNoAttribute() throws IOException {
+ assertThat(
+ JarManifestParser.findFirstManifestAttribute(TestJob.getTestJobJar()).isPresent(),
+ is(false));
+ }
+
+ @Test
+ public void testFindFirstManifestAttributeWithAttributes() throws IOException {
+ Optional<String> optionalValue =
+ JarManifestParser.findFirstManifestAttribute(
+ TestJob.getTestJobJar(),
+ "some-attr",
Review comment:
I added the tests because I made the `JarManifestParser.findFirstManifestAttribute` publicly accessible. Hence, I thought of adding test cases for it might be good as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
* 44c00395058d15d3fdee19f54c7c5261db213234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777)
* 7fe6639f640124b804662177ea510334834f761f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661979916
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
Review comment:
The method that's being tested. I updated the comment to make this more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663782747
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
Review comment:
The jar doesn't have to be explicitly specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
* 44c00395058d15d3fdee19f54c7c5261db213234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617)
* d5aa380c6fbe941651d2b095a6d80e92fee04ab3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662012221
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProviderTest.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.commons.io.FilenameUtils;
+import org.hamcrest.CoreMatchers;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsCollectionContaining;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.Collection;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+/**
+ * {@code FromClasspathEntryClassInformationProviderTest} tests {@link
+ * FromClasspathEntryClassInformationProvider}.
+ */
+public class FromClasspathEntryClassInformationProviderTest extends TestLogger {
+
+ @Rule public final ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testJobClassOnUserClasspathWithExplicitJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.create(
+ classpathProvider.getJobClassName(),
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = FlinkException.class)
+ public void testJobClassOnUserClasspathWithOnlyTestFileOnClasspath()
+ throws MalformedURLException, FlinkException {
+ // we want to check that the right exception is thrown if the user classpath is empty
+ FromClasspathEntryClassInformationProvider.create(
+ "SomeJobClassName", classpathProvider.getURLUserClasspathWithOnlyTextFile());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingJobClassName()
+ throws MalformedURLException, FlinkException {
+ FromClasspathEntryClassInformationProvider.create(
+ null, classpathProvider.getURLUserClasspathWithEntryClass());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithMissingUserClasspath() throws FlinkException {
+ FromClasspathEntryClassInformationProvider.create("jobClassName", null);
+ }
+
+ @Test
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithEntryClass());
+
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassOnUserClasspathWithoutExplicitJobClassName() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithoutEntryClass());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnUserClasspath() throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testJobClassOnUserClasspathWithoutExplicitJobClassNameAndMissingUserClasspath()
+ throws IOException {
+ FromClasspathEntryClassInformationProvider.createFromClasspath(null);
+ }
+
+ @Test
+ public void testJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+ FromClasspathEntryClassInformationProvider testInstance =
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ assertThat(testInstance.getJobClassName().isPresent(), is(true));
+ assertThat(testInstance.getJobClassName().get(), is(classpathProvider.getJobClassName()));
+ assertThat(testInstance.getJarFile().isPresent(), is(false));
+ }
+
+ @Test(expected = NoSuchElementException.class)
+ public void testMissingJobClassNameFromSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithoutEntryClass();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testTooManyMainMethodsOnSystemClasspath() throws IOException {
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+ FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ }
+
+ @Test
+ public void testJarFromSystemClasspathSanityCheck() {
Review comment:
That's a sanity check I copied over from the old `ClassPathPackagedProgramRetrieverTest`. I thought it's reasonable to check whether the jar loading without modifying the `java.class.path` System property. Isn't that essentially the same as what you suggest? ...just with less code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
* 2a812aa5c826e278b0679572a5b37025aecbb24b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895)
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658777887
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
Review comment:
I moved the logic which was included in the legacy `ClassPathPackagedProgramRetrieverTest` into its own class making it available as a `@Rule` annotated member
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 58ae82a70a8915e4cae5b661ce4def47151d92af Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714)
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol merged pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol merged pull request #16286:
URL: https://github.com/apache/flink/pull/16286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r659659300
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
Review comment:
I left the test here for now for transparency reasons. I think we should remove it. The legacy test was testing something that was never used in production: We always relied on `ClassPathPackagedProgramRetriever.JarsOnClassPath`. [setJarsOnClassPath](https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ClassPathPackagedProgramRetriever.java#L269) was only set within the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r659714742
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code EntryClassInformationProvider} provides information about the entry class that shall be
+ * used by the {@link org.apache.flink.client.program.PackagedProgramRetrieverImpl}.
Review comment:
Could `{@link org.apache.flink.client.program.PackagedProgramRetrieverImpl}` update to `{@link PackagedProgramRetrieverImpl}`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662243494
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir, userClasspath, jarFile, jobClassName, programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable File userLibDir,
+ Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userLibDir != null) {
Review comment:
I don't know why I didn't see the obvious. 🤦 But yeah, that's way better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661131964
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
+ temporaryFolder.newFolder("_test_user_dir_has_two_entry_classes");
+ copyJar(JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(TEST_JOB_JAR_PATH, userDirHasTwoEntryClasses);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasTwoEntryClasses);
+ createTestFile(userDirHasTwoEntryClasses);
+
+ userDirHasEntryClass = temporaryFolder.newFolder("_test_user_dir_has_entry_class");
+ copyJar(JOB_JAR_PATH, userDirHasEntryClass);
+ copyJar(JOB_LIB_JAR_PATH, userDirHasEntryClass);
+ createTestFile(userDirHasEntryClass);
+
+ userDirHasNotEntryClass = temporaryFolder.newFolder("_test_user_dir_has_not_entry_class");
+ copyJar(JOB_LIB_JAR_PATH, userDirHasNotEntryClass);
+ createTestFile(userDirHasNotEntryClass);
+
+ userDirWithTestJob = temporaryFolder.newFolder("_with_test_job");
+ copyJar(TEST_JOB_JAR_PATH, userDirWithTestJob);
+
+ userDirWithNoJarButTextFile = temporaryFolder.newFolder("_with_no_jar");
+ createTestFile(userDirWithNoJarButTextFile);
+ }
+
+ private static void copyJar(Path sourcePath, File targetDir) throws IOException {
+ Files.copy(sourcePath, targetDir.toPath().resolve(sourcePath.toFile().getName()));
+ }
+
+ private static void createTestFile(File targetDir) throws IOException {
+ Files.createFile(targetDir.toPath().resolve("test.txt"));
+ }
+
+ @Override
+ protected void after() {
+ temporaryFolder.delete();
+ resetSystemClasspath();
+ }
+
+ public String getJobClassName() {
+ return extractEntryClassNameFromJar(JOB_JAR_PATH.toFile());
+ }
+
+ public String getTestJobClassName() {
+ return extractEntryClassNameFromJar(TEST_JOB_JAR_PATH.toFile());
+ }
+
+ private static String extractEntryClassNameFromJar(File f) {
+ try {
+ return JarManifestParser.findFirstManifestAttribute(
+ f, PackagedProgram.MANIFEST_ATTRIBUTE_MAIN_CLASS)
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "The passed file does not contain a main class: "
+ + f.getAbsolutePath()));
+ } catch (Throwable t) {
+ throw new AssertionError(
+ "Something went wrong with retrieving the main class from "
+ + f.getAbsolutePath(),
+ t);
+ }
+ }
+
+ public File getUserDirectoryWithNoEntryClass() {
+ return userDirHasNotEntryClass;
+ }
+
+ public File getUserDirectoryWithOneEntryClass() {
+ return userDirHasEntryClass;
+ }
+
+ public File getUserDirectoryWithTwoEntryClasses() {
+ return userDirHasTwoEntryClasses;
+ }
+
+ public File getUserDirectoryWithTestJob() {
+ return userDirWithTestJob;
+ }
+
+ public String[] getTestJobArgs(String expectedSuffix) {
+ return new String[] {"--arg", expectedSuffix};
+ }
+
+ public Iterable<URL> getURLUserClasspathWithTwoEntryClasses() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasTwoEntryClasses);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithEntryClass() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasEntryClass);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithoutEntryClass() throws MalformedURLException {
+ return getURLUserClasspath(userDirHasNotEntryClass);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithTestJob() throws MalformedURLException {
+ return getURLUserClasspath(userDirWithTestJob);
+ }
+
+ public Iterable<URL> getURLUserClasspathWithOnlyTextFile() throws MalformedURLException {
+ return getURLUserClasspath(userDirWithNoJarButTextFile);
+ }
+
+ public void setSystemClasspathWithTwoEntryClasses() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithTwoEntryClasses());
+ }
+
+ public void setSystemClasspathWithEntryClass() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithEntryClass());
+ }
+
+ public void setSystemClasspathWithoutEntryClass() throws MalformedURLException {
+ setSystemClasspath(getURLUserClasspathWithoutEntryClass());
+ }
+
+ public void setSystemClasspathWithTestJob() throws MalformedURLException {
Review comment:
Never used. Right?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
Review comment:
IIUC, we do not need to inheritance from `FromClasspathEntryClassInformationProvider`.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ File jarFile, String jobClassName) {
Review comment:
Maybe add `Nullable` here is better.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -372,6 +372,35 @@ public void testRetrieveCorrectUserClasspathsWithoutSpecifiedEntryClass()
IsIterableContainingInAnyOrder.containsInAnyOrder(expectedClasspath.toArray()));
}
+ @Test
+ public void testRetrieveCorrectUserClasspathsWithSpecifiedEntryClass()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ classpathProvider.getJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"));
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+ final List<String> actualClasspath =
+ jobGraph.getClasspaths().stream().map(URL::toString).collect(Collectors.toList());
+
+ final Path workingDirectory = FileUtils.getCurrentWorkingDirectory();
+ final List<String> expectedClasspath = new ArrayList<>();
+ for (File file : classpathProvider.getUserDirectoryWithOneEntryClass().listFiles()) {
+ if (!file.getName().endsWith("jar")) {
+ // only jars are expected
+ continue;
+ }
+
+ Path relativePath = FileUtils.relativizePath(workingDirectory, file.toPath());
+ expectedClasspath.add(FileUtils.toURL(relativePath).toString());
+ }
Review comment:
Could be deduplicated by introducing a new method.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
Review comment:
Actually, this comment confused me. Do you mean the related tests from legacy `ClassPathPackagedProgramRetrieverTest` will be removed or is outdated?
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProviderTest.java
##########
@@ -72,9 +73,13 @@ public void testEitherJobClassNameOrJarHasToBeSet() {
FromJarEntryClassInformationProvider.createFromCustomJar(null, null);
}
- @Ignore // FLINK_OPT_DIR has to be set
@Test
public void testPythonJarFile() {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
Review comment:
Maybe we need to use `System.getenv`. Refer to `PackagedProgramUtils#getPythonJar`.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -285,6 +287,26 @@ public void testSavepointRestoreSettings()
assertThat(jobGraph.getJobID(), is(jobId));
}
+ @Test
+ public void testFailIfJobDirDoesNotHaveEntryClass() throws IOException {
+ try {
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithNoEntryClass(),
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs("suffix"));
+ fail("This case should throw exception !");
+ } catch (FlinkException e) {
+ assertThat(
+ ExceptionUtils.findThrowableWithMessage(
Review comment:
I think we could use `assertThat(e, FlinkMatchers.containsMessage(msg))` here.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
Review comment:
I guess you mean `{@code FromJarEntryClassInformationProvider}` returns a `{@link EntryClassInformationProvider}`.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
Review comment:
Yes. The legacy `ClassPathPackagedProgramRetriever.Builder#setJarsOnClassPath` is never used in production code. And it seems not have some benefits to specify the jar name on classpath instead of entry class.
+1 to remove this test.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -328,18 +329,15 @@ public void testFailIfJobDirDoesNotHaveEntryClass() {
}
}
- @Test
- public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException, IOException {
- classpathProvider.setSystemClasspathWithoutEntryClass();
- final PackagedProgramRetrieverImpl retriever =
- PackagedProgramRetrieverImpl.create(
- null,
- classpathProvider.getTestJobClassName(),
- new String[0],
- new Configuration());
- assertThat(
- retriever.getPackagedProgram().getMainClassName(),
- is(classpathProvider.getTestJobClassName()));
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
Review comment:
IIUC, this test could be subsumed by `testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName`.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -215,20 +215,6 @@ public void testCreateWithUserLibDir() throws FlinkException {
is(classpathProvider.getJobClassName()));
}
- @Test
- public void testCreateWithoutUserLibDir() throws IOException, FlinkException {
Review comment:
I could not fully understand why we need to remove the `testCreateWithoutUserLibDir` here. IIUC, it is used to verify the system classpath pickup.
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
Review comment:
I agree with that we also need to check the system classpath contains specified JobClass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
* 2a812aa5c826e278b0679572a5b37025aecbb24b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* e3b0831a1476a562452576d854dc2d43a3ab3e21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523)
* 349d6855c21fed4d4368c56646551c3524437384 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661757587
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
Review comment:
We're testing the code path where a jar is set. Admittedly, it's a quite basic test. But I tried to cover all code paths of `createEntryClassInformationProvider`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r658541693
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
Review comment:
Is this a correct concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r660042732
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/JarManifestParser.java
##########
@@ -118,7 +118,7 @@ static JarFileWithEntryClass findOnlyEntryClass(Iterable<File> jarFiles) throws
* @return Optional holding value of first found attribute
* @throws IOException If there is an error accessing the JAR
*/
- private static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
+ public static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
Review comment:
I use it in `ClasspathProvider.extractEntryClassNameFromJar`. Making the method `protected` would make it unavailable there.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} returns a is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
Review comment:
Where do you see the benefits of using inheritance here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662024758
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(
+ informationProvider.getJobClassName(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void deriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
+ null,
+ // we have to specify the job class - otherwise the call would fail due to
+ // two main method being present
+ classpathProvider.getJobClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithNoJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithEntryClass(),
+ null,
+ // no job class name is specified which enables looking for the entry class
+ // on the user classpath
+ null,
+ new String[0]);
+ assertThat(informationProvider.getJobClassName(), is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testCreateWithUserLibDir() throws FlinkException {
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithOneEntryClass(),
+ null,
+ classpathProvider.getJobClassName(),
+ new String[0],
+ new Configuration());
+
+ // the right information is picked up without any error
+ assertThat(
+ retriever.getPackagedProgram().getMainClassName(),
+ is(classpathProvider.getJobClassName()));
+ }
+
+ @Test
+ public void testJobGraphRetrieval()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final int parallelism = 42;
+ final JobID jobId = new JobID();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, parallelism);
+ configuration.set(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID, jobId.toHexString());
+
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retriever =
+ PackagedProgramRetrieverImpl.create(
+ null,
+ classpathProvider.getTestJobClassName(),
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retriever, configuration);
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ assertThat(jobGraph.getSavepointRestoreSettings(), is(SavepointRestoreSettings.none()));
+ assertThat(jobGraph.getMaximumParallelism(), is(parallelism));
+ assertThat(jobGraph.getJobID(), is(jobId));
+ }
+
+ @Test
+ public void testJobGraphRetrievalFromJar()
+ throws IOException, FlinkException, ProgramInvocationException {
+ final String expectedSuffix = "suffix";
+ final PackagedProgramRetriever retrieverUnderTest =
+ PackagedProgramRetrieverImpl.create(
+ classpathProvider.getUserDirectoryWithTestJob(),
+ null,
+ null,
+ classpathProvider.getTestJobArgs(expectedSuffix),
+ new Configuration());
+
+ final JobGraph jobGraph = retrieveJobGraph(retrieverUnderTest, new Configuration());
+
+ assertThat(
+ jobGraph.getName(),
+ is(classpathProvider.getTestJobClassName() + "-" + expectedSuffix));
+ }
+
+ @Test
+ public void testJobGraphRetrievalJobClassNameHasPrecedenceOverClasspath()
Review comment:
This test only succeeds in the old implementation because we're not checking whether the given job class name is on the system classpath. But @zentol is right with checking that the parameters are acknowledged for this specific case. Although, it should be moved to `PackagedProgramTest`, I guess. But I leave it here (and just rename it) because I feel that that refactoring the testcases in this matter is out of scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662807298
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import java.io.File;
+import java.util.Optional;
+
+/** {@code EntryClassInformationProvider} provides information about the entry class. */
+public interface EntryClassInformationProvider {
+
+ /**
+ * Returns the {@link File} referring to the Jar file that contains the job class or no {@code
+ * File} if the job class is located on the classpath.
+ *
+ * @return The {@code File} referring to the job's Jar archive or an empty {@code Optional} in
+ * case the job shall be extracted from the classpath.
+ */
+ Optional<File> getJarFile();
+
+ /**
+ * Returns the name of the job class.
Review comment:
Needs a remark for when the optional is empty. Currently possible guesses:
- it is empty if getJarFile returns an empty optional
- it is empty if no entry point can be found on the classpath
- it is empty if both above conditions apply
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromClasspathEntryClassInformationProvider.java
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/**
+ * {@code FromClasspathEntryClassInformationProvider} assumes the passed job class being available
+ * on some classpath.
+ */
+public class FromClasspathEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} based on the passed job class
+ * and classpath.
+ *
+ * @param jobClassName The job's class name.
+ * @param classpath The classpath the job class should be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} instances collecting the
+ * necessary information.
+ * @throws FlinkException If the passed job class is not present on the passed classpath.
+ */
+ public static FromClasspathEntryClassInformationProvider create(
+ String jobClassName, Iterable<URL> classpath) throws FlinkException {
+ Preconditions.checkNotNull(jobClassName, "No job class name passed.");
+ Preconditions.checkNotNull(classpath, "No classpath passed.");
+ if (!userClasspathContainsJobClass(jobClassName, classpath)) {
+ throw new FlinkException(
+ String.format(
+ "Could not find the provided job class (%s) in the user lib directory.",
+ jobClassName));
+ }
+
+ return new FromClasspathEntryClassInformationProvider(jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromClasspathEntryClassInformationProvider} looking for the entry class
+ * providing the main method on the passed classpath.
+ *
+ * @param classpath The classpath the job class is expected to be part of.
+ * @return The {@code FromClasspathEntryClassInformationProvider} providing the job class found
+ * on the passed classpath.
+ * @throws IOException If some Jar listed on the classpath wasn't accessible.
Review comment:
can we align this behavior with `create`? userClasspathContainsJobClass throws a RuntimeException if a jar isn't accessible.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir == null ? null : userClasspath,
+ jarFile,
+ jobClassName,
+ programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userClasspath != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
Review comment:
```suggestion
@VisibleForTesting
static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
```
?
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir == null ? null : userClasspath,
+ jarFile,
+ jobClassName,
+ programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userClasspath != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
+ throws FlinkException {
+ if (jobClassName != null) {
+ return FromClasspathEntryClassInformationProvider
+ .createWithJobClassAssumingOnSystemClasspath(jobClassName);
+ }
+
+ try {
+ return FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ } catch (IOException | NoSuchElementException | IllegalArgumentException t) {
+ throw createGenericFlinkException(t);
+ }
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider fromUserClasspath(
+ @Nullable String jobClassName, Iterable<URL> userClasspath) throws FlinkException {
+ if (jobClassName != null) {
+ return FromClasspathEntryClassInformationProvider.create(jobClassName, userClasspath);
+ }
+
+ try {
+ return FromClasspathEntryClassInformationProvider.createFromClasspath(userClasspath);
+ } catch (IOException e) {
+ throw createGenericFlinkException(e);
+ }
+ }
+
+ private static FlinkException createGenericFlinkException(Throwable t) {
+ return new FlinkException("An error occurred while access the provided classpath.", t);
+ }
+
+ private PackagedProgramRetrieverImpl(
+ EntryClassInformationProvider entryClassInformationProvider,
+ String[] programArguments,
+ List<URL> userClasspath,
+ Configuration configuration) {
+ this.entryClassInformationProvider =
+ Preconditions.checkNotNull(
+ entryClassInformationProvider, "No EntryClassInformationProvider passed.");
+ this.programArguments =
+ Preconditions.checkNotNull(programArguments, "No program parameter array passed.");
+ this.userClasspath = Preconditions.checkNotNull(userClasspath, "No user classpath passed.");
+ this.configuration =
+ Preconditions.checkNotNull(configuration, "No Flink configuration was passed.");
+ }
+
+ @Override
+ public PackagedProgram getPackagedProgram() throws FlinkException {
+ try {
+ final PackagedProgram.Builder packagedProgramBuilder =
+ PackagedProgram.newBuilder()
+ .setUserClassPaths(userClasspath)
+ .setArguments(programArguments)
+ .setConfiguration(configuration);
+
+ entryClassInformationProvider
+ .getJobClassName()
+ .ifPresent(packagedProgramBuilder::setEntryPointClassName);
+ entryClassInformationProvider
+ .getJarFile()
+ .ifPresent(packagedProgramBuilder::setJarFile);
+
+ return packagedProgramBuilder.build();
+ } catch (ProgramInvocationException e) {
+ throw new FlinkException("Could not load the provided entrypoint class.", e);
+ }
+ }
+
+ private static List<URL> extractUserClasspath(@Nullable File jobDir) throws IOException {
Review comment:
rename: jobDir -> userLibDir for consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662129429
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -197,8 +197,8 @@ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
// the user directory must be specified
multipleEntryClassesClasspathProvider.getDirectory(),
- // the user classpath is derived from the user directory outside of the
- // method
+ // the user classpath is derived from the user directory outside of
+ // createEntryClassInformationProvider
Review comment:
I find it a bit confusing to pass in the user directory and the user classpath. But this was required to match the original functionality. That's why I added the comment to make it obvious that the user classpath and the user directory are actually referring to the same thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r660042732
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/JarManifestParser.java
##########
@@ -118,7 +118,7 @@ static JarFileWithEntryClass findOnlyEntryClass(Iterable<File> jarFiles) throws
* @return Optional holding value of first found attribute
* @throws IOException If there is an error accessing the JAR
*/
- private static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
+ public static Optional<String> findFirstManifestAttribute(File jarFile, String... attributes)
Review comment:
I use it in `ClasspathProvider.extractEntryClassNameFromJar`. Making the method `protected` would make it unavailable there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
* 44c00395058d15d3fdee19f54c7c5261db213234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-869566357
@SteNicholas @wangyang0918 I updated the PR. It would be nice to get your thoughts on that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661761328
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
Review comment:
Good idea, that's more explicit. I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766)
* 9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771)
* 44c00395058d15d3fdee19f54c7c5261db213234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777)
* 7fe6639f640124b804662177ea510334834f761f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 7fe6639f640124b804662177ea510334834f761f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
* 5e03a511663c7cadda1e3f037c30ba3dfda1b05e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842)
* 99a22eedf2d53a35559d9fe31fac76f76766faac UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661755496
##########
File path: flink-clients/src/test/java/org/apache/flink/client/testjar/ClasspathProvider.java
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.client.deployment.application.JarManifestParser;
+import org.apache.flink.client.program.PackagedProgram;
+
+import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+/** {@code ClasspathProvider} offers methods to generate classpaths based on actual jars. */
+public class ClasspathProvider extends ExternalResource {
+
+ private static final String CLASSPATH_PROPERTY_NAME = "java.class.path";
+
+ private static final Path TEST_JOB_JAR_PATH = Paths.get("target", "maven-test-jar.jar");
+
+ private static final Path JOB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-jar.jar");
+ private static final Path JOB_LIB_JAR_PATH =
+ Paths.get("target", "maven-test-user-classloader-job-lib-jar.jar");
+
+ private final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File userDirHasTwoEntryClasses;
+
+ private File userDirHasEntryClass;
+
+ private File userDirHasNotEntryClass;
+
+ // this was added to support the legacy tests of ClassPathPackagedProgramRetrieverTest in the
+ // refactored PackagedProgramRetrieverImplTest
+ private File userDirWithTestJob;
+
+ private File userDirWithNoJarButTextFile;
+
+ private String originalSystemClasspath;
+
+ @Override
+ public void before() throws IOException {
+ temporaryFolder.create();
+
+ userDirHasTwoEntryClasses =
+ temporaryFolder.newFolder("_test_user_dir_has_two_entry_classes");
+ copyJar(JOB_JAR_PATH, userDirHasTwoEntryClasses);
Review comment:
`PackagedProgramRetrieverImpl.createEntryClassInformationProvider` requires a File instance referring to the folder containing the files. That's why I kept the logic for creating the folders as it already was in the original code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 2a812aa5c826e278b0679572a5b37025aecbb24b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895)
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 878a107e0364b415c288f58eea4a4a716797e77d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* e3b0831a1476a562452576d854dc2d43a3ab3e21 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661979428
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.executors.PipelineExecutorUtils;
+import org.apache.flink.client.testjar.ClasspathProvider;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.ConfigUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.PipelineOptions;
+import org.apache.flink.configuration.PipelineOptionsInternal;
+import org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.util.ChildFirstClassLoader;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.TestLogger;
+
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.hamcrest.core.IsInstanceOf;
+import org.junit.Assume;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsNot.not;
+import static org.junit.Assert.fail;
+
+/** {@code PackagedProgramRetrieverImplTest} tests {@link PackagedProgramRetrieverImpl}. */
+public class PackagedProgramRetrieverImplTest extends TestLogger {
+
+ @Rule public ClasspathProvider classpathProvider = new ClasspathProvider();
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnJobName() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null,
+ Collections.emptyList(),
+ null,
+ PackagedProgramUtils.getPythonDriverClassName(),
+ new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForPythonBasedOnParameter() throws FlinkException {
+ // TODO: see FLINK-23154
+ Assume.assumeTrue(
+ "This tests only succeeds if " + ConfigConstants.ENV_FLINK_OPT_DIR + " is set.",
+ System.getProperty(ConfigConstants.ENV_FLINK_OPT_DIR) != null);
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, Collections.emptyList(), null, null, new String[] {"--python"});
+
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(PackagedProgramUtils.getPythonDriverClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJarFile().get(),
+ is(PackagedProgramUtils.getPythonJar().getFile()));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationForCustomJar()
+ throws FlinkException, MalformedURLException {
+ // make loading from system classpath fail to make sure that it's not triggered
+ classpathProvider.setSystemClasspathWithTwoEntryClasses();
+
+ final String jobClassName = "SomeJobClassName";
+ final File jarFile = new File("some/jar/file.jar");
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, jarFile, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(true));
+ assertThat(informationProvider.getJarFile().get(), is(jarFile));
+ }
+
+ @Test
+ // TODO: we might want to change this behavior triggering a failure here
+ public void testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final String jobClassName = "SomeJobClassNotBeingOnTheSystemClasspath";
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, jobClassName, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(informationProvider.getJobClassName().get(), is(jobClassName));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathWithExistingJobClassName()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, classpathProvider.getJobClassName(), new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromSystemClasspathExtractingTheJobClassFromThere()
+ throws IOException, FlinkException {
+ classpathProvider.setSystemClasspathWithEntryClass();
+
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ null, null, null, null, new String[0]);
+ assertThat(informationProvider.getJobClassName().isPresent(), is(true));
+ assertThat(
+ informationProvider.getJobClassName().get(),
+ is(classpathProvider.getJobClassName()));
+ assertThat(informationProvider.getJarFile().isPresent(), is(false));
+ }
+
+ @Test
+ public void testDeriveEntryClassInformationFromClasspathWithJobClass()
+ throws IOException, FlinkException {
+ final EntryClassInformationProvider informationProvider =
+ PackagedProgramRetrieverImpl.createEntryClassInformationProvider(
+ // the user directory must be specified
+ classpathProvider.getUserDirectoryWithTwoEntryClasses(),
+ // the user classpath is derived from the user directory outside of the
+ // method
+ classpathProvider.getURLUserClasspathWithTwoEntryClasses(),
Review comment:
There's a slight difference between providing a user classpath through the user directory and not providing one. In the former case, the code checks whether the job class appears in the user classpath. For the latter one, this check is not done.
That's the difference between `testDeriveEntryClassInformationFromClasspathWithJobClass` and `testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19895",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
}, {
"hash" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a91715f0a81b12c7cc5de3d91ed13194ee91ab93",
"triggerType" : "PUSH"
}, {
"hash" : "878a107e0364b415c288f58eea4a4a716797e77d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920",
"triggerID" : "878a107e0364b415c288f58eea4a4a716797e77d",
"triggerType" : "PUSH"
}, {
"hash" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "74c80baae62fa14f9f62f8707fd95093a86fd302",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* a91715f0a81b12c7cc5de3d91ed13194ee91ab93 UNKNOWN
* 878a107e0364b415c288f58eea4a4a716797e77d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19920)
* 74c80baae62fa14f9f62f8707fd95093a86fd302 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663778409
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
@Override
public Optional<File> getJarFile() {
return Optional.ofNullable(jarFile);
}
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
Review comment:
emtpy -> empty
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
Review comment:
The condition when it returns an empty optional is unnecessary and just creates confusion. The jar _and_ classname not being specified is a case that a consumer of this APIn ever has to be concerned with because the constructor rejects it.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
Review comment:
The jar doesn't have to be explicitly specified.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
Review comment:
```suggestion
* system classpath. It can return an empty {@code Optional} despite the respective {@code jarFile}
* being set if the job class is the entry class of the jar.
```
The descriptions are bit difficult to understand :/
Note also that this behavior is super weird and we should just return the entry class imo.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.util.Optional;
+
+/**
+ * {@code FromJarEntryClassInformationProvider} is used for cases where the Jar archive is
+ * explicitly specified.
+ */
+public class FromJarEntryClassInformationProvider implements EntryClassInformationProvider {
+
+ private final File jarFile;
+ private final String jobClassName;
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a custom Jar archive. At least the
+ * {@code jarFile} or the {@code jobClassName} has to be set.
+ *
+ * @param jarFile The Jar archive.
+ * @param jobClassName The name of the job class.
+ * @return The {@code FromJarEntryClassInformationProvider} referring to the passed information.
+ */
+ public static FromJarEntryClassInformationProvider createFromCustomJar(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ return new FromJarEntryClassInformationProvider(jarFile, jobClassName);
+ }
+
+ /**
+ * Creates a {@code FromJarEntryClassInformationProvider} for a job implemented in Python.
+ *
+ * @return A {@code FromJarEntryClassInformationProvider} for a job implemented in Python
+ */
+ public static FromJarEntryClassInformationProvider createFromPythonJar() {
+ return new FromJarEntryClassInformationProvider(
+ new File(PackagedProgramUtils.getPythonJar().getPath()),
+ PackagedProgramUtils.getPythonDriverClassName());
+ }
+
+ private FromJarEntryClassInformationProvider(
+ @Nullable File jarFile, @Nullable String jobClassName) {
+ Preconditions.checkArgument(
+ jarFile != null || jobClassName != null,
+ "Either the jar file or the job class name, or both need to be set.");
+ this.jarFile = jarFile;
+ this.jobClassName = jobClassName;
+ }
+
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
+ @Override
+ public Optional<File> getJarFile() {
+ return Optional.ofNullable(jarFile);
+ }
+
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
+ * is set. In that case one should assume the job class being the entry class of the respective
+ * {@code jarFile}.
+ *
+ * @return Returns the job class that can either be found in the respective {@code jarFile} or
+ * on the system classpath. It can also return an empty {@code Optional} if the {@code
+ * jarFile} is specified. In that case the caller should assume the job class being provided
+ * by the {@code jarFile} as the entry class.
Review comment:
similar suggestion to above
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
Review comment:
```suggestion
* @param jobClassName The job class to use; if null the system classpath will be scanned for possible main classes.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
Review comment:
Well this is quite a non-comment isn't it. It could say something like "default X implementation that can retrieve a packaged program from a specific jar or the system classpath."
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
Review comment:
Recently we used `Default*` for the default interface implementations.
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
Review comment:
```suggestion
* @param jarFile The jar archive expected to contain the job class; null if the job class is on the system classpath.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-874463733
I reorganized the commits squashing most of them together and rebased the branch to the most recent `master`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-872037829
Thanks @SteNicholas , @wangyang0918 and @zentol for your detailed feedback. I addressed all the comments or commented on specific items. Please have another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 349d6855c21fed4d4368c56646551c3524437384 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531)
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* d5aa380c6fbe941651d2b095a6d80e92fee04ab3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610)
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 7fe6639f640124b804662177ea510334834f761f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
* 5e03a511663c7cadda1e3f037c30ba3dfda1b05e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842)
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662085256
##########
File path: flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramRetrieverImplTest.java
##########
@@ -328,18 +329,15 @@ public void testFailIfJobDirDoesNotHaveEntryClass() {
}
}
- @Test
- public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException, IOException {
- classpathProvider.setSystemClasspathWithoutEntryClass();
- final PackagedProgramRetrieverImpl retriever =
- PackagedProgramRetrieverImpl.create(
- null,
- classpathProvider.getTestJobClassName(),
- new String[0],
- new Configuration());
- assertThat(
- retriever.getPackagedProgram().getMainClassName(),
- is(classpathProvider.getTestJobClassName()));
+ // TODO: this test checks the same code path as
+ // testDeriveEntryClassInformationFromSystemClasspathWithNonExistingJobClassName
+ // We should make it fail early if the class is not present on the system classpath
+ // Right now, the test is failing because no error is thrown
+ @Ignore
+ @Test(expected = FlinkException.class)
+ public void testEntryClassNotFoundOnSystemClasspath() throws FlinkException {
Review comment:
Correct, but I would keep it in as an additional test in case we decide to make the code more strict (as mentioned in [this comment](https://github.com/apache/flink/pull/16286#discussion_r658542629) and [here](https://github.com/apache/flink/pull/16286/files#r661394188)).
Right now, I lean towards keeping it like it was implemented in the old implementation to not open any hidden problems which might make the PR become even bigger. Instead, I'd create a Flink issue to resolve that. But I wanted to get your opinions on that one. I wasn't 100% sure whether it's actually a desired behavior not checking the class existence on the system classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663011315
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/EntryClassInformationProvider.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.deployment.application;
+
+import java.io.File;
+import java.util.Optional;
+
+/** {@code EntryClassInformationProvider} provides information about the entry class. */
+public interface EntryClassInformationProvider {
+
+ /**
+ * Returns the {@link File} referring to the Jar file that contains the job class or no {@code
+ * File} if the job class is located on the classpath.
+ *
+ * @return The {@code File} referring to the job's Jar archive or an empty {@code Optional} in
+ * case the job shall be extracted from the classpath.
+ */
+ Optional<File> getJarFile();
+
+ /**
+ * Returns the name of the job class.
Review comment:
I updated the JavaDoc in the implementing classes. That way the documentation is closer to the actual implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662992497
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir == null ? null : userClasspath,
+ jarFile,
+ jobClassName,
+ programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userClasspath != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
Review comment:
Good catch. That one I must have missed. But the `@VisibleForTesting` is actually obsolete and the visibility of `fromSystemClasspath` and `fromUserClasspath` wrong. The testing is entirely done on the calling method `createEntryClassInformationProvider`. I made `fromSystemClasspath` and `fromUserClasspath` private and removed the `@VisibleForTesting` annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5e03a511663c7cadda1e3f037c30ba3dfda1b05e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842)
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r663778409
##########
File path: flink-clients/src/main/java/org/apache/flink/client/deployment/application/FromJarEntryClassInformationProvider.java
##########
@@ -68,11 +68,33 @@ private FromJarEntryClassInformationProvider(
this.jobClassName = jobClassName;
}
+ /**
+ * Returns the specified {@code jarFile}. If no {@code jarFile} was specified it can only return
+ * an empty {@code Optional} if the {@code jobClassName} is specified instead. In that case, the
+ * caller should assume that the passed job class can be found on the system classpath.
+ *
+ * @return The specified {@code jarFile}. If returning an empty {@code Optional} the caller
+ * should assume to find the job class on the system classpath.
+ * @see #getJobClassName()
+ */
@Override
public Optional<File> getJarFile() {
return Optional.ofNullable(jarFile);
}
+ /**
+ * Returns the specified job class name that is either available in the corresponding {@code
+ * jarFile} or, if the {@code jarFile} is not set, should be assumed to be contained in the
+ * system classpath. It can return an emtpy {@code Optional} if the respective {@code jarFile}
Review comment:
emtpy -> empty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 1ef57a0f7a4530accbe4817a9f931e1c6467bd92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550)
* 49197ce4e1108e4e09b9328ee956b196fe98fe4d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610)
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-869949832
> @XComp , thanks for your efforts to update the pull request again me. I have reviewed the changes of the refactors. I have confusion about why not validate whether to retrever job graph in tests, which originally validate through retrieveJobGraph.
Thanks for your review. I addressed two of the changes and rebased the branch. About your questions with not using the retriever job graph. Building the `PackagedProgram` will trigger the class loading internally. This is problematic for testing the system classpath functionality.
I tried to use legacy test code as much as possible. New tests, in contrast, avoid loading the class from classpath but rather check whether the correct class is selected. That's why the test implementation focuses on the `EntryClassInformationProvider`. It's more of a unit test rather than a test testing multiple classes interacting with each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662007483
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir, userClasspath, jarFile, jobClassName, programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable File userLibDir,
+ Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userLibDir != null) {
+ return fromUserClasspath(jobClassName, userClasspath);
+ }
+
+ return fromSystemClasspath(jobClassName);
+ }
+
+ static EntryClassInformationProvider fromSystemClasspath(@Nullable String jobClassName)
+ throws FlinkException {
+ if (jobClassName != null) {
+ return FromClasspathEntryClassInformationProvider
+ .createWithJobClassAssumingOnSystemClasspath(jobClassName);
+ }
+
+ try {
+ return FromClasspathEntryClassInformationProvider.createFromSystemClasspath();
+ } catch (Throwable t) {
Review comment:
You're right. Initialliy, we were only expecting `IOException`, `NoSuchElementException`, and `IllegalArgumentException`. I, initially, thought that it would be good to have all errors covered. But I make it more explicit now. Additionally, I transform the `NoSuchElementException` and `IllegalArgumentException` into a checked `FlinkException` within the `FromClasspathEntryClassInformationProvider` to make these two error cases more prominent in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 7fe6639f640124b804662177ea510334834f761f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785)
* 5e03a511663c7cadda1e3f037c30ba3dfda1b05e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r662230770
##########
File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramRetrieverImpl.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.client.deployment.application.EntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromClasspathEntryClassInformationProvider;
+import org.apache.flink.client.deployment.application.FromJarEntryClassInformationProvider;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.FunctionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
+
+/** {@code PackageProgramRetrieverImpl} implements {@link PackagedProgramRetriever}. */
+public class PackagedProgramRetrieverImpl implements PackagedProgramRetriever {
+
+ private final EntryClassInformationProvider entryClassInformationProvider;
+ private final String[] programArguments;
+ private final List<URL> userClasspath;
+ private final Configuration configuration;
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ return create(userLibDir, null, jobClassName, programArgs, configuration);
+ }
+
+ /**
+ * Creates a {@code PackageProgramRetrieverImpl} with the given parameters.
+ *
+ * @param userLibDir The user library directory that is used for generating the user classpath
+ * if specified. The system classpath is used if not specified.
+ * @param jarFile The jar archive used expecting to have the job class included.
+ * @param jobClassName The job class that will be used if specified. The classpath is used to
+ * detect any main class if not specified.
+ * @param programArgs The program arguments.
+ * @param configuration The Flink configuration for the given job.
+ * @return The {@code PackageProgramRetrieverImpl} that can be used to create a {@link
+ * PackagedProgram} instance.
+ * @throws FlinkException If something goes wrong during instantiation.
+ */
+ public static PackagedProgramRetrieverImpl create(
+ @Nullable File userLibDir,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs,
+ Configuration configuration)
+ throws FlinkException {
+ List<URL> userClasspath;
+ try {
+ userClasspath = extractUserClasspath(userLibDir);
+ } catch (IOException e) {
+ throw new FlinkException("An error occurred while extracting the user classpath.", e);
+ }
+
+ final EntryClassInformationProvider entryClassInformationProvider =
+ createEntryClassInformationProvider(
+ userLibDir, userClasspath, jarFile, jobClassName, programArgs);
+ return new PackagedProgramRetrieverImpl(
+ entryClassInformationProvider, programArgs, userClasspath, configuration);
+ }
+
+ @VisibleForTesting
+ static EntryClassInformationProvider createEntryClassInformationProvider(
+ @Nullable File userLibDir,
+ Iterable<URL> userClasspath,
+ @Nullable File jarFile,
+ @Nullable String jobClassName,
+ String[] programArgs)
+ throws FlinkException {
+ if (PackagedProgramUtils.isPython(jobClassName)
+ || PackagedProgramUtils.isPython(programArgs)) {
+ return FromJarEntryClassInformationProvider.createFromPythonJar();
+ }
+
+ if (jarFile != null) {
+ return FromJarEntryClassInformationProvider.createFromCustomJar(jarFile, jobClassName);
+ }
+
+ if (userLibDir != null) {
Review comment:
this is a bit weird isn't it. Couldn't we make `userClasspath` nullable instead? And move the null check for the userlib to the `create` variant that accepts it? That would also address https://github.com/apache/flink/pull/16286/files/a7a79b1944b91a03c7567ce29c561186b1f13d74..c071a23a319c89310e19a40f6d85217936ab1067#r662129429.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
}, {
"hash" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19714",
"triggerID" : "58ae82a70a8915e4cae5b661ce4def47151d92af",
"triggerType" : "PUSH"
}, {
"hash" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19766",
"triggerID" : "6b5d153308edc8cfd3cd3d7c15acb57d7fcb68cc",
"triggerType" : "PUSH"
}, {
"hash" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19771",
"triggerID" : "9e3dc61e6efa4d1c3ac0109ca9fb695bcb246228",
"triggerType" : "PUSH"
}, {
"hash" : "44c00395058d15d3fdee19f54c7c5261db213234",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19777",
"triggerID" : "44c00395058d15d3fdee19f54c7c5261db213234",
"triggerType" : "PUSH"
}, {
"hash" : "7fe6639f640124b804662177ea510334834f761f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19785",
"triggerID" : "7fe6639f640124b804662177ea510334834f761f",
"triggerType" : "PUSH"
}, {
"hash" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19842",
"triggerID" : "5e03a511663c7cadda1e3f037c30ba3dfda1b05e",
"triggerType" : "PUSH"
}, {
"hash" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848",
"triggerID" : "99a22eedf2d53a35559d9fe31fac76f76766faac",
"triggerType" : "PUSH"
}, {
"hash" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a812aa5c826e278b0679572a5b37025aecbb24b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 99a22eedf2d53a35559d9fe31fac76f76766faac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19848)
* 2a812aa5c826e278b0679572a5b37025aecbb24b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16286:
URL: https://github.com/apache/flink/pull/16286#issuecomment-868270639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cbaee738f213985ae8d6850b36005e0ebafd5bf",
"triggerType" : "PUSH"
}, {
"hash" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19523",
"triggerID" : "e3b0831a1476a562452576d854dc2d43a3ab3e21",
"triggerType" : "PUSH"
}, {
"hash" : "349d6855c21fed4d4368c56646551c3524437384",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19531",
"triggerID" : "349d6855c21fed4d4368c56646551c3524437384",
"triggerType" : "PUSH"
}, {
"hash" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19550",
"triggerID" : "1ef57a0f7a4530accbe4817a9f931e1c6467bd92",
"triggerType" : "PUSH"
}, {
"hash" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19610",
"triggerID" : "49197ce4e1108e4e09b9328ee956b196fe98fe4d",
"triggerType" : "PUSH"
}, {
"hash" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0556f30737fd827a1b18f9ec585faa1b26a43793",
"triggerType" : "PUSH"
}, {
"hash" : "5f689da982d423f57ffd8b7205da268d585dc632",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617",
"triggerID" : "5f689da982d423f57ffd8b7205da268d585dc632",
"triggerType" : "PUSH"
}, {
"hash" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642",
"triggerID" : "d5aa380c6fbe941651d2b095a6d80e92fee04ab3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cbaee738f213985ae8d6850b36005e0ebafd5bf UNKNOWN
* 0556f30737fd827a1b18f9ec585faa1b26a43793 UNKNOWN
* 5f689da982d423f57ffd8b7205da268d585dc632 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19617)
* d5aa380c6fbe941651d2b095a6d80e92fee04ab3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19642)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16286: [FLINK-21445] Refactors the PackagedProgramRetriever implementation and adds configuration to PackagedProgram
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16286:
URL: https://github.com/apache/flink/pull/16286#discussion_r661546971
##########
File path: flink-clients/src/test/java/org/apache/flink/client/deployment/application/JarManifestParserTest.java
##########
@@ -157,6 +158,24 @@ public void testFindOnlyEntryClassMultipleJarsWithSingleManifestEntry() throws I
is(equalTo(TestJob.class.getCanonicalName())));
}
+ @Test
+ public void testFindFirstManifestAttributeWithNoAttribute() throws IOException {
+ assertThat(
+ JarManifestParser.findFirstManifestAttribute(TestJob.getTestJobJar()).isPresent(),
+ is(false));
+ }
+
+ @Test
+ public void testFindFirstManifestAttributeWithAttributes() throws IOException {
+ Optional<String> optionalValue =
+ JarManifestParser.findFirstManifestAttribute(
+ TestJob.getTestJobJar(),
+ "some-attr",
Review comment:
but that is this specific attribute for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org