You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/04/13 14:13:18 UTC
[GitHub] [beam] iemejia opened a new pull request #14520: [BEAM-12091] Make file staging uniform among runners
iemejia opened a new pull request #14520:
URL: https://github.com/apache/beam/pull/14520
Follow up of #14417 to avoid divergence of file staging in different runners
R: @ibzib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612895287
##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##########
@@ -102,8 +100,57 @@ public void testPackagingDirectoryResourceToJarFile() throws IOException {
@Test
public void testIfThrowsWhenThereIsNoTemporaryFolderForJars() throws IOException {
- List<String> filesToStage = new ArrayList<>();
- filesToStage.add(tmpFolder.newFolder().getAbsolutePath());
+ List<String> filesToStage = Arrays.asList(tmpFolder.newFolder().getAbsolutePath());
+
+ IllegalArgumentException exception =
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, null));
Review comment:
I guess we don't have nullness checks set up for this package, otherwise we could statically guarantee this can't be null.
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
Review comment:
I did have a problem the context classloader (which is the first choice for `ReflectHelpers.findClassLoader()`) in the Spark runner before - BEAM-10109. But the class loader should only be mutated like that when pipeline construction and Spark execution are happening in the same JVM, so as long as we don't rely on `prepareFilesForStaging` for tests I don't anticipate it being a problem.
##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##########
@@ -102,8 +100,57 @@ public void testPackagingDirectoryResourceToJarFile() throws IOException {
@Test
public void testIfThrowsWhenThereIsNoTemporaryFolderForJars() throws IOException {
- List<String> filesToStage = new ArrayList<>();
- filesToStage.add(tmpFolder.newFolder().getAbsolutePath());
+ List<String> filesToStage = Arrays.asList(tmpFolder.newFolder().getAbsolutePath());
+
+ IllegalArgumentException exception =
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, null));
+
+ assertEquals(
+ "Please provide temporary location for storing the jar files.", exception.getMessage());
+ }
+
+ @Test
+ public void testPrepareFilesForStagingFromOptions() throws IOException {
+ String nonexistentFilePath = tmpFolder.getRoot().getPath() + "/nonexistent/file";
+ String existingFilePath = tmpFolder.newFile("existingFile").getAbsolutePath();
+ List<String> filesToStage = Arrays.asList(nonexistentFilePath, existingFilePath);
+ String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
+
+ FileStagingOptions options = PipelineOptionsFactory.create().as(FileStagingOptions.class);
+ options.setFilesToStage(filesToStage);
+ options.setTempLocation(temporaryLocation);
+
+ assertThrows(
+ "To-be-staged file does not exist: ",
+ IllegalStateException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, temporaryLocation));
+ }
+
+ @Test
+ public void testPackagingDirectoryResourceFromOptions() throws IOException {
+ String directoryPath = tmpFolder.newFolder().getAbsolutePath();
+ List<String> filesToStage = Arrays.asList(directoryPath);
+ String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
+
+ FileStagingOptions options = PipelineOptionsFactory.create().as(FileStagingOptions.class);
+ options.setFilesToStage(filesToStage);
+ options.setTempLocation(temporaryLocation);
+
+ PipelineResources.prepareFilesForStaging(options);
+ List<String> result = options.getFilesToStage();
+
+ assertTrue(new File(result.get(0)).exists());
Review comment:
Should we also assert there is exactly one item in result?
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
Review comment:
Don't we error on nonexistent directories, rather than removing them?
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
Review comment:
We should mention how this method can mutate `options`. Or we could instead return a list of prepared files as before so it's more explicit, like `options.setFilesToStage(PipelineResources.prepareFilesForStaging(options))`.
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
+ LOG.info(
+ "PipelineOptions.filesToStage was not specified. "
+ + "Defaulting to files from the classpath: will stage {} files. "
+ + "Enable logging at DEBUG level to see which files will be staged.",
+ filesToStage.size());
+ LOG.debug("Classpath elements: {}", filesToStage);
+ }
+ options.setFilesToStage(
+ PipelineResources.prepareFilesForStaging(
Review comment:
Can we make `prepareFilesForStaging(List<String>, String)` private, or maybe just merge the two methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612939531
##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##########
@@ -102,8 +100,57 @@ public void testPackagingDirectoryResourceToJarFile() throws IOException {
@Test
public void testIfThrowsWhenThereIsNoTemporaryFolderForJars() throws IOException {
- List<String> filesToStage = new ArrayList<>();
- filesToStage.add(tmpFolder.newFolder().getAbsolutePath());
+ List<String> filesToStage = Arrays.asList(tmpFolder.newFolder().getAbsolutePath());
+
+ IllegalArgumentException exception =
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, null));
+
+ assertEquals(
+ "Please provide temporary location for storing the jar files.", exception.getMessage());
+ }
+
+ @Test
+ public void testPrepareFilesForStagingFromOptions() throws IOException {
+ String nonexistentFilePath = tmpFolder.getRoot().getPath() + "/nonexistent/file";
+ String existingFilePath = tmpFolder.newFile("existingFile").getAbsolutePath();
+ List<String> filesToStage = Arrays.asList(nonexistentFilePath, existingFilePath);
+ String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
+
+ FileStagingOptions options = PipelineOptionsFactory.create().as(FileStagingOptions.class);
+ options.setFilesToStage(filesToStage);
+ options.setTempLocation(temporaryLocation);
+
+ assertThrows(
+ "To-be-staged file does not exist: ",
+ IllegalStateException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, temporaryLocation));
+ }
+
+ @Test
+ public void testPackagingDirectoryResourceFromOptions() throws IOException {
+ String directoryPath = tmpFolder.newFolder().getAbsolutePath();
+ List<String> filesToStage = Arrays.asList(directoryPath);
+ String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
+
+ FileStagingOptions options = PipelineOptionsFactory.create().as(FileStagingOptions.class);
+ options.setFilesToStage(filesToStage);
+ options.setTempLocation(temporaryLocation);
+
+ PipelineResources.prepareFilesForStaging(options);
+ List<String> result = options.getFilesToStage();
+
+ assertTrue(new File(result.get(0)).exists());
Review comment:
yep, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia removed a comment on pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on pull request #14520:
URL: https://github.com/apache/beam/pull/14520#issuecomment-820522347
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r613738572
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
Review comment:
Thanks for the pointer, I am going to try this at least on YARN to be cautious about any possible deployment issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14520:
URL: https://github.com/apache/beam/pull/14520#issuecomment-820522347
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14520:
URL: https://github.com/apache/beam/pull/14520#issuecomment-820547556
Thanks @ibzib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r617968582
##########
File path: runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
##########
@@ -115,13 +114,6 @@ public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo)
: GreedyPipelineFuser.fuse(trimmedPipeline).toPipeline();
// File staging.
- if (pipelineOptions.getFilesToStage() == null) {
- pipelineOptions.setFilesToStage(
- detectClassPathResourcesToStage(
- SparkPipelineRunner.class.getClassLoader(), pipelineOptions));
- LOG.info(
- "PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath");
- }
Review comment:
I think removing this broke Python PostCommits, see BEAM-12207. We're getting an NPE from the LOG statement below, presumably because `pipelineOptions.getFilesToStage() == null`. I'm not sure what the appropriate fix is, is it expected that now the null checking happens somewhere else, or should we add a null check back here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia merged pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia merged pull request #14520:
URL: https://github.com/apache/beam/pull/14520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612486904
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
Review comment:
I am assuming that this is the 'correct' way to find the correct classloader, it is the only possible breaking change of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r613740376
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
+ LOG.info(
+ "PipelineOptions.filesToStage was not specified. "
+ + "Defaulting to files from the classpath: will stage {} files. "
+ + "Enable logging at DEBUG level to see which files will be staged.",
+ filesToStage.size());
+ LOG.debug("Classpath elements: {}", filesToStage);
+ }
+ options.setFilesToStage(
+ PipelineResources.prepareFilesForStaging(
Review comment:
Oups sorry we can't because it is used for tests not only in `PipelineResourcesTest` but also in `FlinkPipelineExecutionEnvironmentTest#testPreparingResourcesToStage(java.lang.String, boolean, boolean)`. I propose that we let it as it is. This is a runners Internal API and little harm can be done from the second method (used mostly for tests) that is immutable versus the big one who is mutable and is the one we are using all around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612940543
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
Review comment:
good idea, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612935409
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
+ LOG.info(
+ "PipelineOptions.filesToStage was not specified. "
+ + "Defaulting to files from the classpath: will stage {} files. "
+ + "Enable logging at DEBUG level to see which files will be staged.",
+ filesToStage.size());
+ LOG.debug("Classpath elements: {}", filesToStage);
+ }
+ options.setFilesToStage(
+ PipelineResources.prepareFilesForStaging(
Review comment:
Sure, my goal was to offer the equivalent functionality for the ones who wanted to do the mutation explictly but since everybody should rely on the other method I will merge both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612488480
##########
File path: runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
##########
@@ -111,22 +109,8 @@ public static SparkStructuredStreamingRunner create(
* @return A pipeline runner that will execute with specified options.
*/
public static SparkStructuredStreamingRunner fromOptions(PipelineOptions options) {
- SparkStructuredStreamingPipelineOptions sparkOptions =
Review comment:
We were executing this relatively heavy step twice because we were doing it in `fromOptions` and in each Spark runner `.run()` method so I removed it from here and it is now up to the runner to take care of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r613739303
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
Review comment:
You are right, documentation updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r612486904
##########
File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##########
@@ -64,6 +70,31 @@
return resourcePath -> !resourcePath.contains("gradle/wrapper");
}
+ /**
+ * Goes through the list of files that need to be staged on runner. Removes nonexistent
+ * directories and packages existing ones. This is necessary for runners that require filesToStage
+ * to be jars only.
+ *
+ * @param options options object with the files to stage and temp location for staging
+ */
+ public static void prepareFilesForStaging(FileStagingOptions options) {
+ List<String> filesToStage = options.getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
+ filesToStage = detectClassPathResourcesToStage(ReflectHelpers.findClassLoader(), options);
Review comment:
I am assuming that this is the 'correct' way to find the correct classloader, before each implementation was doing it relying on the current class, so there is a risk that this can be a breaking change of this PR (hopefully not)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #14520: [BEAM-12091] Make file staging uniform among runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #14520:
URL: https://github.com/apache/beam/pull/14520#discussion_r618291663
##########
File path: runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
##########
@@ -115,13 +114,6 @@ public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo)
: GreedyPipelineFuser.fuse(trimmedPipeline).toPipeline();
// File staging.
- if (pipelineOptions.getFilesToStage() == null) {
- pipelineOptions.setFilesToStage(
- detectClassPathResourcesToStage(
- SparkPipelineRunner.class.getClassLoader(), pipelineOptions));
- LOG.info(
- "PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath");
- }
Review comment:
Fixed by @ibzib via #14616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org