You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/04/02 20:20:44 UTC
[GitHub] [beam] iemejia opened a new pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
iemejia opened a new pull request #14417:
URL: https://github.com/apache/beam/pull/14417
This solves a deployment issue for Spark Structured Streaming runner on YARN and reduces and makes code more consistent among runners.
R: @ibzib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812751458
Thanks @ibzib I will merge this as it is in order to test it with tomorrow's SNAPSHOTs, I will open another PR for your suggestion(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812695862
Run Java Spark PortableValidatesRunner Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812695638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia merged pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia merged pull request #14417:
URL: https://github.com/apache/beam/pull/14417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812696040
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia edited a comment on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia edited a comment on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812751458
Thanks @ibzib I will merge this as it is in order to test it with tomorrow's SNAPSHOTs, We should definitely re evaluate the file staging logic in a subsequent PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia edited a comment on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia edited a comment on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812755129
About the question about the exact cause. The method used on the Structured Streaming runner differed from the Classic/Portable implementation in two aspects:
(1) It did not validate the existence of the files to stage first so it could end up trying to stage non existent files and (2) if the user had not set up a tempLocation it would fail, instead of failing back to a default tmp directory as this implementation does:
```java
if (!isLocalSparkMaster(options)) {
List<String> filesToStage =
options.getFilesToStage().stream()
.map(File::new)
.filter(File::exists)
.map(
file -> {
return file.getAbsolutePath();
})
.collect(Collectors.toList());
options.setFilesToStage(
PipelineResources.prepareFilesForStaging(
filesToStage,
MoreObjects.firstNonNull(
options.getTempLocation(), System.getProperty("java.io.tmpdir"))));
}
```
vs
```java
if (!PipelineTranslator.isLocalSparkMaster(options)) {
options.setFilesToStage(
PipelineResources.prepareFilesForStaging(
options.getFilesToStage(), options.getTempLocation()));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14417:
URL: https://github.com/apache/beam/pull/14417#issuecomment-812755129
About the question about the exact cause. The method used on the Structured Streaming runner differed from the Classic/Portable implementation in two aspects:
(1) It did not validate the existence of the files to stage first so it could end up trying to stage non existent files and (2) if the user had not set up a tempLocation it would fail, instead of failing back to a default tmp directory as this implementation does:
```java
if (!isLocalSparkMaster(options)) {
List<String> filesToStage =
options.getFilesToStage().stream()
.map(File::new)
.filter(File::exists)
.map(
file -> {
return file.getAbsolutePath();
})
.collect(Collectors.toList());
options.setFilesToStage(
PipelineResources.prepareFilesForStaging(
filesToStage,
MoreObjects.firstNonNull(
options.getTempLocation(), System.getProperty("java.io.tmpdir"))));
}
```
vs
```java
if (!PipelineTranslator.isLocalSparkMaster(options)) {
options.setFilesToStage(
PipelineResources.prepareFilesForStaging(
options.getFilesToStage(), options.getTempLocation()));
}
```
I suppose failing back to a local temp directory may be improved but so far this is the only way I found to run this on YARN who seems not to follow exactly Beam's staging pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org