You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Eliaaazzz (via GitHub)" <gi...@apache.org> on 2026/07/15 05:51:14 UTC
[I] [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon [beam]
Eliaaazzz opened a new issue, #39336:
URL: https://github.com/apache/beam/issues/39336
### What happened?
On Windows, submitting any portable pipeline to a job server fails during artifact staging, before the pipeline runs. The staged filename contains a colon, which is not a legal path character on Windows.
Job server log:
```
java.util.concurrent.ExecutionException: java.nio.file.InvalidPathException:
Illegal char <:> at index 3: 1-0:ref_Environment_default-submission_environment_dependencies.txt
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 3: 1-0:ref_Environment_default-submission_environment_dependencies.txt
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:170)
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
```
Reproduced on Windows 11, JDK 11, with the Spark job server (`--spark-master-url local[4]`) and `--environment_type=LOOPBACK`, SDK and job server both built from the same commit. The staging code is in `runners/java-fn-execution` and is shared by the portable runners, so I would expect this to affect the other portable runners too, though I only tried Spark.
### It is not specific to any pipeline shape
Both of these fail identically and produce no output:
- an unbounded `Read` (splittable DoFn)
- `Create([1, 2, 3]) | Map` , a bounded pipeline with no source and no SDF
Across five submissions the job server logged the same `Illegal char <:>` ten times, so it is deterministic rather than intermittent.
### Where the colon comes from
`ArtifactStagingService.createFilename` builds the name out of the environment id:
```java
return clip(
String.format("%s-%s-%s", idGenerator.getId(), clip(environment, 25), base), 100);
```
For the observed name `1-0:ref_Environment_default-submission_environment_dependencies.txt` the parts are id `1`, `clip(environment, 25)` = `0:ref_Environment_default`, which is exactly 25 characters, and base `submission_environment_dependencies.txt`. So the colon comes from the environment id being embedded verbatim.
`getDestination` then resolves that name against the staging directory, and `LocalResourceId` rejects it on Windows.
Sanitizing characters that are not legal in a filename before building the name would fix it. Colon is the one that bites here, but `*`, `?`, `"`, `<`, `>` and `|` are also illegal on Windows.
While reading that method, the splitter just above it also looks off:
```java
// Limit to the last contiguous alpha-numeric sequence. In particular, this will exclude
// all path separators.
List<String> components = Splitter.onPattern("[^A-Za-z-_.]]").splitToList(path);
```
The pattern has a trailing `]` outside the character class, so it matches a disallowed character followed by a literal `]` rather than any disallowed character. That does not cause this bug, since the staged name is already a bare filename, but it means the comment's intent is not met for the other artifact roles.
### Related
#21025 is the same area on Windows but a different failure: an illegal `*` in `removeStagedArtifacts`, which happens during cleanup after the job reaches DONE. In that report the pipeline still ran. This one fails staging, so the job never starts.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Java SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon [beam]
Posted by "AtharvUrunkar (via GitHub)" <gi...@apache.org>.
AtharvUrunkar commented on issue #39336:
URL: https://github.com/apache/beam/issues/39336#issuecomment-4999946529
Hi, I'd like to work on this issue. I'll investigate the artifact filename sanitization and add appropriate tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon [beam]
Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on issue #39336:
URL: https://github.com/apache/beam/issues/39336#issuecomment-4977330333
.add-labels bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #39336: [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon
URL: https://github.com/apache/beam/issues/39336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon [beam]
Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on issue #39336:
URL: https://github.com/apache/beam/issues/39336#issuecomment-5000303665
Hi, feel free to use .take-issue to take this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org