You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/07 23:55:07 UTC
[GitHub] [beam] rahuliyer95 opened a new issue, #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
rahuliyer95 opened a new issue, #22192:
URL: https://github.com/apache/beam/issues/22192
### What happened?
When trying to copy files using `FileSystems.copy` API as follows,
```
FileSystems.copy(srcs, dest, SKIP_IF_DESTINATION_EXISTS)
```
it fails with a `NullPointerException`
```
java.lang.NullPointerException
at org.apache.beam.sdk.io.FileSystems.filterFiles(FileSystems.java:448)
at org.apache.beam.sdk.io.FileSystems.copy(FileSystems.java:280)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.$anonfun$retryableCopyFile$1(ZShuffleCompressor.scala:852)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.twitter.util.Try$.apply(Try.scala:26)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.retryableCopyFile(ZShuffleCompressor.scala:852)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.$anonfun$uploadAndCopyDict$1(ZShuffleCompressor.scala:795)
at com.twitter.util.Return.map(Try.scala:308)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.uploadAndCopyDict(ZShuffleCompressor.scala:790)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.$anonfun$trainAndSave$3(ZShuffleCompressor.scala:730)
at com.twitter.util.Try.transform(Try.scala:230)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.$anonfun$trainAndSave$1(ZShuffleCompressor.scala:727)
at com.twitter.finagle.stats.Stat$.time(Stat.scala:38)
at com.twitter.finagle.stats.Stat$.time(Stat.scala:47)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.trainAndSave(ZShuffleCompressor.scala:706)
at com.twitter.beam.runner.dataflow.worker.compression.ZShuffleCompressor.close(ZShuffleCompressor.scala:472)
at org.apache.beam.runners.dataflow.worker.ShuffleSink$ShuffleSinkWriter.close(ShuffleSink.java:326)
at org.apache.beam.runners.dataflow.worker.SizeReportingSinkWrapper$SizeLimitingWriterWrapper.close(SizeReportingSinkWrapper.java:70)
at org.apache.beam.runners.dataflow.worker.util.common.worker.WriteOperation.finish(WriteOperation.java:99)
at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:94)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:446)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:413)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:331)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
This is because I believe `SKIP_IF_DESTINATION_EXISTS` implicitly depends upon `IGNORE_MISSING_FILES`.
In [FileSystems.java#L448](https://github.com/apache/beam/blob/v2.40.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L448) `matchSrcResults` is `null` because it gets set to `null` in [FileSystems.java#L433](https://github.com/apache/beam/blob/v2.40.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L433) as `IGNORE_MISSING_FILES` is not set. I don't know if this is an explicit requirement for `copy` as I couldn't find it in the documentation. If it is, please have it documented or fix the logic to avoid NPEs when `IGNORE_MISSING_FILES` is not specified.
### Issue Priority
Priority: 1
### Issue Component
Component: build-system
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179178696
Label build
.add-labels cannot be managed because it does not exist in the repo. Please check your spelling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rahuliyer95 commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
rahuliyer95 commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179187342
.add-labels io-java-files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179187548
Label io-java-files cannot be managed because it does not exist in the repo. Please check your spelling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem closed issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
pabloem closed issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
URL: https://github.com/apache/beam/issues/22192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rahuliyer95 commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
rahuliyer95 commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179178520
.remove-labels build
.add-labels io,java,files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rahuliyer95 commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
rahuliyer95 commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179187727
.add-labels io,java,files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rahuliyer95 commented on issue #22192: [Bug]: NullPointerException when copying files using FileSystems.copy and setting StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS
Posted by GitBox <gi...@apache.org>.
rahuliyer95 commented on issue #22192:
URL: https://github.com/apache/beam/issues/22192#issuecomment-1179187234
.remove-labels build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org