You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "MorganRoff-UnlikelyAI (via GitHub)" <gi...@apache.org> on 2023/09/21 22:01:40 UTC
[GitHub] [beam] MorganRoff-UnlikelyAI opened a new issue, #28603: How can I use HadoopFileSystem to read from s3a:// ?
MorganRoff-UnlikelyAI opened a new issue, #28603:
URL: https://github.com/apache/beam/issues/28603
Hello there,
I have a Beam job running on Spark that reads data from S3. I have tried using the [S3FileSystem](https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java) to read an s3:// path directly, but found this to be much slower than when I use s3a:// via HDFS in a non-Beam job. I believe it should be possible to read s3a:// paths using the [HadoopFileSystem](https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java), but I can't seem to get this working.
If I include the org.apache.beam:beam-sdks-java-io-hadoop-file-system dependency, I still see this error:
```
java.lang.IllegalArgumentException: No filesystem found for scheme s3a
at org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:515)
```
This is in an environment where I know the org.apache.hadoop:hadoop-aws dependency is already included (AWS EMR), and s3a:// paths work out-of-the-box with a plain Spark job. To be safe, I tried also including org.apache.hadoop:hadoop-aws and org.apache.hadoop:hadoop-client directly, as recommended in the [Hadoop docs](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Getting_Started), but that still gave the same errors as above and below.
From looking at the [HadoopFileSystemRegistrar](https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java), it looks like the only way to register a custom scheme is to use an option like `--hdfsConfiguration=[{\"fs.default.name\":\"s3a://{bucket_path}\"}]` to get the scheme registered, but this still results in an error:
```
org.apache.beam.sdk.util.UserCodeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3a"
at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
at org.apache.beam.sdk.io.FileIO$MatchAll$MatchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
```
Has anyone managed to make this work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #28603: How can I use HadoopFileSystem to read from s3a:// ?
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #28603:
URL: https://github.com/apache/beam/issues/28603#issuecomment-1738791125
@MorganRoff-UnlikelyAI
Please, ask user-related questions either on [user@beam.apache.org](https://lists.apache.org/list.html?user@beam.apache.org) or [Apache Beam category in Stackoverflow](https://stackoverflow.com/questions/tagged/apache-beam) or [The ASF slack workspace](https://s.apache.org/beam-slack-channel)
More info about this here https://beam.apache.org/community/contact-us/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How can I use HadoopFileSystem to read from s3a:// ? [beam]
Posted by "shai-rubin (via GitHub)" <gi...@apache.org>.
shai-rubin commented on issue #28603:
URL: https://github.com/apache/beam/issues/28603#issuecomment-2711075638
@MorganRoff-UnlikelyAI were you able to solve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev closed issue #28603: How can I use HadoopFileSystem to read from s3a:// ?
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev closed issue #28603: How can I use HadoopFileSystem to read from s3a:// ?
URL: https://github.com/apache/beam/issues/28603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How can I use HadoopFileSystem to read from s3a:// ? [beam]
Posted by "MorganRoff-UnlikelyAI (via GitHub)" <gi...@apache.org>.
MorganRoff-UnlikelyAI commented on issue #28603:
URL: https://github.com/apache/beam/issues/28603#issuecomment-2711294844
I don't think I did. We moved to Spark in the end, as it was easier to manage on AWS EMR than Beam. We found Beam's abstractions to be more in the way than they were helpful (I expect this may differ in other environments, just sharing our specific experience).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org