You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/08/15 12:32:44 UTC
[GitHub] [druid] 11noskcire opened a new issue #11598: druid failed to start when including druid-ranger-security with hdfs as deep storage
11noskcire opened a new issue #11598:
URL: https://github.com/apache/druid/issues/11598
### Affected Version
0.20.2
### Description
#### Cluster size
nano-quickstart (i'm using this to see if the extension works or not before doing it on real cluster)
#### Steps to reproduce
- Configure hdfs as deep storage
- Include `druid-ranger-security` in `druid.extensions.loadList`
- Start druid services
As a side note i only included it in `druid.extensions.loadList`, not yet using it in authorization config
#### Error message
the error message is same on all druid services
```
2021-08-15T12:09:09,486 INFO [main] com.google.inject.Guice - An exception was caught and reported. Message: java.io.IOException: No FileSystem for scheme: hdfs
java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:59) ~[?:?]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198) ~[guice-4.1.0.jar:?]
at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) [guice-4.1.0.jar:?]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) [guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:99) [guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:73) [guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:62) [guice-4.1.0.jar:?]
at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:433) [druid-server-0.20.2.jar:0.20.2]
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69) [druid-services-0.20.2.jar:0.20.2]
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62) [druid-services-0.20.2.jar:0.20.2]
at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.20.2.jar:0.20.2]
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799) ~[?:?]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810) ~[?:?]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[?:?]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) ~[?:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) ~[?:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[?:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) ~[?:?]
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:56) ~[?:?]
... 15 more
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) An exception was caught and reported. Message: java.io.IOException: No FileSystem for scheme: hdfs
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
1 error
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:72)
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)
at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) An exception was caught and reported. Message: java.io.IOException: No FileSystem for scheme: hdfs
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:433)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
... 2 more
Caused by: java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:59)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
... 7 more
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:56)
... 15 more
```
#### Any debugging
I don't know if this is correct, but i searched the source code [here](https://github.com/apache/druid/blob/177264c6490871e6a557f087270dfbd29723e60b/extensions-core/druid-ranger-security/src/main/java/org/apache/druid/security/ranger/authorizer/RangerSecurityDruidModule.java#L56) which using module `org.apache.hadoop.fs.FileSystem`, which i found later in the [pom.xml](https://github.com/apache/druid/blob/177264c6490871e6a557f087270dfbd29723e60b/extensions-core/druid-ranger-security/pom.xml#L250) only include `hadoop-common` as dependency (according to [this SO thread](https://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file) you must use hadoop-hdfs instead)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 11noskcire closed issue #11598: druid failed to start when including druid-ranger-security with hdfs as deep storage
Posted by GitBox <gi...@apache.org>.
11noskcire closed issue #11598:
URL: https://github.com/apache/druid/issues/11598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 11noskcire commented on issue #11598: druid failed to start when including druid-ranger-security with hdfs as deep storage
Posted by GitBox <gi...@apache.org>.
11noskcire commented on issue #11598:
URL: https://github.com/apache/druid/issues/11598#issuecomment-900025915
for now i solved it with manually copying the hadoop-hdfs-client.jar to druid-ranger-security extension folder
`cp druid/extensions/druid-hdfs-storage/hadoop-hdfs-client-2.8.5.jar druid/extensions/druid-ranger-security/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org