You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tank Sui (JIRA)" <ji...@apache.org> on 2018/10/12 02:38:00 UTC
[jira] [Commented] (HADOOP-15824) RawLocalFileSystem initialize()
raises Null Pointer Exception
[ https://issues.apache.org/jira/browse/HADOOP-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647320#comment-16647320 ]
Tank Sui commented on HADOOP-15824:
-----------------------------------
For me, the workaround is run the app under target/universal folder after sbt dist. i will trace more about the dependency issue.
> RawLocalFileSystem initialize() raises Null Pointer Exception
> -------------------------------------------------------------
>
> Key: HADOOP-15824
> URL: https://issues.apache.org/jira/browse/HADOOP-15824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.8.3, 2.8.4, 3.1.1
> Environment: Hadoop 2.8.4 + Spark & yarn client launch
> Reporter: Tank Sui
> Priority: Minor
>
> {code:java}
> [ERROR]09:33:13.143 [main] org.apache.spark.SparkContext - Error initializing SparkContext.
> 10/6/2018 5:33:13 PM java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> 10/6/2018 5:33:13 PM at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> 10/6/2018 5:33:13 PM at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:351)
> 10/6/2018 5:33:13 PM at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:649)
> 10/6/2018 5:33:13 PM at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:863)
> 10/6/2018 5:33:13 PM at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
> 10/6/2018 5:33:13 PM at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
> 10/6/2018 5:33:13 PM at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
> 10/6/2018 5:33:13 PM at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
> 10/6/2018 5:33:13 PM at org.apache.spark.SparkContext.<init>(SparkContext.scala:126)
> 10/6/2018 5:33:13 PM at services.SparkService.tryInit(SparkService.scala:49)
> 10/6/2018 5:33:13 PM at controllers.DataController.<init>(DataController.scala:38)
> 10/6/2018 5:33:13 PM at controllers.DataController$$FastClassByGuice$$9ed55d7d.newInstance(<generated>)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
> 10/6/2018 5:33:13 PM at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:409)
> 10/6/2018 5:33:13 PM at play.api.inject.ContextClassLoaderInjector$$anonfun$instanceOf$3.apply(Injector.scala:118)
> 10/6/2018 5:33:13 PM at play.api.inject.ContextClassLoaderInjector.withContext(Injector.scala:126)
> 10/6/2018 5:33:13 PM at play.api.inject.ContextClassLoaderInjector.instanceOf(Injector.scala:118)
> 10/6/2018 5:33:13 PM at play.api.inject.RoutesProvider$$anonfun$2.apply(BuiltinModule.scala:107)
> 10/6/2018 5:33:13 PM at play.api.inject.RoutesProvider$$anonfun$2.apply(BuiltinModule.scala:107)
> 10/6/2018 5:33:13 PM at scala.Option.fold(Option.scala:158)
> 10/6/2018 5:33:13 PM at play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:107)
> 10/6/2018 5:33:13 PM at play.api.inject.RoutesProvider.get(BuiltinModule.scala:103)
> 10/6/2018 5:33:13 PM at play.api.inject.RoutesProvider.get(BuiltinModule.scala:102)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
> 10/6/2018 5:33:13 PM at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
> 10/6/2018 5:33:13 PM at com.google.inject.Guice.createInjector(Guice.java:99)
> 10/6/2018 5:33:13 PM at com.google.inject.Guice.createInjector(Guice.java:84)
> 10/6/2018 5:33:13 PM at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:185)
> 10/6/2018 5:33:13 PM at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:137)
> 10/6/2018 5:33:13 PM at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
> 10/6/2018 5:33:13 PM at play.core.server.ProdServerStart$.start(ProdServerStart.scala:51)
> 10/6/2018 5:33:13 PM at play.core.server.ProdServerStart$.main(ProdServerStart.scala:25)
> 10/6/2018 5:33:13 PM at play.core.server.ProdServerStart.main(ProdServerStart.scala)
> 10/6/2018 5:33:13 PM Caused by: java.lang.reflect.InvocationTargetException: null
> 10/6/2018 5:33:13 PM at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 10/6/2018 5:33:13 PM at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 10/6/2018 5:33:13 PM at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 10/6/2018 5:33:13 PM at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
> 10/6/2018 5:33:13 PM ... 80 common frames omitted
> 10/6/2018 5:33:13 PM Caused by: java.lang.NullPointerException: null
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.Path.<init>(Path.java:141)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.Path.makeQualified(Path.java:535)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:485)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.RawLocalFileSystem.getInitialWorkingDirectory(RawLocalFileSystem.java:581)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.RawLocalFileSystem.<init>(RawLocalFileSystem.java:73)
> 10/6/2018 5:33:13 PM at org.apache.hadoop.fs.LocalFileSystem.<init>(LocalFileSystem.java:41)
> 10/6/2018 5:33:13 PM ... 85 common frames omitted
> {code}
> This error happened in hadoop 2.8.4
> after investigation, i think below code in Path.java of apache common project has some issue.
> {code:java}
> /**
> * Create a new Path based on the child path resolved against the parent path.
> *
> * @param parent the parent path
> * @param child the child path
> */
> public Path(Path parent, Path child) {
> // Add a slash to parent's path so resolution is compatible with URI's
> URI parentUri = parent.uri;
> String parentPath = parentUri.getPath();
> if (!(parentPath.equals("/") || parentPath.isEmpty())) {
> try {
> {code}
> *parent* should be null while *parent.uri* called.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org