You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ferdy Galema <fe...@kalooga.com> on 2011/09/06 16:55:47 UTC
Permission error trying to read map file.
Please see following exception. It looks like it is caused by the
_SUCCESS file created by Hadoop when trying to open map files in a
permission checked HDFS on CDH3u1. After deleting the _SUCCESS it works.
What is a better solution for this problem?
Exception in thread "main"
org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=ferdy, access=EXECUTE,
inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:699)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:542)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
at
org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:302)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:273)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:260)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:253)
at
org.apache.hadoop.mapred.MapFileOutputFormat.getReaders(MapFileOutputFormat.java:93)
at
org.apache.nutch.crawl.CrawlDbReader.openReaders(CrawlDbReader.java:81)
at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:379)
at org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386)
at org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=ferdy, access=EXECUTE,
inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:203)
Re: Permission error trying to read map file.
Posted by Markus Jelsma <ma...@openindex.io>.
Committed in 1.4 per https://issues.apache.org/jira/browse/NUTCH-1110
> Thanks for pointing me to Nutch-1029. This will fix it. It seems
> completely Hadoop related though, because Nutch tries to open map files
> using merely Hadoop code. CrawlDbReader delegates it to
> MapFileOutputFormat.getReaders(...). I'd except at least Hadoop itself
> to handle floating _SUCCESS files correctly.
>
> On 09/06/2011 05:03 PM, Markus Jelsma wrote:
> > Strange, i've not seen permission issues with the _SUCCESS file! Anyway,
> > deleting or not creating the file fixes it. I've committed some fixes for
> > readdb -stats and readlinkdb but not for updatedb.
> >
> > Its related to:
> > https://issues.apache.org/jira/browse/NUTCH-1029
> > Can you open an issue?
> >
> >> Please see following exception. It looks like it is caused by the
> >> _SUCCESS file created by Hadoop when trying to open map files in a
> >> permission checked HDFS on CDH3u1. After deleting the _SUCCESS it works.
> >> What is a better solution for this problem?
> >>
> >> Exception in thread "main"
> >> org.apache.hadoop.security.AccessControlException:
> >> org.apache.hadoop.security.AccessControlException: Permission denied:
> >> user=ferdy, access=EXECUTE,
> >> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r
> >> --
> >>
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>
> >> Method)
> >>
> >> at
> >>
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA
> >> cce ssorImpl.java:39) at
> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
> >> tru ctorAccessorImpl.java:27) at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
> >> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcepti
> >> on. java:95) at
> >> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExcept
> >> ion .java:57) at
> >> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:699) at
> >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi
> >> leS ystem.java:542) at
> >> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) at
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
> >>
> >> at
> >>
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
> >>
> >> at
> >>
> >> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:30
> >> 2)
> >>
> >> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:273)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:260)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:253)
> >> at
> >>
> >> org.apache.hadoop.mapred.MapFileOutputFormat.getReaders(MapFileOutputFor
> >> mat .java:93) at
> >> org.apache.nutch.crawl.CrawlDbReader.openReaders(CrawlDbReader.java:81)
> >>
> >> at
> >> org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:379)
> >> at
> >>
> >> org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386) at
> >> org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511) at
> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> >> a:3 9) at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> >> Imp l.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> >>
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> >>
> >> Caused by: org.apache.hadoop.ipc.RemoteException:
> >> org.apache.hadoop.security.AccessControlException: Permission denied:
> >> user=ferdy, access=EXECUTE,
> >> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r
> >> --
> >>
> >> at
> >>
> >> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermi
> >> ssi onChecker.java:203)
Re: Permission error trying to read map file.
Posted by Markus Jelsma <ma...@openindex.io>.
Yes, i expected the reader to handle it as well and couldn't find some setting
to force it to handle it. The choice remained just not to write the file in
the first place.
> Thanks for pointing me to Nutch-1029. This will fix it. It seems
> completely Hadoop related though, because Nutch tries to open map files
> using merely Hadoop code. CrawlDbReader delegates it to
> MapFileOutputFormat.getReaders(...). I'd except at least Hadoop itself
> to handle floating _SUCCESS files correctly.
>
> On 09/06/2011 05:03 PM, Markus Jelsma wrote:
> > Strange, i've not seen permission issues with the _SUCCESS file! Anyway,
> > deleting or not creating the file fixes it. I've committed some fixes for
> > readdb -stats and readlinkdb but not for updatedb.
> >
> > Its related to:
> > https://issues.apache.org/jira/browse/NUTCH-1029
> > Can you open an issue?
> >
> >> Please see following exception. It looks like it is caused by the
> >> _SUCCESS file created by Hadoop when trying to open map files in a
> >> permission checked HDFS on CDH3u1. After deleting the _SUCCESS it works.
> >> What is a better solution for this problem?
> >>
> >> Exception in thread "main"
> >> org.apache.hadoop.security.AccessControlException:
> >> org.apache.hadoop.security.AccessControlException: Permission denied:
> >> user=ferdy, access=EXECUTE,
> >> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r
> >> --
> >>
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>
> >> Method)
> >>
> >> at
> >>
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA
> >> cce ssorImpl.java:39) at
> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
> >> tru ctorAccessorImpl.java:27) at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
> >> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcepti
> >> on. java:95) at
> >> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExcept
> >> ion .java:57) at
> >> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:699) at
> >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi
> >> leS ystem.java:542) at
> >> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) at
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
> >>
> >> at
> >>
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
> >>
> >> at
> >>
> >> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:30
> >> 2)
> >>
> >> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:273)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:260)
> >> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:253)
> >> at
> >>
> >> org.apache.hadoop.mapred.MapFileOutputFormat.getReaders(MapFileOutputFor
> >> mat .java:93) at
> >> org.apache.nutch.crawl.CrawlDbReader.openReaders(CrawlDbReader.java:81)
> >>
> >> at
> >> org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:379)
> >> at
> >>
> >> org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386) at
> >> org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511) at
> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> >> a:3 9) at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> >> Imp l.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> >>
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> >>
> >> Caused by: org.apache.hadoop.ipc.RemoteException:
> >> org.apache.hadoop.security.AccessControlException: Permission denied:
> >> user=ferdy, access=EXECUTE,
> >> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r
> >> --
> >>
> >> at
> >>
> >> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermi
> >> ssi onChecker.java:203)
Re: Permission error trying to read map file.
Posted by Ferdy Galema <fe...@kalooga.com>.
Thanks for pointing me to Nutch-1029. This will fix it. It seems
completely Hadoop related though, because Nutch tries to open map files
using merely Hadoop code. CrawlDbReader delegates it to
MapFileOutputFormat.getReaders(...). I'd except at least Hadoop itself
to handle floating _SUCCESS files correctly.
On 09/06/2011 05:03 PM, Markus Jelsma wrote:
> Strange, i've not seen permission issues with the _SUCCESS file! Anyway,
> deleting or not creating the file fixes it. I've committed some fixes for
> readdb -stats and readlinkdb but not for updatedb.
>
> Its related to:
> https://issues.apache.org/jira/browse/NUTCH-1029
> Can you open an issue?
>
>> Please see following exception. It looks like it is caused by the
>> _SUCCESS file created by Hadoop when trying to open map files in a
>> permission checked HDFS on CDH3u1. After deleting the _SUCCESS it works.
>> What is a better solution for this problem?
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.AccessControlException:
>> org.apache.hadoop.security.AccessControlException: Permission denied:
>> user=ferdy, access=EXECUTE,
>> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
>> ssorImpl.java:39) at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
>> ctorAccessorImpl.java:27) at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
>> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.
>> java:95) at
>> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException
>> .java:57) at
>> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:699) at
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileS
>> ystem.java:542) at
>> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
>> at
>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:302)
>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
>> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:273)
>> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:260)
>> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:253)
>> at
>> org.apache.hadoop.mapred.MapFileOutputFormat.getReaders(MapFileOutputFormat
>> .java:93) at
>> org.apache.nutch.crawl.CrawlDbReader.openReaders(CrawlDbReader.java:81)
>> at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:379)
>> at
>> org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386) at
>> org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
>> 9) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>> l.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>> Caused by: org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.security.AccessControlException: Permission denied:
>> user=ferdy, access=EXECUTE,
>> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissi
>> onChecker.java:203)
Re: Permission error trying to read map file.
Posted by Markus Jelsma <ma...@openindex.io>.
Strange, i've not seen permission issues with the _SUCCESS file! Anyway,
deleting or not creating the file fixes it. I've committed some fixes for
readdb -stats and readlinkdb but not for updatedb.
Its related to:
https://issues.apache.org/jira/browse/NUTCH-1029
Can you open an issue?
> Please see following exception. It looks like it is caused by the
> _SUCCESS file created by Hadoop when trying to open map files in a
> permission checked HDFS on CDH3u1. After deleting the _SUCCESS it works.
> What is a better solution for this problem?
>
> Exception in thread "main"
> org.apache.hadoop.security.AccessControlException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=ferdy, access=EXECUTE,
> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
> ssorImpl.java:39) at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
> ctorAccessorImpl.java:27) at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.
> java:95) at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException
> .java:57) at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:699) at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileS
> ystem.java:542) at
> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
> at
> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:302)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:273)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:260)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:253)
> at
> org.apache.hadoop.mapred.MapFileOutputFormat.getReaders(MapFileOutputFormat
> .java:93) at
> org.apache.nutch.crawl.CrawlDbReader.openReaders(CrawlDbReader.java:81)
> at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:379)
> at
> org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386) at
> org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
> 9) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
> l.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=ferdy, access=EXECUTE,
> inode="/user/ferdy/root10/crawldb/current/_SUCCESS":ferdy:ferdy:-rw-r--r--
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissi
> onChecker.java:203)