You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Chalitha Perera <cp...@zaizi.com> on 2016/03/08 06:13:03 UTC
Getting class not found exception with DistributedFileSystem (New
Connector for MICO)
Hi All,
We are writing a new transformation connector[1] to utilize MICO
platform[2]. Platform back-end uses hadoop for storage.
When trying to get the outputstream within connector which is a
HdfsDataOutputStream I am getting class not found exception with
DistributedFileSystem. ManifoldCF already includes hadoop-hdfs jar in
connectors-lib which should contain this class.
This error is generated only when used within manifoldcf. We have tried the
platform utilization code from a separate jar and it worked without any
errors.
Any help for this matter is greatly appreciated.
Here is the error I am getting,
Error tossed: java.lang.ClassNotFoundException: Class
org.apache.hadoop.hdfs.DistributedFileSystem not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.hadoop.hdfs.DistributedFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
eu.mico.platform.storage.impl.StorageServiceHDFS.getOutputStream(StorageServiceHDFS.java:62)
at
eu.mico.platform.persistence.impl.MarmottaContent.getOutputStream(MarmottaContent.java:266)
at
org.apache.manifoldcf.agents.transformation.mico.MicoExtractor.addOrReplaceDocumentWithException(MicoExtractor.java:182)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3221)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:404)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.hdfs.DistributedFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2072)
Thanks,
Chalitha
[1] https://github.com/zaizi/sensefy-mico-connector
[2] https://bitbucket.org/mico-project/platform
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Getting class not found exception with DistributedFileSystem (New
Connector for MICO)
Posted by Karl Wright <da...@gmail.com>.
Hi Chalitha,
This seems to be going through a Hadoop class loader of some kind, which
could well not be written in the standard way and would thus be unable to
locate the class in question.
Best practice for class loaders is to use the current class's classloader
which loading the desired class, but I suspect that Hadoop has its own way
of doing this, because it's Hadoop, and because Hadoop is pretty much its
own world.
If you look at the HDFS connector, it uses another path to get into the
file system and doesn't go in through the main door.
Thanks,
Karl
On Tue, Mar 8, 2016 at 12:13 AM, Chalitha Perera <cp...@zaizi.com> wrote:
> Hi All,
>
> We are writing a new transformation connector[1] to utilize MICO
> platform[2]. Platform back-end uses hadoop for storage.
> When trying to get the outputstream within connector which is a
> HdfsDataOutputStream I am getting class not found exception with
> DistributedFileSystem. ManifoldCF already includes hadoop-hdfs jar in
> connectors-lib which should contain this class.
>
> This error is generated only when used within manifoldcf. We have tried the
> platform utilization code from a separate jar and it worked without any
> errors.
>
> Any help for this matter is greatly appreciated.
>
> Here is the error I am getting,
>
> Error tossed: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
>
> eu.mico.platform.storage.impl.StorageServiceHDFS.getOutputStream(StorageServiceHDFS.java:62)
> at
>
> eu.mico.platform.persistence.impl.MarmottaContent.getOutputStream(MarmottaContent.java:266)
> at
>
> org.apache.manifoldcf.agents.transformation.mico.MicoExtractor.addOrReplaceDocumentWithException(MicoExtractor.java:182)
> at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3221)
> at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
> at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706)
> at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756)
> at
>
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583)
> at
>
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548)
> at
>
> org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:404)
> at
>
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2072)
>
>
> Thanks,
> Chalitha
>
>
> [1] https://github.com/zaizi/sensefy-mico-connector
> [2] https://bitbucket.org/mico-project/platform
>
> --
>
> ------------------------------
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number
> 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> London W6 7AN.
>