You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Joe Wang <me...@joewang.net> on 2017/03/21 00:01:38 UTC

NullPointerException when using IGFS

Hi!

I'm running into this stack when trying to read through IGFS to an
underlying HDFS instance:

java.lang.NullPointerException
        at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsInputStream$FetchBufferPart.flatten(HadoopIgfsInputStream.java:458)
        at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsInputStream$DoubleFetchBuffer.flatten(HadoopIgfsInputStream.java:511)
        at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsInputStream.read(HadoopIgfsInputStream.java:177)
        at java.io.DataInputStream.read(DataInputStream.java:100)

and I'm just wondering what the expected behavior of IGFS was. I'm standing
up an ignite cluster (with empty caches), configuring the secondary
filesystem to be an HDFS endpoint, and then trying to read a file from
IGFS. Was IGFS designed to handle this scenario?

Thanks!

Joe

Re: NullPointerException when using IGFS

Posted by dkarachentsev <dk...@gridgain.com>.
Hi,

Thanks for testing, hadoop specialists will fix it. Please use
sequentialReadsBeforePrefetch = 64 as a workaround for now.

Thanks!

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/NullPointerException-when-using-IGFS-tp11328p11486.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NullPointerException when using IGFS

Posted by joewang <me...@joewang.net>.
Yes--we were experiencing this in 1.8 before upgrading to 1.9.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/NullPointerException-when-using-IGFS-tp11328p11438.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NullPointerException when using IGFS

Posted by dkarachentsev <dk...@gridgain.com>.
Joe,

It seems a bug, I've filed a ticket [1]. Could you please check if this
problem exists in 1.8 version?

[1] https://issues.apache.org/jira/browse/IGNITE-4862

Thanks!

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/NullPointerException-when-using-IGFS-tp11328p11431.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NullPointerException when using IGFS

Posted by joewang <me...@joewang.net>.
Attached.

codexrt-igfs-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n11401/codexrt-igfs-config.xml>  
hadoop-console.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n11401/hadoop-console.txt>  
ignite-3a3aac90.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n11401/ignite-3a3aac90.log>  

Additional things I've noticed:
- Writing through IGFS and then reading the file back immediately succeeds.
- Specifying sequentialReadsBeforePrefetch = 64 makes the problem go away,
but I feel like this is affecting performance. Specifying
sequentialReadsBeforePrefetch = 32 sometimes gives me buffer underflow
issues when trying to parse data out of the file format.

Thanks!

Joe



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/NullPointerException-when-using-IGFS-tp11328p11401.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.