You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Divya Darshan DD <di...@gmail.com> on 2018/09/25 11:00:25 UTC
Nullpointer exception in IgniteHadoopIgfsSecondaryFileSystem
Hi Team,
I am exploring Ignite for my use case (explained in the later section of
this email). Currently I am trying to create a directory in HDFS
(stand-alone) with Java code but I am getting Null pointer exception. Could
you please help me out. Necessary information is as under. Please let me
know if anymore information is needed from my end.
*Java code:*
IgniteHadoopIgfsSecondaryFileSystem i = new
IgniteHadoopIgfsSecondaryFileSystem();
IgfsPath workDir = new IgfsPath("/ddd/fs");
i.mkdirs(workDir);
*Error:*
Line IGFSExample.java:169 à i.mkdirs(workDir)
Exception in thread "main" java.lang.NullPointerException
at
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.mkdirs(IgniteHadoopIgfsSecondaryFileSystem.java:198)
at IGFSExample.mkdirs(IGFSExample.java:169)
at IGFSExample.main(IGFSExample.java:82)
[15:44:15] Ignite node stopped OK [uptime=00:00:00.090]
*Default-config.xml (only important bits)*
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="
http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!--
Initialize property configurer so we can reference environment
variables.
-->
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName"
value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
<property name="searchSystemEnvironment" value="true"/>
</bean>
<!--
Configuration of Ignite node.
-->
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<!--
This port will be used by Apache Hadoop client to connect to
Ignite node as if it was a job tracker.
-->
<property name="connectorConfiguration">
<bean
class="org.apache.ignite.configuration.ConnectorConfiguration">
<property name="port" value="11211"/>
</bean>
</property>
<!--
Configure one IGFS file system instance named "igfs" on this
node.
-->
<property name="fileSystemConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.FileSystemConfiguration">
<!-- IGFS name you will use to access IGFS through
Hadoop API. -->
<property name="name" value="igfs"/>
<!-- Configure TCP endpoint for communication with the
file system instance. -->
<property name="ipcEndpointConfiguration">
<bean
class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
<property name="type" value="TCP" />
<property name="host" value="0.0.0.0" />
<property name="port" value="10500" />
</bean>
</property>
<!--
Configure secondary file system if needed.
-->
<property name="secondaryFileSystem">
<bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
<property name="fileSystemFactory">
<bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
<property name="uri"
value="hdfs://localhost:9000/"/>
</bean>
</property>
</bean>
</property>
</bean>
</list>
</property>
<!--
TCP discovery SPI can be configured with list of addresses if
multicast is not available.
-->
<!--
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
-->
</bean>
</beans>
*Use case:*
We have huge volume of data present in S3. We want to cache the data in
Ignite (Caching will be done either on LRU or custom code as per
requirement). We want to keep only that data which is being frequently used
or have a sliding window i.e keep 5 days of data in cache in sliding window
manner. Whatever data is not eligible for cache needs to be evicted to S3.
Is it possible for Ignite to handle this kind of use case?
Thanks and Regards,
Divya Bamotra
Re: Nullpointer exception in IgniteHadoopIgfsSecondaryFileSystem
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You should put IgniteHadoopIgfsSecondaryFileSystem into
IgniteConfiguration, then you should start Ignite instance, then it will be
initialized properly.
See https://apacheignite-fs.readme.io/docs/secondary-file-system
Regards,
--
Ilya Kasnacheev
вт, 25 сент. 2018 г. в 14:00, Divya Darshan DD <di...@gmail.com>:
> Hi Team,
>
>
>
> I am exploring Ignite for my use case (explained in the later section of
> this email). Currently I am trying to create a directory in HDFS
> (stand-alone) with Java code but I am getting Null pointer exception. Could
> you please help me out. Necessary information is as under. Please let me
> know if anymore information is needed from my end.
>
>
>
> *Java code:*
>
>
>
> IgniteHadoopIgfsSecondaryFileSystem i = new
> IgniteHadoopIgfsSecondaryFileSystem();
> IgfsPath workDir = new IgfsPath("/ddd/fs");
> i.mkdirs(workDir);
>
>
>
>
>
> *Error:*
>
> Line IGFSExample.java:169 à i.mkdirs(workDir)
>
>
>
> Exception in thread "main" java.lang.NullPointerException
>
> at
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.mkdirs(IgniteHadoopIgfsSecondaryFileSystem.java:198)
>
> at IGFSExample.mkdirs(IGFSExample.java:169)
>
> at IGFSExample.main(IGFSExample.java:82)
>
> [15:44:15] Ignite node stopped OK [uptime=00:00:00.090]
>
>
>
> *Default-config.xml (only important bits)*
>
>
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <beans xmlns="http://www.springframework.org/schema/beans"
>
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="
> http://www.springframework.org/schema/util"
>
> xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> http://www.springframework.org/schema/beans/spring-beans.xsd
>
> http://www.springframework.org/schema/util
>
> http://www.springframework.org/schema/util/spring-util.xsd">
>
>
>
> <!--
>
> Initialize property configurer so we can reference environment
> variables.
>
> -->
>
> <bean id="propertyConfigurer"
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>
> <property name="systemPropertiesModeName"
> value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
>
> <property name="searchSystemEnvironment" value="true"/>
>
> </bean>
>
>
>
> <!--
>
> Configuration of Ignite node.
>
> -->
>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <!--
>
> This port will be used by Apache Hadoop client to connect to
> Ignite node as if it was a job tracker.
>
> -->
>
> <property name="connectorConfiguration">
>
> <bean
> class="org.apache.ignite.configuration.ConnectorConfiguration">
>
> <property name="port" value="11211"/>
>
> </bean>
>
> </property>
>
>
>
> <!--
>
> Configure one IGFS file system instance named "igfs" on this
> node.
>
> -->
>
> <property name="fileSystemConfiguration">
>
> <list>
>
> <bean
> class="org.apache.ignite.configuration.FileSystemConfiguration">
>
> <!-- IGFS name you will use to access IGFS through
> Hadoop API. -->
>
> <property name="name" value="igfs"/>
>
>
>
> <!-- Configure TCP endpoint for communication with the
> file system instance. -->
>
> <property name="ipcEndpointConfiguration">
>
> <bean
> class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
>
> <property name="type" value="TCP" />
>
> <property name="host" value="0.0.0.0" />
>
> <property name="port" value="10500" />
>
> </bean>
>
> </property>
>
>
>
> <!--
>
> Configure secondary file system if needed.
>
> -->
>
>
>
> <property name="secondaryFileSystem">
>
> <bean
> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>
> <property name="fileSystemFactory">
>
> <bean
> class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
>
> <property name="uri"
> value="hdfs://localhost:9000/"/>
>
> </bean>
>
> </property>
>
> </bean>
>
> </property>
>
>
>
> </bean>
>
> </list>
>
> </property>
>
>
>
> <!--
>
> TCP discovery SPI can be configured with list of addresses if
> multicast is not available.
>
> -->
>
> <!--
>
> <property name="discoverySpi">
>
> <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>
> <property name="ipFinder">
>
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>
> <property name="addresses">
>
> <list>
>
> <value>127.0.0.1:47500..47509</value>
>
> </list>
>
> </property>
>
> </bean>
>
> </property>
>
> </bean>
>
> </property>
>
> -->
>
> </bean>
>
> </beans>
>
>
>
>
>
> *Use case:*
>
>
>
> We have huge volume of data present in S3. We want to cache the data in
> Ignite (Caching will be done either on LRU or custom code as per
> requirement). We want to keep only that data which is being frequently used
> or have a sliding window i.e keep 5 days of data in cache in sliding window
> manner. Whatever data is not eligible for cache needs to be evicted to S3.
> Is it possible for Ignite to handle this kind of use case?
> Thanks and Regards,
> Divya Bamotra
>