You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mikhail <mi...@gmail.com> on 2017/08/02 12:17:46 UTC
Re: IGFS Error when writing a file
Hi Pradeep,
Please properly subscribe to the mailing list so that the community can
receive email notifications for your messages. To subscribe, send empty
email to user-subscribe@ignite.apache.org and follow simple instructions in
the reply.
Could you please specify the version of all tools that you use?(spark,
ignite, hadoop)
Also could you please share you config file that you use for hadoop, spark
and ignite?
Thanks,
Mikhail.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Error-when-writing-a-file-tp15868p15898.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: IGFS Error when writing a file
Posted by mcherkasov <mc...@gridgain.com>.
Hi Pradeep,
I didn't reproduce the issue yet, I need more time for this.
But could you please share the full stack-trace of the exception, because
the most interesting part it is cut.
Thanks,
Mikhail.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Error-when-writing-a-file-tp15868p16006.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: IGFS Error when writing a file
Posted by pradeepchanumolu <pr...@gmail.com>.
Looks like the files did not get uploaded correctly. I am pasting the
contents of the files here.
*Ignite default-config.xml *:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<description>
Spring file for Ignite node configuration with IGFS and Apache Hadoop
map-reduce support enabled. Ignite node will start with this configuration
by default.
</description>
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName"
value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
<property name="searchSystemEnvironment" value="true"/>
</bean>
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString"
value="spark-master-host-ignite:2181"/>
</bean>
</property>
</bean>
</property>
<property name="fileSystemConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.FileSystemConfiguration">
<property name="name" value="igfs"/>
<property name="ipcEndpointConfiguration">
<bean
class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
<property name="type" value="TCP"/>
<property name="host" value="0.0.0.0"/>
<property name="port" value="10500"/>
</bean>
</property>
<property name="secondaryFileSystem">
<bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
<property name="fileSystemFactory">
<bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
<property name="uri"
value="hdfs://hdfs-namenode-host-ignite:9000/"/>
<property name="configPaths">
<list>
<value>/opt/hadoop/etc/hadoop/core-site.xml</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="defaultMode" value="DUAL_SYNC"/>
<property name="pathModes">
<map>
<entry key="/tmp/.*" value="PRIMARY"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
</beans>
*HADOOP core-site.xml: *
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/data/apps/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>hadoop.name.dir</name>
<value>/opt/data/apps/dfs/name</value>
</property>
<property>
<name>hadoop.data.dir</name>
<value>/opt/data/apps/dfs/data</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hdfs-namenode-host-ignite:9000/</value>
</property>
<property>
<name>JAVA_HOME</name>
<value>${JAVA_HOME}</value>
</property>
<property>
<name>fs.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
</property>
</configuration>
*
Spark spark-env.sh*
IGNITE_HOME=/opt/ignite
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];
then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
export SPARK_CLASSPATH=$IGNITE_LIBS
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Error-when-writing-a-file-tp15868p15915.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: IGFS Error when writing a file
Posted by pradeepchanumolu <pr...@gmail.com>.
Hi Mikhail,
Thanks for the response. Here are the details you asked.
Spark Version : 2.1.0
Secondary FS : HDFS (Hadoop version : 2.6.0)
Ignite Version : 2.1.0 (built from src binaries using command : mvn clean
package -DskipTests -Dignite.edition=hadoop -Dhadoop.version=2.6.0 -Prelease
-Dspark.version=2.1.0)
Ignite Config file : default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n15914/default-config.xml>
Hadoop core-site.xml : core-site.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n15914/core-site.xml>
Spark Config File : spark-env.sh
<http://apache-ignite-users.70518.x6.nabble.com/file/n15914/spark-env.sh>
Thanks,
Pradeep
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Error-when-writing-a-file-tp15868p15914.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.