You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mikhail <mi...@gmail.com> on 2017/06/01 15:57:33 UTC
Re: Ignite Cache returning Null value from Mapper
Hi Prashant,
Could you please provide more details about your configuration?
Maybe you can send a test case that will show the issue?
PS you email didn't get to mail list, because looks like you haven't
subscribed to the list.
Please subscribe to the list before sending further emails.
Thanks,
Mikhail.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13321.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hi,
Please find below logs from Createcache and testmapper class.
Createcache log
---------------------------
INFO: OS: Linux 2.6.32-431.el6.x86_64 amd64
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: OS user: p_hadoop
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: PID: 7251
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Language runtime: Java Platform API Specification ver. 1.8
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM total memory: 22.0GB
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Remote Management [restart: off, REST: on, JMX (remote: off)]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: IGNITE_HOME=null
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM arguments: [-Xmx25g]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: System cache's MemoryPolicy size is configured to 40 MB. Use
MemoryConfiguration.systemCacheMemorySize property to change the setting.
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Configured caches [in 'default' memoryPolicy: ['ignite-sys-cache',
'ignite-atomics-sys-cache']]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Configured plugins:
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- None
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Successfully bound communication NIO server to TCP port [port=47100,
locHost=0.0.0.0/0.0.0.0, selectorsCnt=12, selectorSpins=0, pairedConn=false]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Checkpoints are disabled (to enable configure any GridCheckpointSpi
implementation)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Collision resolution is disabled (all jobs will be activated upon
arrival).
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Security status [authentication=off, tls/ssl=off]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: No user-defined default MemoryPolicy found; system default of 1GB
size will be used.
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11211]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Non-loopback local IPs: 10.10.10.138, fe80:0:0:0:225:b5ff:fe00:30%eth0
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Enabled local MACs: 0025B5000030
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=0802cff4-0a9e-43a1-bcf3-5f0c95e75007]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=ignite-sys-cache, memoryPolicyName=sysMemPlc,
mode=REPLICATED]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=ignite-atomics-sys-cache,
memoryPolicyName=sysMemPlc, mode=PARTITIONED]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=0], evt=NODE_JOINED,
node=0802cff4-0a9e-43a1-bcf3-5f0c95e75007]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Performance suggestions for grid (fix if possible)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Reduce pages swapping ratio (set vm.swappiness=10)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Disable NUMA memory reclaim (set vm.zone_reclaim_mode=0)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Avoid direct reclaim and page allocation failures (set
vm.extra_free_kbytes=1240000)
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.0.0#20170430-sha1:d4eef3c68ff116ee34bc13648cd82c640b3ea072
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 2.6.32-431.el6.x86_64 amd64
>>> CPU(s): 24
>>> Heap: 22.0GB
>>> VM name: 7251@HMCISPRDATAWARE
>>> Local node [ID=0802CFF4-0A9E-43A1-BCF3-5F0C95E75007, order=1,
>>> clientMode=false]
>>> Local node addresses: [HMCISPRDATAWARE.CHM.INTRA/0:0:0:0:0:0:0:1%lo,
>>> /10.10.10.138, /127.0.0.1]
>>> Local ports: TCP:11211 TCP:47100 TCP:47500
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=22.0GB]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=matchCache, memoryPolicyName=null,
mode=REPLICATED]
Jun 09, 2017 12:22:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=0802cff4-0a9e-43a1-bcf3-5f0c95e75007]
Done....cache
Jun 09, 2017 12:23:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0802cff4, name=null, uptime=00:01:00:008]
^-- H/N/C [hosts=1, nodes=1, CPUs=24]
^-- CPU [cur=0%, avg=0.1%, GC=0%]
^-- PageMemory [pages=3276]
^-- Heap [used=262MB, free=98.85%, comm=1454MB]
^-- Non heap [used=34MB, free=-1%, comm=35MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
Jun 09, 2017 12:23:28 AM org.apache.ignite.logger.java.JavaLogger info
INFO: FreeList [name=null, buckets=256, dataPages=1, reusePages=0]
Jun 09, 2017 12:23:38 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Failed to read message due to ClassNotFoundException (make sure
same versions of all classes are available on all nodes)
[rmtNodeId=0a0516db-8178-41b4-87a8-aaea87dbbd2d,
err=o.a.i.i.processors.hadoop.HadoopAttributes]
Jun 09, 2017 12:23:38 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Update status is not available.
-----------------------------------------------------------------------------------------------
Testmapper Log
----------------------------------------------------------------------------------------------
17/06/09 00:26:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
17/06/09 00:26:08 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
17/06/09 00:26:08 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
17/06/09 00:26:08 WARN mapreduce.JobResourceUploader: Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
17/06/09 00:26:08 INFO input.FileInputFormat: Total input paths to process :
1
17/06/09 00:26:08 INFO mapreduce.JobSubmitter: number of splits:1
17/06/09 00:26:08 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local2117899413_0001
17/06/09 00:26:08 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
17/06/09 00:26:08 INFO mapreduce.Job: Running job: job_local2117899413_0001
17/06/09 00:26:08 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
17/06/09 00:26:08 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
17/06/09 00:26:08 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/06/09 00:26:08 INFO mapred.LocalJobRunner: Waiting for map tasks
17/06/09 00:26:08 INFO mapred.LocalJobRunner: Starting task:
attempt_local2117899413_0001_m_000000_0
17/06/09 00:26:08 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
17/06/09 00:26:08 INFO mapred.Task: Using ResourceCalculatorProcessTree : [
]
17/06/09 00:26:08 INFO mapred.MapTask: Processing split:
hdfs://localhost:9001/pravin/input/TestFileForPR.txt:0+4921
17/06/09 00:26:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/06/09 00:26:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/06/09 00:26:08 INFO mapred.MapTask: soft limit at 83886080
17/06/09 00:26:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/06/09 00:26:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/06/09 00:26:08 INFO mapred.MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
setup called
TcpDiscoveryVmIpFinder [addrs=[/127.0.0.1:0, /127.0.0.1:47500,
/127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
/127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
/127.0.0.1:47509], super=TcpDiscoveryIpFinderAdapter [shared=false]]
Jun 09, 2017 12:26:08 AM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.0.0#20170430-sha1:d4eef3c6
>>> 2017 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Config URL: n/a
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Daemon mode: off
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: OS: Linux 2.6.32-431.el6.x86_64 amd64
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: OS user: p_hadoop
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: PID: 12913
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Language runtime: Java Platform API Specification ver. 1.8
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM total memory: 18.0GB
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Remote Management [restart: off, REST: on, JMX (remote: off)]
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: IGNITE_HOME=null
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: VM arguments: [-Xmx1000m, -Djava.net.preferIPv4Stack=true,
-Dhadoop.log.dir=/home/p_hadoop/hadoop-2.7.3/logs,
-Dhadoop.log.file=hadoop.log, -Dhadoop.home.dir=/home/p_hadoop/hadoop-2.7.3,
-Dhadoop.id.str=p_hadoop, -Dhadoop.root.logger=INFO,console,
-Dhadoop.policy.file=hadoop-policy.xml, -Djava.net.preferIPv4Stack=true,
-Xmx4096m, -Xmx20g, -Dhadoop.security.logger=INFO,NullAppender]
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: System cache's MemoryPolicy size is configured to 40 MB. Use
MemoryConfiguration.systemCacheMemorySize property to change the setting.
Jun 09, 2017 12:26:08 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Configured caches [in 'default' memoryPolicy: ['ignite-sys-cache',
'ignite-hadoop-mr-sys-cache', 'ignite-atomics-sys-cache']]
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Configured plugins:
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- None
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Successfully bound communication NIO server to TCP port [port=47101,
locHost=0.0.0.0/0.0.0.0, selectorsCnt=12, selectorSpins=0, pairedConn=false]
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Checkpoints are disabled (to enable configure any GridCheckpointSpi
implementation)
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Collision resolution is disabled (all jobs will be activated upon
arrival).
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Security status [authentication=off, tls/ssl=off]
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: No user-defined default MemoryPolicy found; system default of 1GB
size will be used.
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11212]
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: HADOOP_HOME is set to /home/p_hadoop/hadoop-2.7.3
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Resolved Hadoop classpath locations: /home/p_hadoop/hadoop-2.7.3,
/home/p_hadoop/hadoop-2.7.3, /home/p_hadoop/hadoop-2.7.3
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Hadoop libraries are found in Ignite classpath, this could lead to
class loading errors (please remove all Hadoop libraries from Ignite
classpath)
[path=file:/home/p_hadoop/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar]
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Non-loopback local IPs: 10.10.10.138
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Enabled local MACs: 0025B5000030
Jun 09, 2017 12:26:09 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Successfully bound to TCP port [port=47501, localHost=0.0.0.0/0.0.0.0,
locNodeId=19853d56-72ea-4d8c-821c-7b4737cf7221]
17/06/09 00:26:09 INFO mapreduce.Job: Job job_local2117899413_0001 running
in uber mode : false
17/06/09 00:26:09 INFO mapreduce.Job: map 0% reduce 0%
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=ignite-hadoop-mr-sys-cache,
memoryPolicyName=sysMemPlc, mode=REPLICATED]
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=ignite-sys-cache, memoryPolicyName=sysMemPlc,
mode=REPLICATED]
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=ignite-atomics-sys-cache,
memoryPolicyName=sysMemPlc, mode=PARTITIONED]
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=0], evt=NODE_JOINED,
node=19853d56-72ea-4d8c-821c-7b4737cf7221]
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Performance suggestions for grid (fix if possible)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Reduce pages swapping ratio (set vm.swappiness=10)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Disable NUMA memory reclaim (set vm.zone_reclaim_mode=0)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: ^-- Avoid direct reclaim and page allocation failures (set
vm.extra_free_kbytes=1240000)
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.0.0#20170430-sha1:d4eef3c68ff116ee34bc13648cd82c640b3ea072
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 2.6.32-431.el6.x86_64 amd64
>>> CPU(s): 24
>>> Heap: 18.0GB
>>> VM name: 12913@HMCISPRDATAWARE.CHM.INTRA
>>> Local node [ID=19853D56-72EA-4D8C-821C-7B4737CF7221, order=1,
>>> clientMode=false]
>>> Local node addresses: [HMCISPRDATAWARE.CHM.INTRA/10.10.10.138,
>>> /127.0.0.1]
>>> Local ports: TCP:11212 TCP:47101 TCP:47501
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=18.0GB]
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started cache [name=matchCache, memoryPolicyName=null,
mode=PARTITIONED]
cache name is matchCache
Ignite started
Jun 09, 2017 12:26:19 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=19853d56-72ea-4d8c-821c-7b4737cf7221]
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
in map
null
Cleanup called
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Command protocol successfully stopped: TCP binary
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Stopped cache: ignite-sys-cache
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Stopped cache: ignite-hadoop-mr-sys-cache
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Stopped cache: ignite-atomics-sys-cache
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Stopped cache: matchCache
Jun 09, 2017 12:26:20 AM org.apache.ignite.logger.java.JavaLogger info
INFO:
>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 2.0.0#20170430-sha1:d4eef3c68ff116ee34bc13648cd82c640b3ea072
>>> stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 00:00:00:131
17/06/09 00:26:20 INFO mapred.LocalJobRunner:
17/06/09 00:26:20 INFO mapred.MapTask: Starting flush of map output
17/06/09 00:26:20 INFO mapred.MapTask: Spilling map output
17/06/09 00:26:20 INFO mapred.MapTask: bufstart = 0; bufend = 5605; bufvoid
= 104857600
17/06/09 00:26:20 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend
= 26213716(104854864); length = 681/6553600
17/06/09 00:26:20 INFO mapred.MapTask: Finished spill 0
17/06/09 00:26:20 INFO mapred.Task:
Task:attempt_local2117899413_0001_m_000000_0 is done. And is in the process
of committing
17/06/09 00:26:20 INFO mapred.LocalJobRunner: map
17/06/09 00:26:20 INFO mapred.Task: Task
'attempt_local2117899413_0001_m_000000_0' done.
17/06/09 00:26:20 INFO mapred.LocalJobRunner: Finishing task:
attempt_local2117899413_0001_m_000000_0
17/06/09 00:26:20 INFO mapred.LocalJobRunner: map task executor complete.
17/06/09 00:26:20 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/06/09 00:26:20 INFO mapred.LocalJobRunner: Starting task:
attempt_local2117899413_0001_r_000000_0
17/06/09 00:26:20 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
17/06/09 00:26:20 INFO mapred.Task: Using ResourceCalculatorProcessTree : [
]
17/06/09 00:26:20 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin:
org.apache.hadoop.mapreduce.task.reduce.Shuffle@6c48fcaa
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: MergerManager:
memoryLimit=13362160640, maxSingleShuffleLimit=3340540160,
mergeThreshold=8819025920, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/06/09 00:26:20 INFO reduce.EventFetcher:
attempt_local2117899413_0001_r_000000_0 Thread started: EventFetcher for
fetching Map Completion Events
17/06/09 00:26:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
output of map attempt_local2117899413_0001_m_000000_0 decomp: 5949 len: 5953
to MEMORY
17/06/09 00:26:20 INFO reduce.InMemoryMapOutput: Read 5949 bytes from
map-output for attempt_local2117899413_0001_m_000000_0
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
map-output of size: 5949, inMemoryMapOutputs.size() -> 1, commitMemory -> 0,
usedMemory ->5949
17/06/09 00:26:20 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
17/06/09 00:26:20 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: finalMerge called with 1
in-memory map-outputs and 0 on-disk map-outputs
17/06/09 00:26:20 INFO mapred.Merger: Merging 1 sorted segments
17/06/09 00:26:20 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 5942 bytes
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: Merged 1 segments, 5949
bytes to disk to satisfy reduce memory limit
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: Merging 1 files, 5953 bytes
from disk
17/06/09 00:26:20 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes
from memory into reduce
17/06/09 00:26:20 INFO mapred.Merger: Merging 1 sorted segments
17/06/09 00:26:20 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 5942 bytes
17/06/09 00:26:20 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/06/09 00:26:20 INFO Configuration.deprecation: mapred.skip.on is
deprecated. Instead, use mapreduce.job.skiprecords
17/06/09 00:26:20 INFO mapred.Task:
Task:attempt_local2117899413_0001_r_000000_0 is done. And is in the process
of committing
17/06/09 00:26:20 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/06/09 00:26:20 INFO mapred.Task: Task
attempt_local2117899413_0001_r_000000_0 is allowed to commit now
17/06/09 00:26:20 INFO output.FileOutputCommitter: Saved output of task
'attempt_local2117899413_0001_r_000000_0' to
hdfs://localhost:9001/pravin/output/_temporary/0/task_local2117899413_0001_r_000000
17/06/09 00:26:20 INFO mapred.LocalJobRunner: reduce > reduce
17/06/09 00:26:20 INFO mapred.Task: Task
'attempt_local2117899413_0001_r_000000_0' done.
17/06/09 00:26:20 INFO mapred.LocalJobRunner: Finishing task:
attempt_local2117899413_0001_r_000000_0
17/06/09 00:26:20 INFO mapred.LocalJobRunner: reduce task executor complete.
17/06/09 00:26:20 INFO mapreduce.Job: map 100% reduce 100%
17/06/09 00:26:20 INFO mapreduce.Job: Job job_local2117899413_0001 completed
successfully
17/06/09 00:26:20 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=150814666
FILE: Number of bytes written=152574059
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=9842
HDFS: Number of bytes written=286
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Map-Reduce Framework
Map input records=19
Map output records=171
Map output bytes=5605
Map output materialized bytes=5953
Input split bytes=117
Combine input records=0
Combine output records=0
Reduce input groups=9
Reduce shuffle bytes=5953
Reduce input records=171
Reduce output records=9
Spilled Records=342
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=108
Total committed heap usage (bytes)=5053087744
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=4921
File Output Format Counters
Bytes Written=286
------------------------------------------------------------------------------------------
Thanks,
Prashant
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13518.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Michael Cherkasov <mi...@gmail.com>.
Hi Prashant,
Could you please collect more logs, please add "-DIGNITE_QUIET=false" arg
to Createcache and TestMapper runs?
I even tried CentOS with the same version as you have, but it still works
fine for me.
I have a suggestion, usually there's separate account for Hadoop, and when
you try to run
Ignite from hadoop mapper it runs it under hadoop's account, so may this
some doesn't allow
TestMapper to see CreateCache. I didn't check this suggestion, but you may
be try to run hadoop
and CreateCache under the same account.
Thanks,
Mikhail.
2017-06-07 7:37 GMT+03:00 Prashant312 <p....@crif.com>:
> Hi,
>
> Please see the linux version..
>
> [p_hadoop@HMCISPRDATAWARE ~]$ uname -a
> Linux HMCISPRDATAWARE 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC
> 2013 x86_64 x86_64 x86_64 GNU/Linux
> [p_hadoop@HMCISPRDATAWARE ~]$ cat /etc/redhat-release
> CentOS release 6.5 (Final)
> [p_hadoop@HMCISPRDATAWARE ~]$ lsb_release -a
> LSB Version:
> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.
> 0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-
> 4.0-amd64:printing-4.0-noarch
> Distributor ID: CentOS
> Description: CentOS release 6.5 (Final)
> Release: 6.5
> Codename: Final
> [p_hadoop@HMCISPRDATAWARE ~]$ cat /proc/version
> Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org)
> (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22
> 03:15:09 UTC 2013
> [p_hadoop@HMCISPRDATAWARE ~]$ cat /etc/issue
> CentOS release 6.5 (Final)
> Kernel \r on an \m
>
> [p_hadoop@HMCISPRDATAWARE ~]$
>
> ------------------------------------------------------------------------
>
> Below is the log for createcache class...
>
> [p_hadoop@HMCISPRDATAWARE ~]$ tail -100f
> /home/p_hadoop/pravin/igniteCache.log
> Jun 07, 2017 3:35:04 PM java.util.logging.LogManager$RootLogger log
> SEVERE: Failed to resolve default logging config file:
> config/java.util.logging.properties
> [15:35:04] __________ ________________
> [15:35:04] / _/ ___/ |/ / _/_ __/ __/
> [15:35:04] _/ // (7 7 // / / / / _/
> [15:35:04] /___/\___/_/|_/___/ /_/ /___/
> [15:35:04]
> [15:35:04] ver. 2.0.0#20170430-sha1:d4eef3c6
> [15:35:04] 2017 Copyright(C) Apache Software Foundation
> [15:35:04]
> [15:35:04] Ignite documentation: http://ignite.apache.org
> [15:35:04]
> [15:35:04] Quiet mode.
> [15:35:04] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
> or "-v" to ignite.{sh|bat}
> [15:35:04]
> [15:35:04] OS: Linux 2.6.32-431.el6.x86_64 amd64
> [15:35:04] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13
> Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
> [15:35:04] Configured plugins:
> [15:35:04] ^-- None
> [15:35:04]
> [15:35:04] Message queue limit is set to 0 which may lead to potential
> OOMEs
> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
> message queues growth on sender and receiver sides.
> [15:35:04] Security status [authentication=off, tls/ssl=off]
> [15:35:04] Performance suggestions for grid (fix if possible)
> [15:35:04] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> [15:35:04] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
> options)
> [15:35:04] ^-- Set max direct memory size if getting 'OOME: Direct buffer
> memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
> [15:35:04] ^-- Disable processing of calls to System.gc() (add
> '-XX:+DisableExplicitGC' to JVM options)
> [15:35:04] ^-- Speed up flushing of dirty pages by OS (alter
> vm.dirty_expire_centisecs parameter by setting to 500)
> [15:35:04] ^-- Reduce pages swapping ratio (set vm.swappiness=10)
> [15:35:04] ^-- Disable NUMA memory reclaim (set vm.zone_reclaim_mode=0)
> [15:35:04] ^-- Avoid direct reclaim and page allocation failures (set
> vm.extra_free_kbytes=1240000)
> [15:35:04] Refer to this page for more performance suggestions:
> https://apacheignite.readme.io/docs/jvm-and-system-tuning
> [15:35:04]
> [15:35:04] To start Console Management & Monitoring run
> ignitevisorcmd.{sh|bat}
> [15:35:04]
> [15:35:04] Ignite node started OK (id=b9c64c6d)
> [15:35:04] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24,
> heap=22.0GB]
> Done....cache
>
> Thanks,
> Prashant
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cache-returning-Null-value-
> from-Mapper-tp13158p13448.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hi,
Please see the linux version..
[p_hadoop@HMCISPRDATAWARE ~]$ uname -a
Linux HMCISPRDATAWARE 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux
[p_hadoop@HMCISPRDATAWARE ~]$ cat /etc/redhat-release
CentOS release 6.5 (Final)
[p_hadoop@HMCISPRDATAWARE ~]$ lsb_release -a
LSB Version:
:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final
[p_hadoop@HMCISPRDATAWARE ~]$ cat /proc/version
Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org)
(gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22
03:15:09 UTC 2013
[p_hadoop@HMCISPRDATAWARE ~]$ cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m
[p_hadoop@HMCISPRDATAWARE ~]$
------------------------------------------------------------------------
Below is the log for createcache class...
[p_hadoop@HMCISPRDATAWARE ~]$ tail -100f
/home/p_hadoop/pravin/igniteCache.log
Jun 07, 2017 3:35:04 PM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
[15:35:04] __________ ________________
[15:35:04] / _/ ___/ |/ / _/_ __/ __/
[15:35:04] _/ // (7 7 // / / / / _/
[15:35:04] /___/\___/_/|_/___/ /_/ /___/
[15:35:04]
[15:35:04] ver. 2.0.0#20170430-sha1:d4eef3c6
[15:35:04] 2017 Copyright(C) Apache Software Foundation
[15:35:04]
[15:35:04] Ignite documentation: http://ignite.apache.org
[15:35:04]
[15:35:04] Quiet mode.
[15:35:04] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[15:35:04]
[15:35:04] OS: Linux 2.6.32-431.el6.x86_64 amd64
[15:35:04] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
[15:35:04] Configured plugins:
[15:35:04] ^-- None
[15:35:04]
[15:35:04] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[15:35:04] Security status [authentication=off, tls/ssl=off]
[15:35:04] Performance suggestions for grid (fix if possible)
[15:35:04] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[15:35:04] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[15:35:04] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[15:35:04] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[15:35:04] ^-- Speed up flushing of dirty pages by OS (alter
vm.dirty_expire_centisecs parameter by setting to 500)
[15:35:04] ^-- Reduce pages swapping ratio (set vm.swappiness=10)
[15:35:04] ^-- Disable NUMA memory reclaim (set vm.zone_reclaim_mode=0)
[15:35:04] ^-- Avoid direct reclaim and page allocation failures (set
vm.extra_free_kbytes=1240000)
[15:35:04] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[15:35:04]
[15:35:04] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[15:35:04]
[15:35:04] Ignite node started OK (id=b9c64c6d)
[15:35:04] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24,
heap=22.0GB]
Done....cache
Thanks,
Prashant
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13448.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Michael Cherkasov <mi...@gmail.com>.
Hi Prashant,
Could you please also send CreateCache's log ?
Also please try to specify non loop-back ip instead of 127.0.0.1 ?
what linux dist do you use?
Thanks,
Mikhail.
2017-06-06 16:36 GMT+03:00 Prashant312 <p....@crif.com>:
> Hi,
>
> We are first running the Createcache.java and then running testMapper
> class.
> But still we are getting the same issue, From another Java Program not MR
> we
> are able to see Ignite cache variable. Please see below logs....
>
> [p_hadoop@HMCISPRDATAWARE ~]$ jps
> 28720 DataNode
> 29236 ResourceManager
> 31492 Bootstrap
> 30005 HMaster
> 29960 RunJar
> 29000 SecondaryNameNode
> 24728 CreateCache
> 20025 Elasticsearch
> 28587 NameNode
> 5724 Jps
> 29374 NodeManager
> [p_hadoop@HMCISPRDATAWARE ~]$
>
> ------------------------------------------------------------------------
>
> [00:31:11] Message queue limit is set to 0 which may lead to potential
> OOMEs
> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
> message queues growth on sender and receiver sides.
> [00:31:11] Security status [authentication=off, tls/ssl=off]
> [00:31:11] REST protocols do not start on client node. To start the
> protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
> property.
> [00:31:11] HADOOP_HOME is set to /home/p_hadoop/hadoop-2.7.3
> [00:31:11] Resolved Hadoop classpath locations:
> /home/p_hadoop/hadoop-2.7.3,
> /home/p_hadoop/hadoop-2.7.3, /home/p_hadoop/hadoop-2.7.3
> 17/06/07 00:31:12 INFO mapreduce.Job: Job job_local606348237_0001 running
> in
> uber mode : false
> 17/06/07 00:31:12 INFO mapreduce.Job: map 0% reduce 0%
> [00:31:31] Failed to connect to any address from IP finder (will retry to
> join topology every 2 secs): [/127.0.0.1:47500, /127.0.0.1:47500,
> /127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
> /127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
> /127.0.0.1:47509]
>
>
> Thanks,
> Prashant
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cache-returning-Null-value-
> from-Mapper-tp13158p13415.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hi,
We are first running the Createcache.java and then running testMapper class.
But still we are getting the same issue, From another Java Program not MR we
are able to see Ignite cache variable. Please see below logs....
[p_hadoop@HMCISPRDATAWARE ~]$ jps
28720 DataNode
29236 ResourceManager
31492 Bootstrap
30005 HMaster
29960 RunJar
29000 SecondaryNameNode
24728 CreateCache
20025 Elasticsearch
28587 NameNode
5724 Jps
29374 NodeManager
[p_hadoop@HMCISPRDATAWARE ~]$
------------------------------------------------------------------------
[00:31:11] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[00:31:11] Security status [authentication=off, tls/ssl=off]
[00:31:11] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
[00:31:11] HADOOP_HOME is set to /home/p_hadoop/hadoop-2.7.3
[00:31:11] Resolved Hadoop classpath locations: /home/p_hadoop/hadoop-2.7.3,
/home/p_hadoop/hadoop-2.7.3, /home/p_hadoop/hadoop-2.7.3
17/06/07 00:31:12 INFO mapreduce.Job: Job job_local606348237_0001 running in
uber mode : false
17/06/07 00:31:12 INFO mapreduce.Job: map 0% reduce 0%
[00:31:31] Failed to connect to any address from IP finder (will retry to
join topology every 2 secs): [/127.0.0.1:47500, /127.0.0.1:47500,
/127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
/127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
/127.0.0.1:47509]
Thanks,
Prashant
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13415.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Michael Cherkasov <mi...@gmail.com>.
Hi Prashant,
In Mapper you run Ignite in client mode, it requires server instance to be
running.
I can the same error:
[15:18:56] Failed to connect to any address from IP finder (will retry to
join topology every 2 secs): [/127.0.0.1:47500, /127.0.0.1:47500, /
127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504, /
127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508, /
127.0.0.1:47509]
if I don't run CreateCache.java.
Make sure that you first run CreateCache.java and there're no exceptions in
log.
Before run TestMapper.java make sure that CreateCache.java is working,
run the following command: jps
CreateCache should be in the list, only then run TestMapper.java
Thanks,
Mikhail.
2017-06-06 6:02 GMT+03:00 Prashant312 <p....@crif.com>:
> [13:57:27] Failed to connect to any address from IP finder (will retry to
> join topology every 2 secs): [/127.0.0.1:47500, /127.0.0.1:47500,
> /127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
> /127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
> /127.0.0.1:47509]
>
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hi,
I just copy paste your code but still it is not working in our environment.
The mapper program is getting stuck with below logs
setup called
TcpDiscoveryVmIpFinder [addrs=[/127.0.0.1:0, /127.0.0.1:47500,
/127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
/127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
/127.0.0.1:47509], super=TcpDiscoveryIpFinderAdapter [shared=false]]
Jun 06, 2017 1:57:06 PM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
[13:57:06] __________ ________________
[13:57:06] / _/ ___/ |/ / _/_ __/ __/
[13:57:06] _/ // (7 7 // / / / / _/
[13:57:06] /___/\___/_/|_/___/ /_/ /___/
[13:57:06]
[13:57:06] ver. 2.0.0#20170430-sha1:d4eef3c6
[13:57:06] 2017 Copyright(C) Apache Software Foundation
[13:57:06]
[13:57:06] Ignite documentation: http://ignite.apache.org
[13:57:06]
[13:57:06] Quiet mode.
[13:57:06] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[13:57:06]
[13:57:06] OS: Linux 2.6.32-431.el6.x86_64 amd64
[13:57:06] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
[13:57:07] Configured plugins:
[13:57:07] ^-- None
[13:57:07]
[13:57:07] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[13:57:07] Security status [authentication=off, tls/ssl=off]
[13:57:07] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
[13:57:07] HADOOP_HOME is set to /home/p_hadoop/hadoop-2.7.3
[13:57:07] Resolved Hadoop classpath locations: /home/p_hadoop/hadoop-2.7.3,
/home/p_hadoop/hadoop-2.7.3, /home/p_hadoop/hadoop-2.7.3
[13:57:27] Failed to connect to any address from IP finder (will retry to
join topology every 2 secs): [/127.0.0.1:47500, /127.0.0.1:47500,
/127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503, /127.0.0.1:47504,
/127.0.0.1:47505, /127.0.0.1:47506, /127.0.0.1:47507, /127.0.0.1:47508,
/127.0.0.1:47509]
We are using single node cluster of hadoop and createcache and mapper
program both are running on the same server.
Thanks,
Prashant Verma
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13394.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Michael Cherkasov <mi...@gmail.com>.
Hi Prashant,
I've run your code and it works fine for me, I see "abc" in output.
However I wan't able to run it as is, I changed it a little bit.
First, I added TcpDiscoverySpi in CreateCache.java:
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1", "127.0.0.1:47500
..47509"));
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(spi);
then I run CreateCache.java and after this run TestMapper.java , in output
there was:
"Ignite started
in map
abc
Cleanup called"
So looks like it works fine.
Please find my mvn project there:
https://www.dropbox.com/s/4xvnvdjoxjrvmgm/general.tgz?dl=0
Thanks,
Mikhail.
2017-06-02 8:37 GMT+03:00 Prashant312 <p....@crif.com>:
> Hello Michael,
>
> Thanks for writing, We are just using Core Java to start the Ignite
> instance
> and fetch value from that Ignite instance through mapper.
> Below are the 3 class file which we have used.
>
> Createcache is used to put data.
>
> Startignite is called from setup method of maaper.
>
> Inside testmapper we are starting Ignite from setup method and
> trying to
> get values in map method, which is returning NULL.
>
> PS - Without mapper class if we try to access the cache value from other
> Java Program we are able to do so.
>
> Waiting for your positive reply.
> -------------------
>
> package ignite;
>
>
> import java.util.Arrays;
>
> import javax.cache.configuration.Factory;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> import org.apache.poi.hssf.record.formula.functions.Minute;
>
>
> public class CreateCache {
>
> public static IgniteCache<String, String> igniteCache=null;
>
>
> public static void main(String[] args) {
>
>
> /*TcpDiscoverySpi spi = new TcpDiscoverySpi();
>
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("10.10.10.138",
> "10.10.10.138:47500..47509"));
> spi.setIpFinder(ipFinder);
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> // Override default discovery SPI.
> cfg.setDiscoverySpi(spi);*/
> //Ignition.setClientMode(false);
> Ignite ignite = Ignition.start();
>
>
> CacheConfiguration<String, String> cacheConfig=new
> CacheConfiguration<String, String>();
>
> cacheConfig.setBackups(1);
>
> cacheConfig.setName("matchCache");
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfig.setCacheMode(CacheMode.REPLICATED);
>
> igniteCache=ignite.getOrCreateCache(cacheConfig);
>
> igniteCache.put("100", "abc");
> //igniteCache.
>
> System.out.println("Done....cache");
> }
>
> public static Object getCacheData(String key){
>
> if(igniteCache!=null){
> return igniteCache.get(key);
> }
> return key;
> }
>
> }
> -------------------------------------------------------
> package ignite;
>
> import java.util.Arrays;
> import java.util.Properties;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteSystemProperties;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> import static
> org.apache.ignite.IgniteSystemProperties.IGNITE_REST_START_ON_CLIENT;
>
>
> public class StartIgnite {
>
> public static IgniteCache<String, String> igniteCache=null;
>
> public static void startIgnite(){
>
>
> Ignition.setClientMode(true);
> /*TcpDiscoverySpi spi = new TcpDiscoverySpi();
>
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
>
> // Set initial IP addresses.
> // Note that you can optionally specify a port or a port
> range.
> ipFinder.setAddresses(Arrays.asList("127.0.0.1",
> "127.0.0.1:47500..47509"));
>
> spi.setIpFinder(ipFinder);*/
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1",
> "127.0.0.1:47500..47509"));
> spi.setIpFinder(ipFinder);
> System.out.println(ipFinder);
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> cfg.setDiscoverySpi(spi);
>
> Ignite ignite = Ignition.start(cfg);
>
> CacheConfiguration<String, String> cacheConfig=new
> CacheConfiguration<String, String>();
> cacheConfig.setName("matchCache");
> //cacheConfig.
> //cacheConfig.s
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> igniteCache=ignite.getOrCreateCache(cacheConfig);
> System.out.println("cache name is "+
> igniteCache.getName());
>
> }
>
> public static void stopIgnite(){
> Ignition.stop(true);
> }
>
> public static String getCacheData(String id){
>
> return igniteCache.get(id);
> }
>
> public static void main(String[] args) {
>
> startIgnite();
> //System.out.println(Cache.getCacheData(CacheConstants.
> CLEANSING_TOKENS));
>
> String obj=igniteCache.get("200");
> String obj2=igniteCache.get("100");
> System.out.println(obj);
> System.out.println(obj2);
>
> stopIgnite();
>
> }
>
> }
> ---------------------------------------------------------------
> package mapper;
>
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import ignite.StartIgnite;
>
> public class TestMapper {
>
> public static void main(String[] args) throws Exception
>
> {
> Configuration c = new Configuration();
> Path input = new Path(args[0]);
> Path output = new Path(args[1]);
> Job j = new Job(c, "wordcount");
> j.setJarByClass(TestMapper.class);
> j.setMapperClass(MapForWordCount.class);
> j.setReducerClass(ReduceForWordCount.class);
> j.setOutputKeyClass(Text.class);
> j.setOutputValueClass(IntWritable.class);
> FileInputFormat.addInputPath(j, input);
> FileOutputFormat.setOutputPath(j, output);
> System.exit(j.waitForCompletion(true) ? 0 : 1);
>
> }
>
> public static class MapForWordCount extends Mapper<LongWritable,
> Text,
> Text, IntWritable> {
>
> boolean flag=false;
>
>
> @Override
> protected void setup(Context context) throws IOException,
> InterruptedException {
> System.out.println("setup called");
> StartIgnite.startIgnite();
> System.out.println("Ignite started ");
> }
>
> @Override
> protected void cleanup(Context context) throws IOException,
> InterruptedException {
> System.out.println("Cleanup called");
> StartIgnite.stopIgnite();
> }
>
> public void map(LongWritable key, Text value, Context con)
> throws
> IOException, InterruptedException
>
> {
> System.out.println("in map");
> String obj=StartIgnite.getCacheData("100");
> System.out.println(obj);
> String line = value.toString();
>
> String[] words = line.split(" ");
>
> for (String word : words){
> Text outputKey = new
> Text(word.toUpperCase().trim());
> IntWritable outputValue = new
> IntWritable(1);
> con.write(outputKey, outputValue);
> }
> }
> }
> public static class ReduceForWordCount extends Reducer<Text,
> IntWritable,
> Text, IntWritable>
>
> {
>
> public void reduce(Text word, Iterable<IntWritable>
> values, Context con)
> throws IOException, InterruptedException
>
> {
> int sum = 0;
> for (IntWritable value : values){
> sum += value.get();
> }
> con.write(word, new IntWritable(sum));
> }
> }
> }
> -------------------------------------------------------
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cache-returning-Null-value-
> from-Mapper-tp13158p13332.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hello Mikhail,
By setting system property *java.net.preferIPv4Stack* to true we are able to
get the required cache data from mapper.
Thanks a lot for your help.
Thanks,
Prashant Verma
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13560.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Cache returning Null value from Mapper
Posted by Michael Cherkasov <mi...@gmail.com>.
Hi Prashant,
I was able to reproduce the issue. It happens because TestMapper knows all
about hadoop
and CreateCache doesn't. So I believe TestMapper has the following
dependency:
Do you have this dependency? :
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-hadoop</artifactId>
<version>2.0.0</version>
</dependency>
Try to exclude it.
This dependency says to Ignite that the whole cluster works with hadoop and
each node
has hadoop classes, but in you case CreateCache doesn't have hadoop classes
in class path.
So you should exclude ignite-hadoop from dependencies or add all hadoop
related classes to
CreateCache class path.
Thanks,
Mikhail.
2017-06-02 8:37 GMT+03:00 Prashant312 <p....@crif.com>:
> Hello Michael,
>
> Thanks for writing, We are just using Core Java to start the Ignite
> instance
> and fetch value from that Ignite instance through mapper.
> Below are the 3 class file which we have used.
>
> Createcache is used to put data.
>
> Startignite is called from setup method of maaper.
>
> Inside testmapper we are starting Ignite from setup method and
> trying to
> get values in map method, which is returning NULL.
>
> PS - Without mapper class if we try to access the cache value from other
> Java Program we are able to do so.
>
> Waiting for your positive reply.
> -------------------
>
> package ignite;
>
>
> import java.util.Arrays;
>
> import javax.cache.configuration.Factory;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> import org.apache.poi.hssf.record.formula.functions.Minute;
>
>
> public class CreateCache {
>
> public static IgniteCache<String, String> igniteCache=null;
>
>
> public static void main(String[] args) {
>
>
> /*TcpDiscoverySpi spi = new TcpDiscoverySpi();
>
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("10.10.10.138",
> "10.10.10.138:47500..47509"));
> spi.setIpFinder(ipFinder);
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> // Override default discovery SPI.
> cfg.setDiscoverySpi(spi);*/
> //Ignition.setClientMode(false);
> Ignite ignite = Ignition.start();
>
>
> CacheConfiguration<String, String> cacheConfig=new
> CacheConfiguration<String, String>();
>
> cacheConfig.setBackups(1);
>
> cacheConfig.setName("matchCache");
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfig.setCacheMode(CacheMode.REPLICATED);
>
> igniteCache=ignite.getOrCreateCache(cacheConfig);
>
> igniteCache.put("100", "abc");
> //igniteCache.
>
> System.out.println("Done....cache");
> }
>
> public static Object getCacheData(String key){
>
> if(igniteCache!=null){
> return igniteCache.get(key);
> }
> return key;
> }
>
> }
> -------------------------------------------------------
> package ignite;
>
> import java.util.Arrays;
> import java.util.Properties;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteSystemProperties;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> import static
> org.apache.ignite.IgniteSystemProperties.IGNITE_REST_START_ON_CLIENT;
>
>
> public class StartIgnite {
>
> public static IgniteCache<String, String> igniteCache=null;
>
> public static void startIgnite(){
>
>
> Ignition.setClientMode(true);
> /*TcpDiscoverySpi spi = new TcpDiscoverySpi();
>
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
>
> // Set initial IP addresses.
> // Note that you can optionally specify a port or a port
> range.
> ipFinder.setAddresses(Arrays.asList("127.0.0.1",
> "127.0.0.1:47500..47509"));
>
> spi.setIpFinder(ipFinder);*/
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1",
> "127.0.0.1:47500..47509"));
> spi.setIpFinder(ipFinder);
> System.out.println(ipFinder);
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> cfg.setDiscoverySpi(spi);
>
> Ignite ignite = Ignition.start(cfg);
>
> CacheConfiguration<String, String> cacheConfig=new
> CacheConfiguration<String, String>();
> cacheConfig.setName("matchCache");
> //cacheConfig.
> //cacheConfig.s
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> igniteCache=ignite.getOrCreateCache(cacheConfig);
> System.out.println("cache name is "+
> igniteCache.getName());
>
> }
>
> public static void stopIgnite(){
> Ignition.stop(true);
> }
>
> public static String getCacheData(String id){
>
> return igniteCache.get(id);
> }
>
> public static void main(String[] args) {
>
> startIgnite();
> //System.out.println(Cache.getCacheData(CacheConstants.
> CLEANSING_TOKENS));
>
> String obj=igniteCache.get("200");
> String obj2=igniteCache.get("100");
> System.out.println(obj);
> System.out.println(obj2);
>
> stopIgnite();
>
> }
>
> }
> ---------------------------------------------------------------
> package mapper;
>
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import ignite.StartIgnite;
>
> public class TestMapper {
>
> public static void main(String[] args) throws Exception
>
> {
> Configuration c = new Configuration();
> Path input = new Path(args[0]);
> Path output = new Path(args[1]);
> Job j = new Job(c, "wordcount");
> j.setJarByClass(TestMapper.class);
> j.setMapperClass(MapForWordCount.class);
> j.setReducerClass(ReduceForWordCount.class);
> j.setOutputKeyClass(Text.class);
> j.setOutputValueClass(IntWritable.class);
> FileInputFormat.addInputPath(j, input);
> FileOutputFormat.setOutputPath(j, output);
> System.exit(j.waitForCompletion(true) ? 0 : 1);
>
> }
>
> public static class MapForWordCount extends Mapper<LongWritable,
> Text,
> Text, IntWritable> {
>
> boolean flag=false;
>
>
> @Override
> protected void setup(Context context) throws IOException,
> InterruptedException {
> System.out.println("setup called");
> StartIgnite.startIgnite();
> System.out.println("Ignite started ");
> }
>
> @Override
> protected void cleanup(Context context) throws IOException,
> InterruptedException {
> System.out.println("Cleanup called");
> StartIgnite.stopIgnite();
> }
>
> public void map(LongWritable key, Text value, Context con)
> throws
> IOException, InterruptedException
>
> {
> System.out.println("in map");
> String obj=StartIgnite.getCacheData("100");
> System.out.println(obj);
> String line = value.toString();
>
> String[] words = line.split(" ");
>
> for (String word : words){
> Text outputKey = new
> Text(word.toUpperCase().trim());
> IntWritable outputValue = new
> IntWritable(1);
> con.write(outputKey, outputValue);
> }
> }
> }
> public static class ReduceForWordCount extends Reducer<Text,
> IntWritable,
> Text, IntWritable>
>
> {
>
> public void reduce(Text word, Iterable<IntWritable>
> values, Context con)
> throws IOException, InterruptedException
>
> {
> int sum = 0;
> for (IntWritable value : values){
> sum += value.get();
> }
> con.write(word, new IntWritable(sum));
> }
> }
> }
> -------------------------------------------------------
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
> Mikhail wrote
> > Hi Prashant,
> >
> > Could you please provide more details about your configuration?
> > Maybe you can send a test case that will show the issue?
> >
> > PS you email didn't get to mail list, because looks like you haven't
> > subscribed to the list.
> > Please subscribe to the list before sending further emails.
> >
> > Thanks,
> > Mikhail.
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cache-returning-Null-value-
> from-Mapper-tp13158p13332.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Cache returning Null value from Mapper
Posted by Prashant312 <p....@crif.com>.
Hello Michael,
Thanks for writing, We are just using Core Java to start the Ignite instance
and fetch value from that Ignite instance through mapper.
Below are the 3 class file which we have used.
Createcache is used to put data.
Startignite is called from setup method of maaper.
Inside testmapper we are starting Ignite from setup method and trying to
get values in map method, which is returning NULL.
PS - Without mapper class if we try to access the cache value from other
Java Program we are able to do so.
Waiting for your positive reply.
-------------------
package ignite;
import java.util.Arrays;
import javax.cache.configuration.Factory;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import org.apache.poi.hssf.record.formula.functions.Minute;
public class CreateCache {
public static IgniteCache<String, String> igniteCache=null;
public static void main(String[] args) {
/*TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("10.10.10.138",
"10.10.10.138:47500..47509"));
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);*/
//Ignition.setClientMode(false);
Ignite ignite = Ignition.start();
CacheConfiguration<String, String> cacheConfig=new
CacheConfiguration<String, String>();
cacheConfig.setBackups(1);
cacheConfig.setName("matchCache");
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfig.setCacheMode(CacheMode.REPLICATED);
igniteCache=ignite.getOrCreateCache(cacheConfig);
igniteCache.put("100", "abc");
//igniteCache.
System.out.println("Done....cache");
}
public static Object getCacheData(String key){
if(igniteCache!=null){
return igniteCache.get(key);
}
return key;
}
}
-------------------------------------------------------
package ignite;
import java.util.Arrays;
import java.util.Properties;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteSystemProperties;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import static
org.apache.ignite.IgniteSystemProperties.IGNITE_REST_START_ON_CLIENT;
public class StartIgnite {
public static IgniteCache<String, String> igniteCache=null;
public static void startIgnite(){
Ignition.setClientMode(true);
/*TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("127.0.0.1",
"127.0.0.1:47500..47509"));
spi.setIpFinder(ipFinder);*/
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1",
"127.0.0.1:47500..47509"));
spi.setIpFinder(ipFinder);
System.out.println(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(spi);
Ignite ignite = Ignition.start(cfg);
CacheConfiguration<String, String> cacheConfig=new
CacheConfiguration<String, String>();
cacheConfig.setName("matchCache");
//cacheConfig.
//cacheConfig.s
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
igniteCache=ignite.getOrCreateCache(cacheConfig);
System.out.println("cache name is "+ igniteCache.getName());
}
public static void stopIgnite(){
Ignition.stop(true);
}
public static String getCacheData(String id){
return igniteCache.get(id);
}
public static void main(String[] args) {
startIgnite();
//System.out.println(Cache.getCacheData(CacheConstants.CLEANSING_TOKENS));
String obj=igniteCache.get("200");
String obj2=igniteCache.get("100");
System.out.println(obj);
System.out.println(obj2);
stopIgnite();
}
}
---------------------------------------------------------------
package mapper;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import ignite.StartIgnite;
public class TestMapper {
public static void main(String[] args) throws Exception
{
Configuration c = new Configuration();
Path input = new Path(args[0]);
Path output = new Path(args[1]);
Job j = new Job(c, "wordcount");
j.setJarByClass(TestMapper.class);
j.setMapperClass(MapForWordCount.class);
j.setReducerClass(ReduceForWordCount.class);
j.setOutputKeyClass(Text.class);
j.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(j, input);
FileOutputFormat.setOutputPath(j, output);
System.exit(j.waitForCompletion(true) ? 0 : 1);
}
public static class MapForWordCount extends Mapper<LongWritable, Text,
Text, IntWritable> {
boolean flag=false;
@Override
protected void setup(Context context) throws IOException,
InterruptedException {
System.out.println("setup called");
StartIgnite.startIgnite();
System.out.println("Ignite started ");
}
@Override
protected void cleanup(Context context) throws IOException,
InterruptedException {
System.out.println("Cleanup called");
StartIgnite.stopIgnite();
}
public void map(LongWritable key, Text value, Context con) throws
IOException, InterruptedException
{
System.out.println("in map");
String obj=StartIgnite.getCacheData("100");
System.out.println(obj);
String line = value.toString();
String[] words = line.split(" ");
for (String word : words){
Text outputKey = new Text(word.toUpperCase().trim());
IntWritable outputValue = new IntWritable(1);
con.write(outputKey, outputValue);
}
}
}
public static class ReduceForWordCount extends Reducer<Text, IntWritable,
Text, IntWritable>
{
public void reduce(Text word, Iterable<IntWritable> values, Context con)
throws IOException, InterruptedException
{
int sum = 0;
for (IntWritable value : values){
sum += value.get();
}
con.write(word, new IntWritable(sum));
}
}
}
-------------------------------------------------------
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
Mikhail wrote
> Hi Prashant,
>
> Could you please provide more details about your configuration?
> Maybe you can send a test case that will show the issue?
>
> PS you email didn't get to mail list, because looks like you haven't
> subscribed to the list.
> Please subscribe to the list before sending further emails.
>
> Thanks,
> Mikhail.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-returning-Null-value-from-Mapper-tp13158p13332.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.