You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Alena Melnikova <al...@74.ru> on 2017/05/02 14:45:02 UTC
Re: HDP, Hive + Ignite
Hi Ivan,
I have some progress)
*1. TEZ on Ignite (with IGFS, without Ignite MR)*
I could run Hive queries on TEZ and Ignite with next settings:
$IGNITE_HOME/bin/ignite.sh -v -J"-Xms10g -Xmx10g -XX:MaxMetaspaceSize=4g"
(every server has RAM 16Gb )
beeline --hiveconf fs.default.name=igfs://dev-dn1:10500 --hiveconf
ignite.job.shared.classloader=false
set tez.use.cluster.hadoop-libs = true; (to avoid
"java.lang.ClassNotFoundException: Class
org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem not found")
ignite.job.shared.classloader = false;
hive.rpc.query.plan = true;
hive.execution.engine = tez;
select calday, count(*) from price.toprice where calday between '2017-03-01'
and '2017-03-21' group by calday order by calday;
I run this query 8 times on TEZ+Ingnite and 8 times just on TEZ (without
IGFS), threw out the best and worst result and calculated average.
Results are:
Average execution time TEZ+Ignite: 25 sec
Average execution time just TEZ: 23 sec
Then I run more complex analytical query with joins on the same conditions.
Results are:
Average execution time TEZ+Ignite: 312 sec
Average execution time just TEZ: 313 sec
Results are mostly identical, so I guess IGFS is not used.
May be I should explicitly tell Hive to cache data in IGFS?
Is there any way to understand that Ignite is used besides measuring
execution time?
*2. Ignite MR (with IGFS, with Ignite MR)*
I could run Hive queries on Ignite MR with next settings:
$IGNITE_HOME/bin/ignite.sh -v -J"-Xms10g -Xmx10g -XX:MaxMetaspaceSize=4g"
(every server has RAM 16Gb )
beeline --hiveconf fs.default.name=igfs://dev-dn1:10500 --hiveconf
ignite.job.shared.classloader=false
ignite.job.shared.classloader = false;
mapreduce.jobtracker.address=dev-dn1.co.vectis.local:11211;
hive.rpc.query.plan = true;
hive.execution.engine = mr;
select calday, count(*) from price.toprice where calday between '2017-03-01'
and '2017-03-21' group by calday order by calday;
If I use one ignite node it returns correct answer but much slower - 80 sec
vs 23 sec on TEZ.
If I run this query on two or more nodes then result is not correct. As I
can see there are no any errors in logs.
What is wrong?
ignite-node-dn1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12344/ignite-node-dn1.log>
ignite-node-dn2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12344/ignite-node-dn2.log>
3. When I start ignite nodes on different servers sometimes they do not see
each other. I have to rerun a node a few times, after that they connect in
one cluster. Is it normal?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12344.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
TEZ was on 6 data nodes. So you're right, I can't reliably estimate the
performance of Ignite MR.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12868.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by "Ivan V." <iv...@gridgain.com>.
Alena,
wrt 1.: saying 80 vs. 23 sec , do you compare Ignite MR on 1 node vs. Tez
on *1* node also?
On Mon, May 15, 2017 at 5:03 PM, Alena Melnikova <al...@74.ru> wrote:
> Ivan,
>
> 1. In my environment Ignite MR works correctly only on one node and it
> works
> slower than TEZ (80 sec vs 23 sec). I guess because of one ignite node. On
> multi node cluster result was incorrect.
>
> 2. "Do I correctly understand that Ignite MR was not used in that
> experiment?"
> Yes, it was TEZ+IGFS.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12850.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Ivan,
1. In my environment Ignite MR works correctly only on one node and it works
slower than TEZ (80 sec vs 23 sec). I guess because of one ignite node. On
multi node cluster result was incorrect.
2. "Do I correctly understand that Ignite MR was not used in that
experiment?"
Yes, it was TEZ+IGFS.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12850.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Evgeniy Stanilovskiy <es...@gridgain.com>.
Ivan, what kind of tests did u run ? plz show SQL requests ?
1-node tests for distributed computing looks like wierd
> My observations (on very simplified 1-node environment) show that
> Ignite-MR ~10% faster than TEZ under equal >conditions.
>
> On Mon, May 15, 2017 at 1:52 PM, Ivan V. <iv...@gridgain.com>
> wrote:
>> Hi, Alena, regarding "1) Ignite MR works slower than Hive on TEZ, but
>> faster than Hive on MR." -- as far as >>I remember, you have observed
>> incorrect results with Ignite MR, and we didn't find the reason, just
>> abandoned that. >>Performance measurements don't have much sense until
>> we have correct query results. So, I would say that there we >>just
>> don't have results we can trust.
>>
>> Regarding ""Out of memory: Kill process" -- this means that Ignite node
>> process requested so much memory that OS >>failed to give. This may be
>> investigated further -- all the memory limits set for Ignite node
>> should be checked and >>compared to the real memory physically
>> available on the host. Do I correctly understand that Ignite MR was not
>> used >>in that experiment?
>>
>> On Mon, May 15, 2017 at 9:32 AM, Alena Melnikova <al...@74.ru> wrote:
>>> Hi Ivan,
>>>
>>> You're right. In kernel log there is message: "Out of memory: Kill
>>> process
>>> 19988 (java)"
>>>
>>> Let me sum up, please, correct me if I'm wrong.
>>> If we use Hive + Tez we don't need Hadoop Accelerator because:
>>> 1) Ignite MR works slower than Hive on TEZ, but faster than Hive on MR.
>>> 2) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can
>>> be
>>> faster in queries with intensive I/O (need to test).
>>>
>>> Many thanks for your patience and prompt help.
>>> I'm going to try Ignite + Spark, I'll open new topic)
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12838.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by "Ivan V." <iv...@gridgain.com>.
My observations (on very simplified 1-node environment) show that Ignite-MR
~10% faster than TEZ under equal conditions.
On Mon, May 15, 2017 at 1:52 PM, Ivan V. <iv...@gridgain.com> wrote:
> Hi, Alena, regarding "1) Ignite MR works slower than Hive on TEZ, but
> faster than Hive on MR." -- as far as I remember, you have observed
> incorrect results with Ignite MR, and we didn't find the reason, just
> abandoned that. Performance measurements don't have much sense until we
> have correct query results. So, I would say that there we just don't have
> results we can trust.
>
> Regarding ""Out of memory: Kill process" -- this means that Ignite node
> process requested so much memory that OS failed to give. This may be
> investigated further -- all the memory limits set for Ignite node should be
> checked and compared to the real memory physically available on the host.
> Do I correctly understand that Ignite MR was not used in that experiment?
>
> On Mon, May 15, 2017 at 9:32 AM, Alena Melnikova <al...@74.ru> wrote:
>
>> Hi Ivan,
>>
>> You're right. In kernel log there is message: "Out of memory: Kill process
>> 19988 (java)"
>>
>> Let me sum up, please, correct me if I'm wrong.
>> If we use Hive + Tez we don't need Hadoop Accelerator because:
>> 1) Ignite MR works slower than Hive on TEZ, but faster than Hive on MR.
>> 2) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can be
>> faster in queries with intensive I/O (need to test).
>>
>> Many thanks for your patience and prompt help.
>> I'm going to try Ignite + Spark, I'll open new topic)
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/HDP-Hive-Ignite-tp12195p12838.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: HDP, Hive + Ignite
Posted by "Ivan V." <iv...@gridgain.com>.
Hi, Alena, regarding "1) Ignite MR works slower than Hive on TEZ, but
faster than Hive on MR." -- as far as I remember, you have observed
incorrect results with Ignite MR, and we didn't find the reason, just
abandoned that. Performance measurements don't have much sense until we
have correct query results. So, I would say that there we just don't have
results we can trust.
Regarding ""Out of memory: Kill process" -- this means that Ignite node
process requested so much memory that OS failed to give. This may be
investigated further -- all the memory limits set for Ignite node should be
checked and compared to the real memory physically available on the host.
Do I correctly understand that Ignite MR was not used in that experiment?
On Mon, May 15, 2017 at 9:32 AM, Alena Melnikova <al...@74.ru> wrote:
> Hi Ivan,
>
> You're right. In kernel log there is message: "Out of memory: Kill process
> 19988 (java)"
>
> Let me sum up, please, correct me if I'm wrong.
> If we use Hive + Tez we don't need Hadoop Accelerator because:
> 1) Ignite MR works slower than Hive on TEZ, but faster than Hive on MR.
> 2) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can be
> faster in queries with intensive I/O (need to test).
>
> Many thanks for your patience and prompt help.
> I'm going to try Ignite + Spark, I'll open new topic)
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12838.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
You're right. In kernel log there is message: "Out of memory: Kill process
19988 (java)"
Let me sum up, please, correct me if I'm wrong.
If we use Hive + Tez we don't need Hadoop Accelerator because:
1) Ignite MR works slower than Hive on TEZ, but faster than Hive on MR.
2) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can be
faster in queries with intensive I/O (need to test).
Many thanks for your patience and prompt help.
I'm going to try Ignite + Spark, I'll open new topic)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12838.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by "Ivan V." <iv...@gridgain.com>.
Alena,
regarding comparison of your Hive query on TEZ+HDFS vs. TEZ+IGFS: my
experiments show same results (~58 sec in average) for both. At least not
distinguishable within dispersion.
I suppose, in this usecase fetching the table data takes negligible time as
compared to overall task processing time.
(I used primary IGFS mode and explicitly loaded the data to avoid any
non-cached data cold start effects.)
IGFS may give noticeable speedup for tasks that really involve large disk
I/O, this one does not seem to be such. Also please note, that some file
data are cached in memory by the operating system, so even if you read
from disk you frequently read from memory, in fact.
On Fri, May 12, 2017 at 3:52 PM, Ivan Veselovsky <iv...@gridgain.com>
wrote:
> Alena,
> as I understand, the message "19988 Killed "$JAVA"" means that the Ignite
> node process was killed by the operating system. Can you please see the
> kernel log -- what does it say near the node crash time?
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12660.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Alena,
as I understand, the message "19988 Killed "$JAVA"" means that the Ignite
node process was killed by the operating system. Can you please see the
kernel log -- what does it say near the node crash time?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12660.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
Yes, it helps to avoid NPEs!
<property name="perNodeParallelBatchCount" value="1"/>
<property name="prefetchBlocks" value="0"/>
Though, from time to time one node dies. Usually this is the node that I
specify when I start the beeline:
beeline --hiveconf fs.default.name=igfs://dev-dn1:1050
/home/ignite/apache-ignite-hadoop-1.9.0-bin/bin/ignite.sh: line 170: 19988
Killed "$JAVA" ${JVM_OPTS} ${QUIET} "${RESTART_SUCCESS_OPT}" ${JMX_MON}
-DIGNITE_HOME="${IGNITE_HOME}" -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp
"${CP}" ${MAIN_CLASS} "${CONFIG}"
Full log in previous post (ignite-node-dn1_1.log).
However, average execution time on TEZ (table in HDFS) and TEZ (table in
IGFS) is comparable:
TEZ: 215 sec (6 nodes)
TEZ+IGFS: 207 sec (6 nodes)
I'm waiting for the results of your tests.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12638.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
As a workaround to IGNITE-4862 propetry
FileSystemConfiguration#perNodeParallelBatchCount can be set to 1.
Also setting FileSystemConfiguration#prefetchBlocks to 0 should help.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12627.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Alena, regarding NPEs in Ignite node logs, this seems to be
https://issues.apache.org/jira/browse/IGNITE-4862 , fixed, but not yet
merged.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12594.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
1. I tried to run analytical query on table that created in IGFS. Here
couple of examples of errors.
beeline_output_1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12588/beeline_output_1.log>
ignite-node-dn1_1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12588/ignite-node-dn1_1.log>
beeline_output_2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12588/beeline_output_2.log>
ignite-node-dn1_2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12588/ignite-node-dn1_2.log>
4. We are looking forward results of your experiments.
p.s. I sent email again on 5th May.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12588.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Evgeniy Stanilovskiy <es...@gridgain.com>.
I keep in mind ticket for TEZ experimenting\investigations.
> Evgeniy, sure, IGNITE-4862, the link is above.
>
> On Thu, May 11, 2017 at 11:07 AM, Evgeniy Stanilovskiy
> <es...@gridgain.com> wrote:
>> Ivan, do we have appropriate jira ticket?
>>
>>> 4. No. We only start experimenting with Tez -- I'm currently setting
>>> it up in my environment to investigate the >>>problems.
Re: HDP, Hive + Ignite
Posted by Evgeniy Stanilovskiy <es...@gridgain.com>.
Ivan, do we have appropriate jira ticket?
> 4. No. We only start experimenting with Tez -- I'm currently setting it
> up in my environment to investigate the >problems.
Re: HDP, Hive + Ignite
Posted by "Ivan V." <iv...@gridgain.com>.
1. Please attach full logs.
3. I might suspect the property "shared.classloader" , but if it is
definitely set to 'false' , and there is no error in the logs, I have no
other ideas at the moment.
4. No. We only start experimenting with Tez -- I'm currently setting it up
in my environment to investigate the problems.
p.s. No, I did not. Can you please send it again.
On Fri, May 5, 2017 at 3:39 PM, Alena Melnikova <al...@74.ru> wrote:
> Hi Ivan,
>
> 1. I still continue to experiment with the table created in IGFS. Currently
> it works if query is executed only once, then either the ignite node fails
> or error: Exception in thread "igfs-#60%null%"
> java.lang.NullPointerException.
>
> 3. You're right! I didn't restart Visor. What's more, I run one Visor as
> root (and forgot about it) and one as user ignite. After killing all of
> them
> new topology starts from ver=1.
> But it doesn't help for correct Ignite MR. Topology is correct, there is no
> any error in logs, but the result is wrong. To be honest, I decided to stop
> experiments with IgniteMR and focus on IGFS+TEZ or Spark.
> beeline_ignite_mr.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n12457/beeline_ignite_mr.log>
> ignite-node-dn1.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n12457/ignite-node-dn1.log>
> ignite-node-dn2.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n12457/ignite-node-dn2.log>
>
> 4. I think Evgeniy said about comparison TEZ vs TEZ+IGFS. So I join the
> question:
> did you conduct some tests Ignite + TEZ?
>
> p.s. Ivan, did you get my email about Hadoop meetup? I sent it couple of
> days ago.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12457.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
1. I still continue to experiment with the table created in IGFS. Currently
it works if query is executed only once, then either the ignite node fails
or error: Exception in thread "igfs-#60%null%"
java.lang.NullPointerException.
3. You're right! I didn't restart Visor. What's more, I run one Visor as
root (and forgot about it) and one as user ignite. After killing all of them
new topology starts from ver=1.
But it doesn't help for correct Ignite MR. Topology is correct, there is no
any error in logs, but the result is wrong. To be honest, I decided to stop
experiments with IgniteMR and focus on IGFS+TEZ or Spark.
beeline_ignite_mr.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12457/beeline_ignite_mr.log>
ignite-node-dn1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12457/ignite-node-dn1.log>
ignite-node-dn2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12457/ignite-node-dn2.log>
4. I think Evgeniy said about comparison TEZ vs TEZ+IGFS. So I join the
question:
did you conduct some tests Ignite + TEZ?
p.s. Ivan, did you get my email about Hadoop meetup? I sent it couple of
days ago.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12457.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Hi, Alena,
3. Looks like we have an answer why the initial topology version is so high:
you possibly do not restart the Visor process, is that true? If so, please
start next experiment with all nodes stopped, as well as the Visor process .
After that initial topology version should start with 1, it is not persisted
anywhere.
We should make sure each new started server joins successfully, with the 1st
attempt, and no "Node left topology" message ever appears. If it does, need
to investigate, why before further experiments.
4. Ignite MR also makes all intermediate operations in memory, so I don't
see any obvious reasons of why Ignite MR vs. TEZ comparison is senseless. I
suppose, the above results (23 sec on Tez vs. 80 sec on IgniteMR) can be
explained by the fact that Ignite was running in 1-node mode, while Tez was
using several (6?) nodes.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12430.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
Hi Ivan,
1. Need more time for experiments...
3. Yes, logs are full. I started every node with this command:
$IGNITE_HOME/bin/ignite.sh -v -J"-Xms10g -Xmx10g -XX:MaxMetaspaceSize=4g"
2>&1 | tee
/home/ignite/apache-ignite-hadoop-1.9.0-bin/work/log/ignite-node-dnX.log
I thought ver=72 because I did 72 attempts))
I don't know how to reset this counter. I stop ignite nodes Ctrl-C or Ctrl-Z
(then kill PID) or in Visor kill -k.
Look, there is no Ignite process, but now ver=109:
*[ignite@dev-dn1 ~]$ ps -ef | grep ignite*
ignite 6796 21325 0 15:04 pts/5 00:00:00 ps -ef
ignite 6797 21325 0 15:04 pts/5 00:00:00 grep ignite
root 21324 21290 0 May02 pts/5 00:00:00 su - ignite
ignite 21325 21324 0 May02 pts/5 00:00:00 -bash
root 27287 17525 0 May02 pts/1 00:00:00 su - ignite
ignite 27288 27287 0 May02 pts/1 00:00:00 -bash
*[ignite@dev-dn1 ~]$ $IGNITE_HOME/bin/ignite.sh -v -J"-Xms10g -Xmx10g
-XX:MaxMetaspaceSize=4g" 2>&1 | tee
/home/ignite/apache-ignite-hadoop-1.9.0-bin/work/log/ignite-node.log*
Ignite Command Line Startup, ver. 1.9.0#20170302-sha1:a8169d0a
2017 Copyright(C) Apache Software Foundation
[15:04:08,630][INFO ][main][IgniteKernal]
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 1.9.0#20170302-sha1:a8169d0a
>>> 2017 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
[15:04:08,630][INFO ][main][IgniteKernal] Config URL:
file:/home/ignite/apache-ignite-hadoop-1.9.0-bin/config/default-config.xml
[15:04:08,630][INFO ][main][IgniteKernal] Daemon mode: off
[15:04:08,631][INFO ][main][IgniteKernal] OS: Linux 2.6.32-696.el6.x86_64
amd64
[15:04:08,631][INFO ][main][IgniteKernal] OS user: ignite
[15:04:08,631][INFO ][main][IgniteKernal] PID: 6891
[15:04:08,631][INFO ][main][IgniteKernal] Language runtime: Java Platform
API Specification ver. 1.8
[15:04:08,631][INFO ][main][IgniteKernal] VM information: Java(TM) SE
Runtime Environment 1.8.0_101-b13 Oracle Corporation Java HotSpot(TM) 64-Bit
Server VM 25.101-b13
[15:04:08,633][INFO ][main][IgniteKernal] VM total memory: 9.6GB
[15:04:08,633][INFO ][main][IgniteKernal] Remote Management [restart: on,
REST: on, JMX (remote: on, port: 49199, auth: off, ssl: off)]
[15:04:08,633][INFO ][main][IgniteKernal]
IGNITE_HOME=/home/ignite/apache-ignite-hadoop-1.9.0-bin
[15:04:08,633][INFO ][main][IgniteKernal] VM arguments: [-Xms1g, -Xmx1g,
-XX:+AggressiveOpts, -XX:MaxMetaspaceSize=256m,
-Djava.library.path=/usr/hdp/current/hadoop-client/lib/native/,
-DIGNITE_QUIET=false,
-DIGNITE_SUCCESS_FILE=/home/ignite/apache-ignite-hadoop-1.9.0-bin/work/ignite_success_6a954010-244e-42bb-9cf7-b4fbbf39519a,
-Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49199,
-Dcom.sun.management.jmxremote.authenticate=false,
-Dcom.sun.management.jmxremote.ssl=false,
-DIGNITE_HOME=/home/ignite/apache-ignite-hadoop-1.9.0-bin,
-DIGNITE_PROG_NAME=/home/ignite/apache-ignite-hadoop-1.9.0-bin/bin/ignite.sh,
-Xms10g, -Xmx10g, -XX:MaxMetaspaceSize=4g]
[15:04:08,634][INFO ][main][IgniteKernal] Configured caches
['ignite-marshaller-sys-cache', 'ignite-sys-cache',
'ignite-hadoop-mr-sys-cache', 'ignite-atomics-sys-cache', 'igfs-meta',
'igfs-data']
[15:04:08,638][INFO ][main][IgniteKernal] 3-rd party licenses can be found
at: /home/ignite/apache-ignite-hadoop-1.9.0-bin/libs/licenses
[15:04:08,725][INFO ][main][IgnitePluginProcessor] Configured plugins:
[15:04:08,725][INFO ][main][IgnitePluginProcessor] ^-- None
[15:04:08,725][INFO ][main][IgnitePluginProcessor]
[15:04:08,786][INFO ][main][TcpCommunicationSpi] Successfully bound
communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0,
selectorsCnt=4, selectorSpins=0, pairedConn=false]
[15:04:08,790][WARN ][main][TcpCommunicationSpi] Message queue limit is set
to 0 which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[15:04:08,810][WARN ][main][NoopCheckpointSpi] Checkpoints are disabled (to
enable configure any GridCheckpointSpi implementation)
[15:04:08,842][WARN ][main][GridCollisionManager] Collision resolution is
disabled (all jobs will be activated upon arrival).
[15:04:08,846][WARN ][main][NoopSwapSpaceSpi] Swap space is disabled. To
enable use FileSwapSpaceSpi.
[15:04:08,847][INFO ][main][IgniteKernal] Security status
[authentication=off, tls/ssl=off]
[15:04:09,292][INFO ][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[15:04:09,718][INFO ][main][IpcServerTcpEndpoint] IPC server loopback
endpoint started [port=10500]
[15:04:09,720][INFO ][main][IpcServerTcpEndpoint] IPC server loopback
endpoint started [port=11400]
[15:04:09,729][INFO ][main][HadoopProcessor] HADOOP_HOME is set to
/usr/hdp/current/hadoop-client
[15:04:09,730][INFO ][main][HadoopProcessor] Resolved Hadoop classpath
locations: /usr/hdp/2.6.0.3-8/hadoop, /usr/hdp/current/hadoop-hdfs-client/,
/usr/hdp/current/hadoop-mapreduce-client/
[15:04:09,756][INFO ][main][IgniteKernal] Non-loopback local IPs:
10.255.242.96, fe80:0:0:0:21c:c4ff:fe5b:663c%bond0
[15:04:09,756][INFO ][main][IgniteKernal] Enabled local MACs: 001CC45B663C
[15:04:09,802][INFO ][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=65d7c352-5a5c-44c2-a4ac-852e98abb46a]
[15:04:10,387][INFO ][main][GridCacheProcessor] Started cache
[name=igfs-data, mode=PARTITIONED]
[15:04:10,406][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-hadoop-mr-sys-cache, mode=REPLICATED]
[15:04:10,408][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-sys-cache, mode=REPLICATED]
[15:04:10,410][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-atomics-sys-cache, mode=PARTITIONED]
[15:04:10,414][INFO ][main][GridCacheProcessor] Started cache
[name=igfs-meta, mode=REPLICATED]
[15:04:10,438][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-marshaller-sys-cache, mode=REPLICATED]
[15:04:10,582][INFO
][exchange-worker-#33%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=109,
minorTopVer=0], evt=NODE_JOINED, node=65d7c352-5a5c-44c2-a4ac-852e98abb46a]
[15:04:10,667][INFO ][main][IgniteKernal] Performance suggestions for grid
(fix if possible)
[15:04:10,667][INFO ][main][IgniteKernal] To disable, set
-DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Enable G1 Garbage Collector
(add '-XX:+UseG1GC' to JVM options)
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Set max direct memory size
if getting 'OOME: Direct buffer memory' (add
'-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Disable processing of calls
to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Speed up flushing of dirty
pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500)
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Reduce pages swapping ratio
(set vm.swappiness=10)
[15:04:10,668][INFO ][main][IgniteKernal] ^-- Avoid direct reclaim and
page allocation failures (set vm.extra_free_kbytes=1240000)
[15:04:10,668][INFO ][main][IgniteKernal] Refer to this page for more
performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[15:04:10,668][INFO ][main][IgniteKernal]
[15:04:10,668][INFO ][main][IgniteKernal] To start Console Management &
Monitoring run ignitevisorcmd.{sh|bat}
[15:04:10,669][INFO ][main][IgniteKernal]
[15:04:10,669][INFO ][main][IgniteKernal]
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 1.9.0#20170302-sha1:a8169d0ac448b8f19b8cdc51ee6f339ee6864c29
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 2.6.32-696.el6.x86_64 amd64
>>> CPU(s): 8
>>> Heap: 10.0GB
>>> VM name: 6891@dev-dn1.co.vectis.local
>>> Local node [ID=65D7C352-5A5C-44C2-A4AC-852E98ABB46A, order=109,
>>> clientMode=false]
>>> Local node addresses: [dev-dn1.co.vectis.local/0:0:0:0:0:0:0:1%lo,
>>> /10.255.242.96, /127.0.0.1]
>>> Local ports: TCP:10500 TCP:11211 TCP:11400 TCP:47100 TCP:47500
[15:04:10,671][INFO ][main][GridDiscoveryManager] Topology snapshot
[*ver=109*, servers=1, clients=0, CPUs=8, heap=10.0GB]
[15:04:20,047][INFO ][ignite-update-notifier-timer][GridUpdateNotifier] Your
version is up to date.
[15:05:10,669][INFO ][grid-timeout-worker-#15%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=65d7c352, name=null, uptime=00:01:00:009]
^-- H/N/C [hosts=1, nodes=1, CPUs=8]
^-- CPU [cur=0.1%, avg=0.64%, GC=0%]
^-- Heap [used=631MB, free=93.57%, comm=9813MB]
^-- Non heap [used=49MB, free=99.08%, comm=50MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
4. Evgeniy said (I found it in email):
Alena, i think you can`t obtain any significant results here, look tez
takes all data into memory and runs DAGs with no flush to disk, using IGFS
here you simply append additional memory burden here. TEZ standalone is
good enough.
Ivan, does Ignite have some tests in case of TEZ usage ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12419.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Alena, I suppose, incorrect results in your environment may be a consequence
of topology troubles. In any way, to have some stable and reproducible
results you need to have stable Ignite cluster topology. To achieve that I
would recommend the following steps:
1) kill all the Ignite processes on all the nodes (you may see them with "ps
-ef | grep ignite" in Unix shell).
2) start 1st Ignite node (preferably with "-v" option, and with a dedicated
console, redirecting the output to a file: "./ignite.sh -v ... |& tee
mylogfile " ) -- find first "Topology snapshot" line in the log. It should
say "Topology snapshot [ver=1, servers=1, clients=0, CPUs=..." . If topology
version is different from 1, that means something is wrong, possibly there
is a stale Ignite process this one attempts to join.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12412.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
WRT item 2. : cannot reproduce the issue yet. Each time I get correct data:
OK
2017-03-15 36564815
2017-03-16 36872463
2017-03-17 36900812
2017-03-18 36904198
2017-03-19 36999930
2017-03-20 37029921
Time taken: 69.603 seconds, Fetched: 6 row(s)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12398.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Hi, Alena,
1. E.g. can you explicitly specify igfs:// as the table data location, like
create table ...... stored as orc location 'igfs://...../path/test_ignite';
?
2. Ok, thanks, will try to reproduce this using the provided data.
3. Here is something very strange. Are these logs full and do they reflect
cluster startup from the beginning (from the state when no node is running)
? For example, it is unclear, why the topology version is 72 at the moment
of the 1st node start:
[13:11:10,358][INFO ][main][GridDiscoveryManager] Topology snapshot
[*ver=72*, servers=1, clients=0, CPUs=8, heap=10.0GB]
4. Can you please specify more exactly, what Evgeniy's comment you're
referring to?
Regards,
Ivan Veselovsky.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12394.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Alena Melnikova <al...@74.ru>.
1. How can I explicitly load the Hive table into the IFGS don't using Java
API? (I don't know Java)
I use DUAL_SYNC. Here is my config.
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/default-config.xml>
2. I attache sample data (test_ignite.rar). These are ORC files for Hive
partitioned table.
create table test_ignite (column1 double) partitioned by (calday string)
stored as orc location '/path/test_ignite';
alter table test_ignite add partition (calday='2017-03-15');
alter table test_ignite add partition (calday='2017-03-16');
alter table test_ignite add partition (calday='2017-03-17');
alter table test_ignite add partition (calday='2017-03-18');
alter table test_ignite add partition (calday='2017-03-19');
alter table test_ignite add partition (calday='2017-03-20');
select calday, count(*) from test_ignite where calday between '2017-03-15'
and '2017-03-20' group by calday order by calday;
Correct result on one ignite node:
+-------------+-----------+--+
| calday | _c1 |
+-------------+-----------+--+
| 2017-03-15 | 36564815 |
| 2017-03-16 | 36872463 |
| 2017-03-17 | 36900812 |
| 2017-03-18 | 36904198 |
| 2017-03-19 | 36999930 |
| 2017-03-20 | 37029921 |
+-------------+-----------+--+
6 rows selected (49.88 seconds)
Wrong result on two ignite nodes:
+-------------+-----------+--+
| calday | _c1 |
+-------------+-----------+--+
| 2017-03-16 | 24582164 |
| 2017-03-17 | 12301380 |
| 2017-03-18 | 36904198 |
| 2017-03-19 | 12332322 |
+-------------+-----------+--+
4 rows selected (45.199 seconds)
test_ignite.rar
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/test_ignite.rar>
3. I started ignite nodes sequentially on 6 servers (dn1, dn2, dn3, dn4,
dn5, dn6). They formed 3 clusters:
dn1-dn3-dn6
dn2-dn4
dn5
ignite-node-dn1.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn1.log>
ignite-node-dn2.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn2.log>
ignite-node-dn3.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn3.log>
ignite-node-dn4.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn4.log>
ignite-node-dn5.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn5.log>
ignite-node-dn6.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n12393/ignite-node-dn6.log>
4. As regard Evgeniy's comment it sounds reasonable, but I'm trying to cache
some hot Hive tables so that different users run their queries faster
because they don't need to read the same data from the disk. Still hope this
is possible)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12393.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDP, Hive + Ignite
Posted by Ivan Veselovsky <iv...@gridgain.com>.
1. Please make sure IGFS is really used: e.g. you may explicitly locate some
table data on IGFS, and run the queries upon. IGFS statistics can partially
be observed through Visor.
Also please note, that upon node(s) start IGFS is empty. In case of dual
modes it caches the data upon file reading. In case of primary mode you need
to put some data onto the file system before you can use it. So, data read
performance boost can be seen only when some data are already cached in
IGFS, and read from there rather than from disk.
2. Can you specify data and the query , so that we could reproduce the
issue? (E.g. you can use some publicly available sample data from Hive
examples.)
3. No. The nodes should connect without additional effort. Can you please
attach full logs of all nodes where this situation happens?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12355.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.