You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by BinaryTree <bi...@foxmail.com> on 2019/04/10 07:14:53 UTC
Memory leak.Ignite runs slower and slower after a period of time.
When my Ignite clients run for a while, it becomes slower and slower, and the outputs can be seen in our gc logs:
2019-04-10T06:42:47.885+0000: 62271.788: [Full GC (Ergonomics) [PSYoungGen: 1494016K->1494005K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591012K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 9.9864029 secs] [Times: user=19.85 sys=0.00, real=9.98 secs]
2019-04-10T06:42:57.874+0000: 62281.777: [Full GC (Ergonomics) [PSYoungGen: 1494015K->1494012K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591019K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 9.9982344 secs] [Times: user=19.87 sys=0.00, real=9.99 secs]
2019-04-10T06:43:07.874+0000: 62291.778: [Full GC (Ergonomics) [PSYoungGen: 1494016K->1494014K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591020K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 10.0803891 secs] [Times: user=19.93 sys=0.00, real=10.08 secs]
According to the outputs, I am sure that some objects have not been recycled. So I dumped the heap and analyzed them in the Eclipse Memory Analyzer, here is the reports that were given by the tool.
From the above picture, I guess there may be some bugs or inappropriate usage caused GridReduceQueryExecutor is not being recycled, but I don't know what the specific reason is.So I hope that you can give me some advice.
The code segments show how I execute a query:
public List<DpCache> query(String key, String value) {
List<DpCache> list = Lists.newArrayList();
String fields = "id, gmtCreate, gmtModified, devId, dpId, code, name, customName, mode, type, value, rawValue, time, status, uuid";
String sql = "select " + fields + " from " + IgniteTableKey.T_DATA_POINT_NEW + " where " + key + "='" + value +"'";
FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(new SqlFieldsQuery(sql));
for (List<?> objects : cursor) {
DpCache cache = convertToDpCache(objects);
list.add(cache);
}
return list;
}public DpCache queryOne(String devId, Integer dpId) {
DpCache cache = null;
String fields = "id, gmtCreate, gmtModified, devId, dpId, code, name, customName, mode, type, value, rawValue, time, status, uuid";
String sql = "select " + fields + " from " + IgniteTableKey.T_DATA_POINT_NEW + " where devId=? and dpId=?";
SqlFieldsQuery query = new SqlFieldsQuery(sql);
query.setArgs(devId, dpId);
FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(query);
Iterator<List<?>> iterator = cursor.iterator();
if (iterator.hasNext()) {
cache = convertToDpCache(iterator.next());
}
turn cache;
}public boolean hasRecord(String devId, Integer dpId) {
boolean hasRecord;
String sql = "select 1 from t_data_point_new where devId=? and dpId=?";
SqlFieldsQuery query = new SqlFieldsQuery(sql);
query.setArgs(devId, dpId);
FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(query);
Iterator<List<?>> iterator = cursor.iterator();
hasRecord = iterator.hasNext();
return hasRecord;
}public void invokeAllAsync(Map<DpKey, BinaryObject> map) {
Map<DpKey, DataPointEntryProcessor> processorMap = Maps.newHashMap();
for (Map.Entry<DpKey, BinaryObject> entry : map.entrySet()) {
processorMap.put(entry.getKey(), new DataPointEntryProcessor(entry.getValue()));
}
newIgniteCache.invokeAllAsync(processorMap);
}
Anyone who can give me advice will be appreciated.
Looking forward to your reply.
Re: Memory leak.Ignite runs slower and slower after a period of
time.
Posted by Justin Ji <bi...@foxmail.com>.
One more question.
According to the org.apache.ignite.cache.query.QueryCursor.getAll API, I
know that the resources will be closed automatically since all results are
fetched.
I would like to know will resources be closed after I iterate over the
Cursor？
The reason why I ask this question is that in my test case I can not
reproduce this issue if I iterate over all the records returned by SQL
query.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Memory leak.Ignite runs slower and slower after a period of time.
Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hi,
> Does it mean the cursor still not closed in Ignite-2.6.0?
Yes, it seems so.
Thanks,
S.
чт, 11 апр. 2019 г. в 15:06, Justin Ji <bi...@foxmail.com>:
> Emmm, Does it mean the cursor still not closed in Ignite-2.6.0?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Memory leak.Ignite runs slower and slower after a period of
time.
Posted by Justin Ji <bi...@foxmail.com>.
Emmm, Does it mean the cursor still not closed in Ignite-2.6.0?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Memory leak.Ignite runs slower and slower after a period of time.
Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hi,
> BTW, I can not find the class you provided(Ignite-2.6.0), do you spell it
correctly?
Oops :) That class was added recently. Please take a look at
https://issues.apache.org/jira/browse/IGNITE-10827
The fix is already available on the master branch
Thanks,
S.
чт, 11 апр. 2019 г. в 14:10, Justin Ji <bi...@foxmail.com>:
> Slava -
>
> Thank you again for your reply, I will close the cursor explicitly.
>
> BTW, I can not find the class you provided(Ignite-2.6.0), do you spell it
> correctly?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Memory leak.Ignite runs slower and slower after a period of
time.
Posted by Justin Ji <bi...@foxmail.com>.
Slava -
Thank you again for your reply, I will close the cursor explicitly.
BTW, I can not find the class you provided(Ignite-2.6.0), do you spell it
correctly?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Memory leak.Ignite runs slower and slower after a period of time.
Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hello,
> I would like to know will resource be closed after I iterate over the
Cursor?
Well, the answer is yes, the cursor will be closed (the underlying iterator
is wrapped by
org.apache.ignite.internal.processors.cache.AutoClosableCursorIterator).
Please take into account that this behavior is implementation details and
it can be changed.
So, I think the best choice would be to explicitly close the cursor using
close()/try-with-resources statement.
Thanks,
S.
чт, 11 апр. 2019 г. в 11:55, Justin Ji <bi...@foxmail.com>:
> One more question.
>
> According to the org.apache.ignite.cache.query.QueryCursor.getAll API, I
> know that the resources will be closed automatically since all results are
> fetched.
>
>
> I would like to know will resources be closed after I iterate over the
> Cursor？
>
> The reason why I ask this question is that in my test case I can not
> reproduce this issue if I iterate over all the records returned by SQL
> query.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Memory leak.Ignite runs slower and slower after a period of
time.
Posted by Justin Ji <bi...@foxmail.com>.
One more question.
According to the org.apache.ignite.cache.query.QueryCursor.getAll API, I
know that the resources will be closed automatically since all results are
fetched.
I would like to know will resources be closed after I iterate over the
Cursor？
The reason why I ask this question is that in my test case I can not
reproduce this issue if I iterate over all the records returned by SQL
query.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Memory leak.Ignite runs slower and slower after a period of
time.
Posted by Justin Ji <bi...@foxmail.com>.
Slava -
Thank for your reply, I will confirm it.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Memory leak.Ignite runs slower and slower after a period of time.
Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hello,
At first glance, your code does not close FieldsQueryCursor instances.
Could you explicitly close the cursors via close() or use
try-with-resources statement.
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/FieldsQueryCursor.html
[2]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/QueryCursor.html#getAll--
Thanks,
S.
ср, 10 апр. 2019 г. в 10:15, BinaryTree <bi...@foxmail.com>:
> When my Ignite clients run for a while, it becomes slower and slower, and
> the outputs can be seen in our gc logs:
>
> 2019-04-10T06:42:47.885+0000: 62271.788: [Full GC (Ergonomics) [PSYoungGen: 1494016K->1494005K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591012K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 9.9864029 secs] [Times: user=19.85 sys=0.00, real=9.98 secs]
> 2019-04-10T06:42:57.874+0000: 62281.777: [Full GC (Ergonomics) [PSYoungGen: 1494015K->1494012K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591019K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 9.9982344 secs] [Times: user=19.87 sys=0.00, real=9.99 secs]
> 2019-04-10T06:43:07.874+0000: 62291.778: [Full GC (Ergonomics) [PSYoungGen: 1494016K->1494014K(1797120K)] [ParOldGen: 2097006K->2097006K(2097152K)] 3591022K->3591020K(3894272K), [Metaspace: 103757K->103757K(1144832K)], 10.0803891 secs] [Times: user=19.93 sys=0.00, real=10.08 secs]
>
> According to the outputs, I am sure that some objects have not been
> recycled. So I dumped the heap and analyzed them in the Eclipse Memory
> Analyzer, here is the reports that were given by the tool.
>
> *From the above picture, I guess there may be some bugs or inappropriate
> usage caused GridReduceQueryExecutor is not being recycled, but I don't
> know what the specific reason is.So I hope that you can give me some
> advice.*
>
> The code segments show how I execute a query:
>
> public List<DpCache> query(String key, String value) {
> List<DpCache> list = Lists.newArrayList();
> String fields = "id, gmtCreate, gmtModified, devId, dpId, code, name, customName, mode, type, value, rawValue, time, status, uuid";
> String sql = "select " + fields + " from " + IgniteTableKey.T_DATA_POINT_NEW + " where " + key + "='" + value +"'";
> FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(new SqlFieldsQuery(sql));
> for (List<?> objects : cursor) {
> DpCache cache = convertToDpCache(objects);
> list.add(cache);
> }
> return list;
> }
>
> public DpCache queryOne(String devId, Integer dpId) {
> DpCache cache = null;
> String fields = "id, gmtCreate, gmtModified, devId, dpId, code, name, customName, mode, type, value, rawValue, time, status, uuid";
> String sql = "select " + fields + " from " + IgniteTableKey.T_DATA_POINT_NEW + " where devId=? and dpId=?";
>
> SqlFieldsQuery query = new SqlFieldsQuery(sql);
> query.setArgs(devId, dpId);
> FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(query);
> Iterator<List<?>> iterator = cursor.iterator();
> if (iterator.hasNext()) {
> cache = convertToDpCache(iterator.next());
> }
> turn cache;
> }
>
> public boolean hasRecord(String devId, Integer dpId) {
> boolean hasRecord;
> String sql = "select 1 from t_data_point_new where devId=? and dpId=?";
> SqlFieldsQuery query = new SqlFieldsQuery(sql);
> query.setArgs(devId, dpId);
>
> FieldsQueryCursor<List<?>> cursor = newIgniteCache.query(query);
>
> Iterator<List<?>> iterator = cursor.iterator();
> hasRecord = iterator.hasNext();
> return hasRecord;
> }
>
> public void invokeAllAsync(Map<DpKey, BinaryObject> map) {
> Map<DpKey, DataPointEntryProcessor> processorMap = Maps.newHashMap();
> for (Map.Entry<DpKey, BinaryObject> entry : map.entrySet()) {
> processorMap.put(entry.getKey(), new DataPointEntryProcessor(entry.getValue()));
> }
> newIgniteCache.invokeAllAsync(processorMap);
> }
>
> Anyone who can give me advice will be appreciated.
>
> Looking forward to your reply.
>