You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@usergrid.apache.org by "Michael Russo (JIRA)" <ji...@apache.org> on 2016/09/07 15:51:20 UTC
[jira] [Commented] (USERGRID-1259) Re-indexing ElasticSearch entity
data from Cassandra - Possible Memory Leaks in Usergrid
[ https://issues.apache.org/jira/browse/USERGRID-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470977#comment-15470977 ]
Michael Russo commented on USERGRID-1259:
-----------------------------------------
In the latest of master branch ( which will eventually become a 2.2.0 release, we've made many memory improvements). We never found any problems specific to re-index and had tested with ~10 million + entities. Regular Usergrid running was having memory pressure in time, and I think re-index in your case accelerated the problem.
> Re-indexing ElasticSearch entity data from Cassandra - Possible Memory Leaks in Usergrid
> ----------------------------------------------------------------------------------------
>
> Key: USERGRID-1259
> URL: https://issues.apache.org/jira/browse/USERGRID-1259
> Project: Usergrid
> Issue Type: Story
> Affects Versions: 2.1.0
> Reporter: Jaskaran
> Fix For: 2.2.0
>
>
> Full system re-index job (http://localhost:8080/system/index/rebuild), seems to stop / hang after 20-30 hours of indexing. Usergrid seems to exhaust the 4.5 GB of RAM. Please see logs below:
> 1. UserGrid logs (out of java heap space)
> Feb 04 17:06:32 Usergrid-2 catalina.out: 06:36:31,961 WARN OioServerSocketPipelineSink:83 - Failed to accept a connection.
> Feb 04 17:06:32 Usergrid-2 catalina.out: java.lang.OutOfMemoryError: Java heap space
> Feb 04 15:05:03 Usergrid-2 catalina.out: 04:34:25,166 WARN jvm:203 - [default] [gc][old][29454][2683] duration [54.2s], collections [3]/[54.3s], total [54.2s]/[13.2h], memory [4.4gb]->[4.4gb]/[4.4gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [62.5mb]->[65mb]/[66.5mb]}{[old] [3.8gb]->[3.8gb]/[3.8gb]}
> Feb 03 20:38:34 Usergrid-2 catalina.out: 10:08:34,616 ERROR AmazonAsyncEventService:361 - Failed to index message: 886ea9bd-708d-4bea-ab1c-844ff97c947c
> 2. ES logs (time out, removed non-data node from cluster)
> Feb 04 16:00:33 Elasticsearch elasticsearch.log: [2016-02-04 05:31:17,243][INFO ][cluster.service ] [Arlette Truffaut] removed {[default][3GuynlamR6GvdiCAhhTBmw][ip-10-0-0-237][inet[/10.0.0.237:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([default][3GuynlamR6GvdiCAhhTBmw][ip-10-0-0-237][inet[/10.0.0.237:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
> Feb 04 16:00:33 Elasticsearch elasticsearch.log: [2016-02-04 05:31:17,256][DEBUG][action.admin.cluster.node.stats] [Arlette Truffaut] failed to execute on node [3GuynlamR6GvdiCAhhTBmw]
> Feb 04 16:00:33 Elasticsearch elasticsearch.log: org.elasticsearch.transport.NodeDisconnectedException: [default][inet[/10.0.0.237:9301]][cluster:monitor/nodes/stats[n]] disconnected
> Is it possible that the re-indexing code in Usergrid could have memory leaks and thus uses up all the java heap memory.
> Please help.
> Thanks
> Jaskaran
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)