You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Tobias Eriksson <to...@qvantel.com> on 2017/10/22 20:58:47 UTC
Hammering Cassandra with Ignite seems to get ignite into an infinite
loop
Hi
I am testing to see if Ignite v2.1 with Cassandra v3.11 is the way to go for us
I have a AWS cluster of 4 nodes,
2 Nodes with Ignite (node B and C)
1 Node with Cassandra (node B)
4 Nodes which act as Client-Ignite nodes updating/inserting values into a simple cache key,value (serialized java class) (node A,B,C and D)
As I was adding client-apps 1,2,3 and eventually 4 clients I noticed the exception below for Ignite node on ( C )
It seems to have run into a bad loop, that it does not get out of
Is this a known bug ?
-Tobias
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=ee941d86, name=null, uptime=00:12:00:054]
^-- H/N/C [hosts=4, nodes=5, CPUs=64]
^-- CPU [cur=12.33%, avg=11.77%, GC=0.03%]
^-- PageMemory [pages=2518090]
^-- Heap [used=365MB, free=63.92%, comm=1013MB]
^-- Non heap [used=77MB, free=94.9%, comm=79MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
2017-10-20 16:57:22 INFO IgniteKernal:475 - FreeList [name=null, buckets=256, dataPages=1666280, reusePages=817]
2017-10-20 16:57:23 INFO GridDiscoveryManager:475 - Node left topology: TcpDiscoveryNode [id=77067a17-7be8-4382-9741-fbfff82a7d84, addrs=[0:0:0:0:0:0:0:1%lo, 10.150.4.223, 127.0.0.1, 172.17.0.1, 172.18.0.1, 172.19.0.1, 172.20.0.1, 172.21.0.1], sockAddrs=[/172.17.0.1:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, ip-172-19-0-1.eu-west-1.compute.internal/172.19.0.1:0, ip-172-18-0-1.eu-west-1.compute.internal/172.18.0.1:0, ip-172-21-0-1.eu-west-1.compute.internal/172.21.0.1:0, ip-172-20-0-1.eu-west-1.compute.internal/172.20.0.1:0, /10.150.4.223:0], discPort=0, order=5, intOrder=5, lastExchangeTime=1508507191082, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true]
2017-10-20 16:57:23 INFO GridDiscoveryManager:475 - Topology snapshot [ver=8, servers=2, clients=2, CPUs=64, heap=15.0GB]
2017-10-20 16:57:23 INFO time:475 - Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false, evt=11, node=TcpDiscoveryNode [id=ee941d86-d0ae-4b6d-a484-2f249c2caa62, addrs=[0:0:0:0:0:0:0:1%lo, 10.150.4.224, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, ip-10-150-4-224.eu-west-1.compute.internal/10.150.4.224:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1508507843554, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=false], evtNode=TcpDiscoveryNode [id=ee941d86-d0ae-4b6d-a484-2f249c2caa62, addrs=[0:0:0:0:0:0:0:1%lo, 10.150.4.224, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, ip-10-150-4-224.eu-west-1.compute.internal/10.150.4.224:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1508507843554, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=false], customEvt=null]
2017-10-20 16:57:23 INFO GridDhtPartitionsExchangeFuture:475 - Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], time=0ms]
2017-10-20 16:57:23 INFO time:475 - Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false]
2017-10-20 16:57:23 INFO GridCachePartitionExchangeManager:475 - Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_LEFT, node=77067a17-7be8-4382-9741-fbfff82a7d84]
2017-10-20 16:57:25 INFO ClockFactory:52 - Using native clock to generate timestamps.
2017-10-20 16:57:25 INFO DCAwareRoundRobinPolicy:95 - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2017-10-20 16:57:25 INFO Cluster:1568 - New Cassandra host /10.150.4.223:9042 added
2017-10-20 16:57:25 WARN CassandraCacheStore:485 - Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0x76fccec193ce5098fb3094cfdb082930. You may have used a PreparedStatement that was created with another Cluster instance.
at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:578)
at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:231)
at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.writeAll(CassandraCacheStore.java:354)
at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.putAll(GridCacheStoreManagerAdapter.java:625)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updatePartialBatch(GridDhtAtomicCache.java:2643)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateWithBatch(GridDhtAtomicCache.java:2296)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1686)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3063)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:745)
2017-10-20 16:57:27 INFO ClockFactory:52 - Using native clock to generate timestamps.
2017-10-20 16:57:27 INFO DCAwareRoundRobinPolicy:95 - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2017-10-20 16:57:27 INFO Cluster:1568 - New Cassandra host /10.150.4.223:9042 added
2017-10-20 16:57:27 WARN CassandraCacheStore:480 - Cassandra session refreshed
2017-10-20 16:57:27 WARN CassandraCacheStore:485 - Prepared statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0x76fccec193ce5098fb3094cfdb082930. You may have used a PreparedStatement that was created with another Cluster instance.
at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:578)
at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:231)
at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.writeAll(CassandraCacheStore.java:354)
at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.putAll(GridCacheStoreManagerAdapter.java:625)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updatePartialBatch(GridDhtAtomicCache.java:2643)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateWithBatch(GridDhtAtomicCache.java:2296)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1686)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3063)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:745)
2017-10-20 16:57:27 WARN CassandraCacheStore:485 - Prepared statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0x76fccec193ce5098fb3094cfdb082930. You may have used a PreparedStatement that was created with another Cluster instance.
--
Tobias Eriksson
Chief Architect Research – CTO Office
Qvantel Sweden AB
Tel; +46 768 832453
e-mail; Tobias.Eriksson@qvantel.com<ma...@qvantel.com>
Re: Hammering Cassandra with Ignite seems to get ignite into an
infinite loop
Posted by Andrey Mashenkov <an...@gmail.com>.
Tobias,
Looks it is a duplicate. I've answered in next tread:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-cassandra-module-issue-td13808.html
You can try to build Ignite from master [1] or ignite-2.3 [2] branch and
check against it and let us know if the issue will gone or not.
[1] https://github.com/apache/ignite/tree/master
[2] https://github.com/apache/ignite/tree/ignite-2.3
On Mon, Oct 23, 2017 at 2:19 PM, slava.koptilin <sl...@gmail.com>
wrote:
> Hi Tobias,
>
> As far as I know, Cassandra module was tested with Cassandra Database 3.3 &
> Cassandra Driver 3.0.0.
> Perhaps, this issue related to Cassandra v3.11. Could you please check that
> the issue is reproducible with Cassandra Database 3.3 & Cassandra Driver
> 3.0.0?
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Hammering Cassandra with Ignite seems to get ignite into an
infinite loop
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Tobias,
As far as I know, Cassandra module was tested with Cassandra Database 3.3 &
Cassandra Driver 3.0.0.
Perhaps, this issue related to Cassandra v3.11. Could you please check that
the issue is reproducible with Cassandra Database 3.3 & Cassandra Driver
3.0.0?
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/