You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Yurii Kartsev (JIRA)" <ji...@apache.org> on 2016/03/16 23:45:33 UTC
[jira] [Commented] (AMQ-6216) Slave can't become master due to
infinite "Store update waiting on 1 replica(s) to catch up to log position
XXX" in replicated LevelDB Store mode.
[ https://issues.apache.org/jira/browse/AMQ-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198348#comment-15198348 ]
Yurii Kartsev commented on AMQ-6216:
------------------------------------
Update: after I simply restart the node which I manually shut down, infinite log from above becomes like this:{code}leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
2016-03-16 18:37:59,471 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
2016-03-16 18:38:00,472 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
2016-03-16 18:38:00,724 | INFO | Slave has connected: 16d6d243-80b5-47ea-b46e-45e6e13ae18f | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | hawtdispatch-DEFAULT-2
2016-03-16 18:38:00,841 | INFO | Slave has now caught up: 16d6d243-80b5-47ea-b46e-45e6e13ae18f | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | hawtdispatch-DEFAULT-2
2016-03-16 18:38:00,985 | INFO | Apache ActiveMQ 5.13.2 (localhost, ID:LM-CN64-34633-1458167880858-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2016-03-16 18:38:01,633 | INFO | Listening for connections at: tcp://LM-CN64:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2016-03-16 18:38:01,634 | INFO | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2016-03-16 18:38:01,636 | INFO | Apache ActiveMQ 5.13.2 (localhost, ID:LM-CN64-34633-1458167880858-0:1) started | org.apache.activemq.broker.BrokerService | main
2016-03-16 18:38:01,636 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2016-03-16 18:38:01,637 | WARN | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /home/yury/Documents/levelDB only has 50505 mb of usable space. - resetting to maximum available disk space: 50505 mb | org.apache.activemq.broker.BrokerService | main
2016-03-16 18:38:01,639 | WARN | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /home/yury/Documents/apache-activemq-5.13.2/data only has 50505 mb of usable space. - resetting to maximum available disk space: 50505 mb | org.apache.activemq.broker.BrokerService | main
2016-03-16 18:38:02,204 | INFO | No Spring WebApplicationInitializer types detected on classpath | /admin | main
2016-03-16 18:38:02,364 | INFO | ActiveMQ WebConsole available at http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2016-03-16 18:38:02,364 | INFO | ActiveMQ Jolokia REST API available at http://0.0.0.0:8161/api/jolokia/ | org.apache.activemq.web.WebConsoleStarter | main
2016-03-16 18:38:02,397 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2016-03-16 18:38:02,772 | INFO | No Spring WebApplicationInitializer types detected on classpath | /api | main
2016-03-16 18:38:02,818 | INFO | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | main{code}
Plus my client which was sending lot of messages to the queue is ab;e to reconnect (because it's using "failover" protocol).
I assume this is related to intense data input to the master node which can't get copied to slaves in time due to master shut down. But I think this should be robust enough to survive not only manual master shutdowns, but also immediate master loss like during network failure. But I maybe just doing something wrong here... Could you please advise?
> Slave can't become master due to infinite "Store update waiting on 1 replica(s) to catch up to log position XXX" in replicated LevelDB Store mode.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-6216
> URL: https://issues.apache.org/jira/browse/AMQ-6216
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.13.2
> Reporter: Yurii Kartsev
>
> Good afternoon dear ActiveMQ support.
> I wanted to test newest ActiveMQ version for stability in Replicated LevelDB Store mode in order to know whether we can upgrade our production copy (since it has similar issues as well), but facing an issue. I am able to reproduce it, so it's not sporadic. It happens when under load I manually shut down master node and new master can not be selected having infinite logging as follows (log taken from initial slave which is promoted to master later):{code}yury@LM-CN64 ~/Documents/apache-activemq-5.13.2/bin $ tail -f ../data/activemq.log
> 2016-03-16 18:07:34,026 | INFO | Client environment:os.version=3.13.0-37-generic | org.apache.zookeeper.ZooKeeper | main
> 2016-03-16 18:07:34,027 | INFO | Client environment:user.name=yury | org.apache.zookeeper.ZooKeeper | main
> 2016-03-16 18:07:34,028 | INFO | Client environment:user.home=/home/yury | org.apache.zookeeper.ZooKeeper | main
> 2016-03-16 18:07:34,031 | INFO | Client environment:user.dir=/home/yury/Documents/apache-activemq-5.13.2/bin | org.apache.zookeeper.ZooKeeper | main
> 2016-03-16 18:07:34,033 | INFO | Initiating client connection, connectString=10.100.211.233:2181 sessionTimeout=2000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4770c18a | org.apache.zookeeper.ZooKeeper | main
> 2016-03-16 18:07:34,049 | WARN | SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/home/yury/Documents/apache-activemq-5.13.2//conf/login.config'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(10.100.211.233:2181)
> 2016-03-16 18:07:34,052 | INFO | Opening socket connection to server 10.100.211.233/10.100.211.233:2181 | org.apache.zookeeper.ClientCnxn | main-SendThread(10.100.211.233:2181)
> 2016-03-16 18:07:34,052 | WARN | unprocessed event state: AuthFailed | org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
> 2016-03-16 18:07:34,056 | INFO | Socket connection established to 10.100.211.233/10.100.211.233:2181, initiating session | org.apache.zookeeper.ClientCnxn | main-SendThread(10.100.211.233:2181)
> 2016-03-16 18:07:34,066 | INFO | Session establishment complete on server 10.100.211.233/10.100.211.233:2181, sessionid = 0x1537b47772c0007, negotiated timeout = 4000 | org.apache.zookeeper.ClientCnxn | main-SendThread(10.100.211.233:2181)
> 2016-03-16 18:07:34,682 | INFO | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[localhost] Task-1
> 2016-03-16 18:07:34,694 | INFO | Attaching to master: tcp://10.100.211.233:61619 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | ActiveMQ BrokerService[localhost] Task-1
> 2016-03-16 18:07:34,703 | INFO | Slave started | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ BrokerService[localhost] Task-1
> 2016-03-16 18:07:34,800 | INFO | Slave skipping download of: log/0000000000000000.log | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,812 | INFO | Slave requested: 00000000006399f2.index/CURRENT | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,815 | INFO | Slave requested: 00000000006399f2.index/000003.log | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,816 | INFO | Slave requested: 00000000006399f2.index/MANIFEST-000002 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,826 | INFO | Attaching... Downloaded 0.02/28.37 kb and 1/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,827 | INFO | Attaching... Downloaded 28.32/28.37 kb and 2/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,829 | INFO | Attaching... Downloaded 28.37/28.37 kb and 3/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:07:34,830 | INFO | Attached | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
> 2016-03-16 18:15:11,459 | INFO | Not enough cluster members have reported their update positions yet. | org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> 2016-03-16 18:15:11,527 | INFO | Slave stopped | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:11,529 | INFO | Not enough cluster members have reported their update positions yet. | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:11,534 | INFO | Not enough cluster members have reported their update positions yet. | org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> 2016-03-16 18:15:11,594 | INFO | Promoted to master | org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> 2016-03-16 18:15:11,616 | INFO | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:12,066 | INFO | Master started: tcp://10.211.55.5:61619 | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ BrokerService[localhost] Task-3
> 2016-03-16 18:15:13,069 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:14,071 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:15,072 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:16,074 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:17,076 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:18,078 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:19,080 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:20,081 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:21,083 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:22,085 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:23,087 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:24,089 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:25,091 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:26,092 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:27,095 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:28,097 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:29,098 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:30,100 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:31,101 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:32,102 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:33,103 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:34,104 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:35,106 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:36,108 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:37,110 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:38,112 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:39,113 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:40,115 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:41,117 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:42,119 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:43,120 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:44,121 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:45,122 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:46,123 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:47,124 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:48,125 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:49,126 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:50,128 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:51,129 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:52,130 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:53,131 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:54,132 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:55,133 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:56,134 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:57,136 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:58,137 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:15:59,139 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:16:00,140 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:16:01,142 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:16:02,144 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:16:03,145 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> 2016-03-16 18:16:04,146 | WARN | Store update waiting on 1 replica(s) to catch up to log position 12074259. | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-2
> ...{code}
> Steps to reproduce:
> 1. Start one ZK node. Set up ActiveMQ cluster in Replicated LevelDB Store mode as described [here|http://activemq.apache.org/replicated-leveldb-store.html]. My cluster contains of 3 nodes: Mac OS X + Windows + Linux Mint. In my case Mac OS X node was started first and becomes initial master.
> 2. Start a process which constantly adds messages to a single queue using URL as "failover(...)".
> 3. In the middle of that process shut down current master node (activemq stop)
> 4. Observe log from above on one of the slave nodes.
> My activeMQ and levelDB is a fresh copy - just downloaded and installed it. My ZK version is 3.4.6.
> Please ask for any additional data you need - I will be able to provide it.
> Thank you very much for your help in advance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)