You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2008/02/15 16:32:07 UTC
[jira] Created: (HADOOP-2837) JobTracker got stuck
JobTracker got stuck
---------------------
Key: HADOOP-2837
URL: https://issues.apache.org/jira/browse/HADOOP-2837
Project: Hadoop Core
Issue Type: Bug
Reporter: Runping Qi
When a job tracker ran a job with a lot of map tasks,
the commit thread complained out of memory:
2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570951#action_12570951 ]
Amar Kamat commented on HADOOP-2837:
------------------------------------
My jobs were back to back sort200. I kept it running for a day in a loop.
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569305#action_12569305 ]
Devaraj Das commented on HADOOP-2837:
-------------------------------------
How many tasks were there in the job? Did you try increasing the heap size for the JT?
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570814#action_12570814 ]
Amar Kamat commented on HADOOP-2837:
------------------------------------
I guess its the history in the JT that causes these heap issues. As of now we keep 24hr history in the JT. I observed this when I ran lot of jobs within 24hrs back to back. Thoughts?
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570834#action_12570834 ]
Arun C Murthy commented on HADOOP-2837:
---------------------------------------
bq. As of now we keep 24hr history in the JT.
The JobTracker keeps only a 100 jobs per user in memory (which you can configure to be a lesser no.) ... so I'm not very convinced - these would have to be significantly large back-to-back jobs. Is that what you observed?
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569322#action_12569322 ]
Runping Qi commented on HADOOP-2837:
------------------------------------
It's not my job, and since the web gui does not response, I don't know the exact number of mappers.
But through log, I know it has at least 60000 mappers.
The heapsize is 1000M (the default, I believe).
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579062#action_12579062 ]
Ashish Thusoo commented on HADOOP-2837:
---------------------------------------
Hi..
We are hitting the same issue. It has happened two times in two days for us.
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569363#action_12569363 ]
Christian Kunz commented on HADOOP-2837:
----------------------------------------
Runping, are these mappers executing relatively fast?
As side-effect of HADOOP-2119, without any patch, our JobTracker needed at least 2GB (1000+ node cluster, 50,000+ mappers).
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569611#action_12569611 ]
Amar Kamat commented on HADOOP-2837:
------------------------------------
Even I have seen it. One can be reproduced as follows
{noformat}
1. Run random writer
2. for i=0; i<20; i++
2.1 Run sort
2.2 delete sort output
{noformat}
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579716#action_12579716 ]
Amar Kamat commented on HADOOP-2837:
------------------------------------
I saw these errors on trunk.
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579701#action_12579701 ]
Pete Wyckoff commented on HADOOP-2837:
--------------------------------------
Amar,
What version of Hadoop are you hitting this issue on. We're seeing it pretty frequently on our 15.3 cluster. I don't see any patch or JIRA pointing to a patch, so I assume this hasn't been fixed even in 0.16.1. Just wondering if reproducible in 16.
thanks, pete
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2837) JobTracker got stuck
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated HADOOP-2837:
---------------------------------
Description:
When a job tracker ran a job with a lot of map tasks,
the commit thread complained out of memory:
2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
then later, the job tracker stopped making any progress, and stopped responding web GUI.
was:
When a job tracker ran a job with a lot of map tasks,
the commit thread complained out of memory:
2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
then later, the job tracker stopped making any progress, and stopped responding web GUI.
Affects Version/s: 0.15.3
Asked Runping. The user was using 0.15.3.
So, HADOOP-2119(commited in 0.17) is not the cause.
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2837) JobTracker got stuck
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582909#action_12582909 ]
Amar Kamat commented on HADOOP-2837:
------------------------------------
I suspect the *queue* in the task commit queue. Pre HADOOP-2119 the queue items were processed one by one and hence the queue used to build up. Now with HADOOP-2119, commit is done in batches of size 5000. Chances of this error occurring with the trunk is less. We tried running 320,000 maps on 250 nodes where each map generated 8mb of data (similar to RandomWriter) and we never saw this error. Can someone confirm this?
> JobTracker got stuck
> ---------------------
>
> Key: HADOOP-2837
> URL: https://issues.apache.org/jira/browse/HADOOP-2837
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Runping Qi
>
> When a job tracker ran a job with a lot of map tasks,
> the commit thread complained out of memory:
> 2008-02-14 02:52:13,809 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:18,543 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:53:26,965 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> 2008-02-14 02:54:27,009 ERROR org.apache.hadoop.mapred.JobTracker: Task Commit Thread got an exception: java.lang.OutOfMemoryError: Java heap space
> then later, the job tracker stopped making any progress, and stopped responding web GUI.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.