You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2011/07/07 21:52:16 UTC
[jira] [Created] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Silent Data Offlining During HDFS Flakiness
-------------------------------------------
Key: HBASE-4078
URL: https://issues.apache.org/jira/browse/HBASE-4078
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.3, 0.89.20100924, 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Blocker
See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064349#comment-13064349 ]
stack commented on HBASE-4078:
------------------------------
This looks like its similar to HBASE-3834
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093326#comment-13093326 ]
Lars Hofhansl commented on HBASE-4078:
--------------------------------------
Ah never mind me... HDFS flakiness is what this is all about.
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127281#comment-13127281 ]
Jonathan Gray commented on HBASE-4078:
--------------------------------------
This seems to have somehow broken cache-on-write again. I think because the verify does a closeReader() which could trigger the evict-on-close.
I'm going to need to extend the close API to take evictOnClose as an argument. I think there's actually a JIRA for this already.
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4078:
----------------------------------
Attachment: (was: 0001-HBASE-4078-Move-corrupt-store-files-out-of-main-stor.patch)
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081341#comment-13081341 ]
Pritam Damania commented on HBASE-4078:
---------------------------------------
Here is the review board link for this patch : https://reviews.apache.org/r/1327/diff/
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg reassigned HBASE-4078:
------------------------------------------
Assignee: Pritam Damania (was: Nicolas Spiegelberg)
Pritam is working on a fix for this.
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Nicolas Spiegelberg (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg resolved HBASE-4078.
----------------------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
0.92.0
added to 89, 92, & 94
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093320#comment-13093320 ]
Lars Hofhansl commented on HBASE-4078:
--------------------------------------
When does the corruption actually happen?
Does any of StoreFile.Writer.{append|appendMetadata|close}(...) silently fail, leaving a corrupt file? If any of these throws any exception we would skip moving the file anyway.
If so, wouldn't it be better to fix that?
Or is this a problem deeper in HDFS?
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4078:
----------------------------------
Attachment: 0001-Validate-store-files-after-compactions-flushes.patch
Updated Patch for HBASE-4078
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081435#comment-13081435 ]
stack commented on HBASE-4078:
------------------------------
I added some comments over on reviewboard but then realized that the patch looks like hbase-4078. Is it same patch? Thanks.
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4078:
----------------------------------
Attachment: 0001-Validate-store-files.patch
This is a patch for HBASE-4078
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4078:
----------------------------------
Attachment: 0001-HBASE-4078-Move-corrupt-store-files-out-of-main-stor.patch
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-HBASE-4078-Move-corrupt-store-files-out-of-main-stor.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127103#comment-13127103 ]
Hudson commented on HBASE-4078:
-------------------------------
Integrated in HBase-0.92 #62 (See [https://builds.apache.org/job/HBase-0.92/62/])
HBASE-4078 Validate store files after flush/compaction
nspiegelberg :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081439#comment-13081439 ]
stack commented on HBASE-4078:
------------------------------
Pardon my sillyness above where I am saying that the patch for this issue is the same as the patch for this issue.
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS
Flakiness
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128381#comment-13128381 ]
Hudson commented on HBASE-4078:
-------------------------------
Integrated in HBase-TRUNK #2325 (See [https://builds.apache.org/job/HBase-TRUNK/2325/])
HBASE-4078 Validate store files after flush/compaction
nspiegelberg :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
> Silent Data Offlining During HDFS Flakiness
> -------------------------------------------
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Pritam Damania
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 0001-Validate-store-files-after-compactions-flushes.patch, 0001-Validate-store-files.patch
>
>
> See HBASE-1436 . The bug fix for this JIRA is a temporary workaround for improperly moving partially-written files from TMP into the region directory when a FS error occurs. Unfortunately, the fix is to ignore all IO exceptions, which masks off-lining due to FS flakiness. We need to permanently fix the problem that created HBASE-1436 & then at least have the option to not open a region during times of flakey FS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira