You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Kan Zhang (JIRA)" <ji...@apache.org> on 2009/05/16 02:39:45 UTC
[jira] Created: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
-------------------------------------------------------------------
Key: HADOOP-5859
URL: https://issues.apache.org/jira/browse/HADOOP-5859
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Kan Zhang
Assignee: Kan Zhang
This JIRA fixes the following warnings:
SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-8.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712631#action_12712631 ]
dhruba borthakur commented on HADOOP-5859:
------------------------------------------
> 2) dataQueue lock and ackQueue lock are merged into one
Is this done to make the code simpler? Or was there some race condition that this fixes?
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-4.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Open (was: Patch Available)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713172#action_12713172 ]
Todd Lipcon commented on HADOOP-5859:
-------------------------------------
bq. Unless the variable is declared as "volatile", right?
Right. volatile longs, though, are still not atomically assigned on 32-bit platforms, so I think it's asking for trouble to use the atomic assignment of volatile ints to your advantage, since someone may co
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Open (was: Patch Available)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch, 5859-5.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714544#action_12714544 ]
Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------
# Remove commented line setting maxPacket to 1000
# {{DataStreamer.run()}} No need to set {{response}} to null, because closeResponse() called prior, sets the response to null
# {{DataStreamer.run()}} code could be refactor to use closeStream() method instead of duplicating the code to close {{blockStream}}. A flag could be passed to indicate indicate end-of-block.
# {{ResponseProcessor.processDataNodeError()}} could use closeStream method.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Patch Available (was: Open)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch, 5859-5.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Open (was: Patch Available)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-36.patch
added a new patch with some refactoring of closeStream().
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713681#action_12713681 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
> As we discussed, in ResponseProcessor.run() add notify when exception is caught
As we discussed again, it's better to avoid adding unnecessary notifications, but rather add a timeout to dataQueue.wait(). Other comments are addressed in the new patch. Thanks.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713061#action_12713061 ]
Todd Lipcon commented on HADOOP-5859:
-------------------------------------
bq. which is ok assuming int assignments are atomic
I haven't had a chance to look through the patch, but in general this is not a safe assumption. Although int assignment itself is atomic, if you don't do access inside a synchronization block, there's no enforcement of a memory barrier between threads, so one thread might not see the other thread's assignment. Also, if someone later decides that size should be a long, long assignment is not atomic on some JVM platforms. Thus the existence of java.util.concurrent.atomic.AtomicInteger.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714650#action_12714650 ]
Hadoop QA commented on HADOOP-5859:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409409/5859-35.patch
against trunk revision 780114.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/429/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/429/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/429/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/429/console
This message is automatically generated.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5859:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Kan!
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Fix For: 0.21.0
>
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714933#action_12714933 ]
Hadoop QA commented on HADOOP-5859:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409430/5859-36.patch
against trunk revision 780465.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/442/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/442/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/442/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/442/console
This message is automatically generated.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-41.patch
trivial updates.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716086#action_12716086 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
my local hdfs unit tests passed. here is the test-patch result.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713307#action_12713307 ]
Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------
Comments on the latest patch:
# Can you add a comment where dataQueue and ackQueue are declared, that both these data structures are synchronized on dataQueue object
# Can you retain the code formatting where you think the existing code is fine. That makes the patch size smaller and applying patches to the changed code easier.
# Move static variable to the beginning of the class {{DFSOutputStream}}
# {{maxRecoveryErrorCount}} should become static
# Do we need so many volatile variables. Can one thread flag close and the DataStreamer does all the closing of internal variables
# As we discussed, in {{ResponseProcessor.run()}} add notify when exception is caught
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-19.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-19.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713180#action_12713180 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
> Unless the variable is declared as "volatile", right?
I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value (does anyone know how long the value will be cached a thread, I hope it's not forever :). In any case, this is a non-issue after my refactoring. In the refactored code, all accesses to dataQueue or ackQueue are sync'ed on dataQueue lock.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-35.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712832#action_12712832 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
> Is this done to make the code simpler? Or was there some race condition that this fixes?
I don't see any race condition so far. One potential case is calling ackQueue.size() in writeChunk() without holding ackQueue lock, which is ok assuming int assignments are atomic. The major reasons for refactoring the locks are 1) existing code is bit confusing. Sometimes the dataQueue lock is used to protect accesses to not only dataQueue but also currentPacket, blockStream, blockReplyStream, etc (see closeInternal() for example), which I think is a misuse. 2) I have moved time-consuming operations like RPC call or writing to socket out of blocks sync'ed on the dataQueue. So now the only times one needs to hold dataQueue or ackQueue lock is when manipulating the queues themselves. By merging the 2 locks, code can be simpler.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-40.patch
uploading a new patch, which doesn't send end-of-block when closing (end-of-block was sent unnecessarily in the original code). also addressed Suresh's earlier comments.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-22.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710155#action_12710155 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
attached 2 versions of the same patch. one with correct indentation, one without, for easier reviewing.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch, 5859-5.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-26.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715636#action_12715636 ]
Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------
# {{DataStreamer.nextBlockOutputStream()}} unnecessary code setting {{nodes}} to null (I see a Dead store to nodes findbugs warnings).
# nit: You can get rid of parameter to {{DataStreamer.closeStream()}} and use {{DataStreamer.sendEndOfBlock}} instead. Simplifies understanding the cases when end of block is sent.
# nit: {{DataStreamer.close()}} - it is a good idea to move printing the log of sending end of block to {{DataStreamer.closeStreamer()}} method.
# nit: Move the comments added for {{DataStreamer.run()}} to the beginning of {{DataStreamer}} class.
# nit: Capitalize {{MaxRecoveryErrorCount}}
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: (was: 5859-19.patch)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714557#action_12714557 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
thanks, Suresh. Attached a new patch that addressed your comments.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713177#action_12713177 ]
dhruba borthakur commented on HADOOP-5859:
------------------------------------------
>From the section 17.7 of the Java Language Specs: "Writes and reads of volatile long and double values are always atomic. ". I think all JVMs should ensure that all "volatile longs" are atomic.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715996#action_12715996 ]
Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------
+1 with all the tests passing
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716271#action_12716271 ]
Hadoop QA commented on HADOOP-5859:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409796/5859-41.patch
against trunk revision 781683.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/461/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/461/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/461/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/461/console
This message is automatically generated.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-38.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-21.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5859) FindBugs : fix "wait()
or sleep() with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713180#action_12713180 ]
Kan Zhang edited comment on HADOOP-5859 at 5/26/09 11:24 AM:
-------------------------------------------------------------
> Unless the variable is declared as "volatile", right?
I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value (does anyone know how long the value will be cached by a thread, I hope it's not forever :)). In any case, this is a non-issue after my refactoring. In the refactored code, all accesses to dataQueue or ackQueue are sync'ed on dataQueue lock.
was (Author: kzhang):
> Unless the variable is declared as "volatile", right?
I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value (does anyone know how long the value will be cached a thread, I hope it's not forever :). In any case, this is a non-issue after my refactoring. In the refactored code, all accesses to dataQueue or ackQueue are sync'ed on dataQueue lock.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Patch Available (was: Open)
submit again to test the newest patch - 5859-36.patch.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Patch Available (was: Open)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713174#action_12713174 ]
Todd Lipcon commented on HADOOP-5859:
-------------------------------------
er... trackpad malfunction. sorry...
continuing:
someone may come along at a later date and replace the int with a long thinking it's a trivial change that couldn't possibly break anything.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Status: Patch Available (was: Open)
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713188#action_12713188 ]
Todd Lipcon commented on HADOOP-5859:
-------------------------------------
Hm, thanks Dhruba - I was recalling incorrectly. The discussion is getting way off-topic to this JIRA, but if people are interested, I highly recommend this video lecture: http://video.google.com/videoplay?docid=8394326369005388010
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-5.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-4.patch, 5859-5.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5859) FindBugs : fix "wait()
or sleep() with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713180#action_12713180 ]
Kan Zhang edited comment on HADOOP-5859 at 6/4/09 5:55 PM:
-----------------------------------------------------------
> Unless the variable is declared as "volatile", right?
I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value. In any case, this is a non-issue after my refactoring. In the refactored code, all accesses to dataQueue or ackQueue are sync'ed on dataQueue lock.
was (Author: kzhang):
> Unless the variable is declared as "volatile", right?
I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value (does anyone know how long the value will be cached by a thread, I hope it's not forever :)). In any case, this is a non-issue after my refactoring. In the refactored code, all accesses to dataQueue or ackQueue are sync'ed on dataQueue lock.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Fix For: 0.21.0
>
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5859:
------------------------------
Attachment: 5859-33.patch
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714631#action_12714631 ]
Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------
+1. Looks good.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713161#action_12713161 ]
dhruba borthakur commented on HADOOP-5859:
------------------------------------------
> so one thread might not see the other thread's assignment.
Unless the variable is declared as "volatile", right?
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715605#action_12715605 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
After discussing with Raghu, I'm uploading a new patch where a new flag is added so that end-of-block is sent only when all data has been sent and ack'ed from datanode.
This patch doesn't include any new test since it's only a refactoring and no new feature is added.
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713894#action_12713894 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
> I just checked, the JDK6 implementation of LinkedList didn't declare the "size" variable volatile. So Todd is right, a second thread may see a stale cached value
I take it back. Actually, in this particular case the writeChunk() thread will be able to see the fresh value of ackQueue size whenever it's been modified. The reason is that writeChunk() reads ackQueue size while holding the dataQueue lock and waits on the dataQueue lock in a loop when condition is not satisfied. There are 2 threads that may modify ackQueue size. One is DataStreamer, who always holds both dataQueue and ackQueue locks when modifying ackQueue. So any changes to ackQueue by DataStreamer will be visible to writeChunk() thread. The other thread that modifies ackQueue is ResponseProcessor. It may modify ackQueue while holding only the ackQueue lock. However, immediately after ackQueue modification, it grabs the dataQueue lock and calls dataQueue.notifyAll() to wake up the writeChunk() thread. So when the writeChunk() thread grabs the dataQueue lock afterwards, the fresh value of ackQueue size will be visible. Make sense?
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
with locks held" warnings in hdfs
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712275#action_12712275 ]
Kan Zhang commented on HADOOP-5859:
-----------------------------------
added a new patch that refactored DFSClient.DFSOutputStream code.
1) socket write operation in DataStreamer is now done without holding dataQueue lock
2) dataQueue lock and ackQueue lock are merged into one
3) removed the above findbugs warnings
> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
> Key: HADOOP-5859
> URL: https://issues.apache.org/jira/browse/HADOOP-5859
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 5859-19.patch, 5859-4.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.