You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2008/07/30 20:01:32 UTC
[jira] Created: (HADOOP-3863) Use a thread-local rather than static
ENCODER/DECODER variables in Text for synchronization
Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
-------------------------------------------------------------------------------------------
Key: HADOOP-3863
URL: https://issues.apache.org/jira/browse/HADOOP-3863
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.18.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.19.0
Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3863) Use a thread-local rather than static
ENCODER/DECODER variables in Text for synchronization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3863:
----------------------------------
Status: Patch Available (was: Open)
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3863) Use a thread-local rather than static
ENCODER/DECODER variables in Text for synchronization
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3863:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Arun!
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618553#action_12618553 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3863:
------------------------------------------------
> Nicholas - we are are using http://java.sun.com/javase/6/docs/api/java/nio/charset/CharsetDecoder.html#decode(java.nio.ByteBuffer) which does the whole 'decoding' operation, ditto for encoding.
Thanks Arun, I missed this.
BTW, TestFileAppend2 also works consistently in my Windows and Linux machines.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618545#action_12618545 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3863:
------------------------------------------------
> The test failure (org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend) is unrelated.
I am not sure whether it is unrelated since this patch changes a core class and this is the first time that TestFileAppend2.testComplexAppend fails. TestFileAppend2.testComplexAppend create a number of threads appending files concurrently. It seems to me that it might be related. Could we try Hudson one more time?
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618514#action_12618514 ]
Owen O'Malley commented on HADOOP-3863:
---------------------------------------
This looks good, but I'd suggest renaming to be {DE,EN}CODER_FACTORY to make it clear that each thread will get a different one.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624749#action_12624749 ]
Hudson commented on HADOOP-3863:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618525#action_12618525 ]
Doug Cutting commented on HADOOP-3863:
--------------------------------------
I filed HADOOP-3874 to address this. (We should file bugs whenever unrelated tests fail, no?)
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618550#action_12618550 ]
Arun C Murthy commented on HADOOP-3863:
---------------------------------------
Fair point, Doug - taken; although it seems TestFileAppend2 works consistently on my machine.
Nicholas - we are are using http://java.sun.com/javase/6/docs/api/java/nio/charset/CharsetDecoder.html#decode(java.nio.ByteBuffer) which does the whole 'decoding' operation, ditto for encoding.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618505#action_12618505 ]
Hadoop QA commented on HADOOP-3863:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12387218/HADOOP-3863_0_20080730.patch
against trunk revision 680823.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2985/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2985/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2985/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2985/console
This message is automatically generated.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618549#action_12618549 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3863:
------------------------------------------------
According to CharsetDecoder javadoc http://java.sun.com/javase/6/docs/api/java/nio/charset/CharsetDecoder.html, there is a specific sequence of *decoding operations*. In particular, we should call reset() before using the decoder. It seems we are not following it.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3863) Use a thread-local rather than static
ENCODER/DECODER variables in Text for synchronization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3863:
----------------------------------
Attachment: HADOOP-3863_1_20080730.patch
Renamed the variable, same patch.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3863) Use a thread-local rather than
static ENCODER/DECODER variables in Text for synchronization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618520#action_12618520 ]
Arun C Murthy commented on HADOOP-3863:
---------------------------------------
The test failure (org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend) is unrelated.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch, HADOOP-3863_1_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3863) Use a thread-local rather than static
ENCODER/DECODER variables in Text for synchronization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3863:
----------------------------------
Attachment: HADOOP-3863_0_20080730.patch
Fixed Text to use ThreadLocal encoders/decoders.
> Use a thread-local rather than static ENCODER/DECODER variables in Text for synchronization
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3863
> URL: https://issues.apache.org/jira/browse/HADOOP-3863
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: HADOOP-3863_0_20080730.patch
>
>
> Text.{ENCODER|DECODER} are static variables which need to be synchronized in Text.{read|write}String; given that lots of RPC code calls Text.{read|write}String the contention for this lock shows up very significantly on large clusters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.