You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Raman Grover (JIRA)" <ji...@apache.org> on 2010/09/16 20:32:33 UTC
[jira] Created: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
--------------------------------------------------------------------------------------------------------
Key: HIVE-1647
URL: https://issues.apache.org/jira/browse/HIVE-1647
Project: Hadoop Hive
Issue Type: Bug
Components: Server Infrastructure
Affects Versions: 0.5.0
Reporter: Raman Grover
Fix For: 0.3.1
Bug in org.apache.hadoop.hive.ql.io.IOContext
in relation to initialization of thread local variable.
public class IOContext {
private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
static {
if (threadLocal.get() == null) {
threadLocal.set(new IOContext());
}
}
In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
any subsequent threads would have their respective threadlocal variable as null. Since IOContext
is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
protected synchronized IOContext initialValue() {
return new IOContext();
}
};
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916676#action_12916676 ]
He Yongqiang commented on HIVE-1647:
------------------------------------
+1 running tests
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: HIVE-1647.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916712#action_12916712 ]
He Yongqiang commented on HIVE-1647:
------------------------------------
The test is still running, but there are a lot of diffs. Can you take a look? Examples like join_map_ppr.q, input_part10.q
You can use ' ant test -Dtestcase=TestCliDriver -Dqfile=join_map_ppr.q,input_part10.q ' to reproduce.
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: HIVE-1647.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang resolved HIVE-1647.
------------------------------
Resolution: Fixed
Release Note: This problem is fixed in Hive-1754
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: HIVE-1647.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1647:
-----------------------------
Status: Open (was: Patch Available)
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: HIVE-1647.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-1647:
--------------------------------
Assignee: Raman Grover
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.5.0
> Reporter: Raman Grover
> Assignee: Raman Grover
> Fix For: 0.3.1
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1647:
-----------------------------
Attachment: HIVE-1647.patch
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: HIVE-1647.patch
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1647:
-----------------------------
Status: Open (was: Patch Available)
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910259#action_12910259 ]
He Yongqiang commented on HIVE-1647:
------------------------------------
It's good to have the fix you proposed. Can you post a patch?
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.5.0
> Reporter: Raman Grover
> Fix For: 0.3.1
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1647:
-----------------------------
Status: Patch Available (was: Open)
Affects Version/s: 0.6.0
0.7.0
(was: 0.5.0)
Assignee: Liyin Tang (was: Raman Grover)
Fix Version/s: 0.7.0
(was: 0.3.1)
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1647) Incorrect initialization of thread
local variable inside IOContext ( implementation is not threadsafe )
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1647:
-----------------------------
Status: Patch Available (was: Open)
> Incorrect initialization of thread local variable inside IOContext ( implementation is not threadsafe )
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1647
> URL: https://issues.apache.org/jira/browse/HIVE-1647
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Server Infrastructure
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Raman Grover
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Original Estimate: 0.17h
> Remaining Estimate: 0.17h
>
> Bug in org.apache.hadoop.hive.ql.io.IOContext
> in relation to initialization of thread local variable.
>
> public class IOContext {
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){ };
>
> static {
> if (threadLocal.get() == null) {
> threadLocal.set(new IOContext());
> }
> }
>
> In a multi-threaded environment, the thread that gets to load the class first for the JVM (assuming threads share the classloader),
> gets to initialize itself correctly by executing the code in the static block. Once the class is loaded,
> any subsequent threads would have their respective threadlocal variable as null. Since IOContext
> is set during initialization of HiveRecordReader, In a scenario where multiple threads get to acquire
> an instance of HiveRecordReader, it would result in a NPE for all but the first thread that gets to load the class in the VM.
>
> Is the above scenario of multiple threads initializing HiveRecordReader a typical one ? or we could just provide the following fix...
>
> private static ThreadLocal<IOContext> threadLocal = new ThreadLocal<IOContext>(){
> protected synchronized IOContext initialValue() {
> return new IOContext();
> }
> };
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.