You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Edward Capriolo <ed...@gmail.com> on 2013/08/10 19:32:22 UTC
Discuss: End of static, thread local
I just committed https://issues.apache.org/jira/browse/HIVE-3772.
For hive-server2 Carl and others did a lot of work to clean up un thread
safe things from hive.
Hive was originally build as a fat client so it is not surprising that many
such constructs exist. Now since we have retrofitted multi-threaded-ness
onto the project we have a number of edge case bugs.
My suggestions here would be for that the next release 0.13 we make a push
to remove all possible non thread safe code and explicitly pass context
objects or serialized structures everywhere thread safety is needed.
I can see this would start with something like the Function Registry, this
would be a per session object passed around rather then a global object
with static hashmap instances in it.
I know that this probably will not be as simple as removing all static
members from our codebase, but does anyone know of specific challenges that
will be intrinsically hard to solve?
Please comment.
Re: Discuss: End of static, thread local
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
+1
That's definitely the right way forward. These thread local variables
should be verboten.
I also agree with Brock's assessment that this will need a design doc
first. My guess is that this will be quite a bit of work.
Thanks,
Gunther.
On Sun, Aug 11, 2013 at 12:34 PM, Prasad Mujumdar <pr...@cloudera.com>wrote:
> +1 !
> That's should be the right approach IMO too.
>
> thanks
> Prasad
>
>
>
> On Sun, Aug 11, 2013 at 10:03 AM, Brock Noland <br...@cloudera.com> wrote:
>
> > HIVE-4226 is a very welcome patch but in the long run I think we want to
> > move away from using static thread local everywhere which that patch
> uses.
> > Rightfully so at this juncture.
> >
> > Meaning I think that HIVE-4226 is a good interim solution but long term I
> > think we want to be passing around some kind of context object as Edward
> > suggests in his original mail. This will make the code cleaner and more
> > testable.
> >
> > This is a big effort and will require some analysis and planning before
> > execution.
> >
> >
> > On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 <na...@nexr.com> wrote:
> >
> > > https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing
> this.
> > >
> > > 2013/8/11 Brock Noland <br...@cloudera.com>:
> > > > I would love to get rid of the static thread local stuff. It was
> > required
> > > > to make hive work in a server model but isn't the correct solution to
> > > this
> > > > problem.
> > > >
> > > > I do think it will be a large amount of work so it'd be great to see
> > > > whoever leads this effort to have a high level plan as opposed to an
> > > adhoc
> > > > effort.
> > > >
> > > >
> > > > On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo <
> > edlinuxguru@gmail.com
> > > >wrote:
> > > >
> > > >> I just committed https://issues.apache.org/jira/browse/HIVE-3772.
> > > >>
> > > >> For hive-server2 Carl and others did a lot of work to clean up un
> > thread
> > > >> safe things from hive.
> > > >>
> > > >> Hive was originally build as a fat client so it is not surprising
> that
> > > many
> > > >> such constructs exist. Now since we have retrofitted
> > multi-threaded-ness
> > > >> onto the project we have a number of edge case bugs.
> > > >>
> > > >> My suggestions here would be for that the next release 0.13 we make
> a
> > > push
> > > >> to remove all possible non thread safe code and explicitly pass
> > context
> > > >> objects or serialized structures everywhere thread safety is needed.
> > > >>
> > > >> I can see this would start with something like the Function
> Registry,
> > > this
> > > >> would be a per session object passed around rather then a global
> > object
> > > >> with static hashmap instances in it.
> > > >>
> > > >> I know that this probably will not be as simple as removing all
> static
> > > >> members from our codebase, but does anyone know of specific
> challenges
> > > that
> > > >> will be intrinsically hard to solve?
> > > >>
> > > >> Please comment.
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> > >
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> >
>
Re: Discuss: End of static, thread local
Posted by Prasad Mujumdar <pr...@cloudera.com>.
+1 !
That's should be the right approach IMO too.
thanks
Prasad
On Sun, Aug 11, 2013 at 10:03 AM, Brock Noland <br...@cloudera.com> wrote:
> HIVE-4226 is a very welcome patch but in the long run I think we want to
> move away from using static thread local everywhere which that patch uses.
> Rightfully so at this juncture.
>
> Meaning I think that HIVE-4226 is a good interim solution but long term I
> think we want to be passing around some kind of context object as Edward
> suggests in his original mail. This will make the code cleaner and more
> testable.
>
> This is a big effort and will require some analysis and planning before
> execution.
>
>
> On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 <na...@nexr.com> wrote:
>
> > https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this.
> >
> > 2013/8/11 Brock Noland <br...@cloudera.com>:
> > > I would love to get rid of the static thread local stuff. It was
> required
> > > to make hive work in a server model but isn't the correct solution to
> > this
> > > problem.
> > >
> > > I do think it will be a large amount of work so it'd be great to see
> > > whoever leads this effort to have a high level plan as opposed to an
> > adhoc
> > > effort.
> > >
> > >
> > > On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo <
> edlinuxguru@gmail.com
> > >wrote:
> > >
> > >> I just committed https://issues.apache.org/jira/browse/HIVE-3772.
> > >>
> > >> For hive-server2 Carl and others did a lot of work to clean up un
> thread
> > >> safe things from hive.
> > >>
> > >> Hive was originally build as a fat client so it is not surprising that
> > many
> > >> such constructs exist. Now since we have retrofitted
> multi-threaded-ness
> > >> onto the project we have a number of edge case bugs.
> > >>
> > >> My suggestions here would be for that the next release 0.13 we make a
> > push
> > >> to remove all possible non thread safe code and explicitly pass
> context
> > >> objects or serialized structures everywhere thread safety is needed.
> > >>
> > >> I can see this would start with something like the Function Registry,
> > this
> > >> would be a per session object passed around rather then a global
> object
> > >> with static hashmap instances in it.
> > >>
> > >> I know that this probably will not be as simple as removing all static
> > >> members from our codebase, but does anyone know of specific challenges
> > that
> > >> will be intrinsically hard to solve?
> > >>
> > >> Please comment.
> > >>
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
Re: Discuss: End of static, thread local
Posted by Brock Noland <br...@cloudera.com>.
HIVE-4226 is a very welcome patch but in the long run I think we want to
move away from using static thread local everywhere which that patch uses.
Rightfully so at this juncture.
Meaning I think that HIVE-4226 is a good interim solution but long term I
think we want to be passing around some kind of context object as Edward
suggests in his original mail. This will make the code cleaner and more
testable.
This is a big effort and will require some analysis and planning before
execution.
On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 <na...@nexr.com> wrote:
> https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this.
>
> 2013/8/11 Brock Noland <br...@cloudera.com>:
> > I would love to get rid of the static thread local stuff. It was required
> > to make hive work in a server model but isn't the correct solution to
> this
> > problem.
> >
> > I do think it will be a large amount of work so it'd be great to see
> > whoever leads this effort to have a high level plan as opposed to an
> adhoc
> > effort.
> >
> >
> > On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo <edlinuxguru@gmail.com
> >wrote:
> >
> >> I just committed https://issues.apache.org/jira/browse/HIVE-3772.
> >>
> >> For hive-server2 Carl and others did a lot of work to clean up un thread
> >> safe things from hive.
> >>
> >> Hive was originally build as a fat client so it is not surprising that
> many
> >> such constructs exist. Now since we have retrofitted multi-threaded-ness
> >> onto the project we have a number of edge case bugs.
> >>
> >> My suggestions here would be for that the next release 0.13 we make a
> push
> >> to remove all possible non thread safe code and explicitly pass context
> >> objects or serialized structures everywhere thread safety is needed.
> >>
> >> I can see this would start with something like the Function Registry,
> this
> >> would be a per session object passed around rather then a global object
> >> with static hashmap instances in it.
> >>
> >> I know that this probably will not be as simple as removing all static
> >> members from our codebase, but does anyone know of specific challenges
> that
> >> will be intrinsically hard to solve?
> >>
> >> Please comment.
> >>
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Discuss: End of static, thread local
Posted by Navis류승우 <na...@nexr.com>.
https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this.
2013/8/11 Brock Noland <br...@cloudera.com>:
> I would love to get rid of the static thread local stuff. It was required
> to make hive work in a server model but isn't the correct solution to this
> problem.
>
> I do think it will be a large amount of work so it'd be great to see
> whoever leads this effort to have a high level plan as opposed to an adhoc
> effort.
>
>
> On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> I just committed https://issues.apache.org/jira/browse/HIVE-3772.
>>
>> For hive-server2 Carl and others did a lot of work to clean up un thread
>> safe things from hive.
>>
>> Hive was originally build as a fat client so it is not surprising that many
>> such constructs exist. Now since we have retrofitted multi-threaded-ness
>> onto the project we have a number of edge case bugs.
>>
>> My suggestions here would be for that the next release 0.13 we make a push
>> to remove all possible non thread safe code and explicitly pass context
>> objects or serialized structures everywhere thread safety is needed.
>>
>> I can see this would start with something like the Function Registry, this
>> would be a per session object passed around rather then a global object
>> with static hashmap instances in it.
>>
>> I know that this probably will not be as simple as removing all static
>> members from our codebase, but does anyone know of specific challenges that
>> will be intrinsically hard to solve?
>>
>> Please comment.
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Discuss: End of static, thread local
Posted by Brock Noland <br...@cloudera.com>.
I would love to get rid of the static thread local stuff. It was required
to make hive work in a server model but isn't the correct solution to this
problem.
I do think it will be a large amount of work so it'd be great to see
whoever leads this effort to have a high level plan as opposed to an adhoc
effort.
On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo <ed...@gmail.com>wrote:
> I just committed https://issues.apache.org/jira/browse/HIVE-3772.
>
> For hive-server2 Carl and others did a lot of work to clean up un thread
> safe things from hive.
>
> Hive was originally build as a fat client so it is not surprising that many
> such constructs exist. Now since we have retrofitted multi-threaded-ness
> onto the project we have a number of edge case bugs.
>
> My suggestions here would be for that the next release 0.13 we make a push
> to remove all possible non thread safe code and explicitly pass context
> objects or serialized structures everywhere thread safety is needed.
>
> I can see this would start with something like the Function Registry, this
> would be a per session object passed around rather then a global object
> with static hashmap instances in it.
>
> I know that this probably will not be as simple as removing all static
> members from our codebase, but does anyone know of specific challenges that
> will be intrinsically hard to solve?
>
> Please comment.
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org