You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Qiang Wang <ws...@gmail.com> on 2013/01/04 09:28:20 UTC
HiveHistoryViewer concurrency problem
new HiveHistoryViewer() throws ConcurrentModificationException when called
concurrently by several threads.
According to the stack trace, HiveHistory.parseLine use *private static
Map<String, String> parseBuffer* to store parsed data and this caused the
exception.
I don't know why a static buffer rather than a local buffer is used!
Anybody have an idea about this?
Re: HiveHistoryViewer concurrency problem
Posted by Qiang Wang <ws...@gmail.com>.
Hi Jie:
As I know, hive history log is structured and class *HiveHistory* is used
to write and read hive history log.
*HiveHistoryViewer* serves as a listener to listen and store parsed log
data. It has two members:
HashMap<String, QueryInfo> *jobInfoMap*, which stores QueryInfo related
with hive query
and
HashMap<String, TaskInfo> *taskInfoMap*, which stores TaskInfo related with
hadoop map/red job
you can dump the two maps and find what you want.
Hope these info helps
Qiang
2013/1/5 Jie Li <ji...@cs.duke.edu>
> Hi Qiang,
>
> Could you describe how HiveHistoryViewer is used? I'm also looking for
> a tool to understand the Hive log.
>
> Thanks,
> Jie
>
> On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang <ws...@gmail.com> wrote:
> > Does Anybody have an idea about this?
> >
> > https://issues.apache.org/jira/browse/HIVE-3857
> >
> >
> > 2013/1/4 Qiang Wang <ws...@gmail.com>
> >>
> >> new HiveHistoryViewer() throws ConcurrentModificationException when
> called
> >> concurrently by several threads.
> >>
> >> According to the stack trace, HiveHistory.parseLine use private static
> >> Map<String, String> parseBuffer to store parsed data and this caused the
> >> exception.
> >>
> >> I don't know why a static buffer rather than a local buffer is used!
> >> Anybody have an idea about this?
> >
> >
>
Re: HiveHistoryViewer concurrency problem
Posted by Qiang Wang <ws...@gmail.com>.
Maybe it's not.
But this exception happens when I create an *HiveHistoryViewer* instance,
in which case only reading, parsing file is invloved and it's not intended
to be shared between threads.
So the exception surprised me and I wonder why a static buffer was used
instead of a local buffer which has no concurrent issue.
2013/1/5 Edward Capriolo <ed...@gmail.com>
> It is likely an oversight. The Majority of hive code was not written to be
> multi-threaded.
>
>
>
> On Fri, Jan 4, 2013 at 10:41 PM, Jie Li <ji...@cs.duke.edu> wrote:
>
>> Hi Qiang,
>>
>> Could you describe how HiveHistoryViewer is used? I'm also looking for
>> a tool to understand the Hive log.
>>
>> Thanks,
>> Jie
>>
>> On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang <ws...@gmail.com> wrote:
>> > Does Anybody have an idea about this?
>> >
>> > https://issues.apache.org/jira/browse/HIVE-3857
>> >
>> >
>> > 2013/1/4 Qiang Wang <ws...@gmail.com>
>> >>
>> >> new HiveHistoryViewer() throws ConcurrentModificationException when
>> called
>> >> concurrently by several threads.
>> >>
>> >> According to the stack trace, HiveHistory.parseLine use private static
>> >> Map<String, String> parseBuffer to store parsed data and this caused
>> the
>> >> exception.
>> >>
>> >> I don't know why a static buffer rather than a local buffer is used!
>> >> Anybody have an idea about this?
>> >
>> >
>>
>
>
Re: HiveHistoryViewer concurrency problem
Posted by Edward Capriolo <ed...@gmail.com>.
It is likely an oversight. The Majority of hive code was not written to be
multi-threaded.
On Fri, Jan 4, 2013 at 10:41 PM, Jie Li <ji...@cs.duke.edu> wrote:
> Hi Qiang,
>
> Could you describe how HiveHistoryViewer is used? I'm also looking for
> a tool to understand the Hive log.
>
> Thanks,
> Jie
>
> On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang <ws...@gmail.com> wrote:
> > Does Anybody have an idea about this?
> >
> > https://issues.apache.org/jira/browse/HIVE-3857
> >
> >
> > 2013/1/4 Qiang Wang <ws...@gmail.com>
> >>
> >> new HiveHistoryViewer() throws ConcurrentModificationException when
> called
> >> concurrently by several threads.
> >>
> >> According to the stack trace, HiveHistory.parseLine use private static
> >> Map<String, String> parseBuffer to store parsed data and this caused the
> >> exception.
> >>
> >> I don't know why a static buffer rather than a local buffer is used!
> >> Anybody have an idea about this?
> >
> >
>
Re: HiveHistoryViewer concurrency problem
Posted by Jie Li <ji...@cs.duke.edu>.
Hi Qiang,
Could you describe how HiveHistoryViewer is used? I'm also looking for
a tool to understand the Hive log.
Thanks,
Jie
On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang <ws...@gmail.com> wrote:
> Does Anybody have an idea about this?
>
> https://issues.apache.org/jira/browse/HIVE-3857
>
>
> 2013/1/4 Qiang Wang <ws...@gmail.com>
>>
>> new HiveHistoryViewer() throws ConcurrentModificationException when called
>> concurrently by several threads.
>>
>> According to the stack trace, HiveHistory.parseLine use private static
>> Map<String, String> parseBuffer to store parsed data and this caused the
>> exception.
>>
>> I don't know why a static buffer rather than a local buffer is used!
>> Anybody have an idea about this?
>
>
Re: HiveHistoryViewer concurrency problem
Posted by Qiang Wang <ws...@gmail.com>.
Does Anybody have an idea about this?
https://issues.apache.org/jira/browse/HIVE-3857
2013/1/4 Qiang Wang <ws...@gmail.com>
> new HiveHistoryViewer() throws ConcurrentModificationException when called
> concurrently by several threads.
>
> According to the stack trace, HiveHistory.parseLine use *private static
> Map<String, String> parseBuffer* to store parsed data and this caused the
> exception.
>
> I don't know why a static buffer rather than a local buffer is used!
> Anybody have an idea about this?
>