You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Josh Ferguson <jo...@besquared.net> on 2008/12/08 21:08:24 UTC
Hadoop JobStatus
When launching off hive queries using hive -e is there a way to get the job
id so that I can just queue them up and go check their statuses later?
What's the general pattern for queueing and monitoring without using the
libraries directly?
Josh Ferguson
RE: Hadoop JobStatus
Posted by Joydeep Sen Sarma <js...@facebook.com>.
Filed https://issues.apache.org/jira/browse/HIVE-176
________________________________
From: Josh Ferguson [mailto:josh@besquared.net]
Sent: Monday, December 08, 2008 8:38 PM
To: hive-user@hadoop.apache.org
Subject: Re: Hadoop JobStatus
Sure I guess that's true, it doesn't matter to me either way, it'd just be great to have some kind of delimited logfile that we could get information from.
Josh
On Dec 8, 2008, at 2:15 PM, Zheng Shao wrote:
^A is not printable. That will create confusion to users looking at the log right?
Zheng
On Mon, Dec 8, 2008 at 1:46 PM, Josh <jo...@besquared.net>> wrote:
I'm gonna throw my vote in for a structured log format. Users could tail it and use whatever queuing or monitoring they wish. It's also probably just a 30 minute project for someone already familiar with the code. I suggest ^A seperated key=value pairs per log line.8
Josh Ferguson
On Dec 8, 2008, at 1:09 PM, Ashish Thusoo <at...@facebook.com>> wrote:
Perhaps we should just have another log4j channel for this instead of debug. The consumers can then just listen on this channel and take appropriate action.
Another option would be to implement this using a message queue (publish/subscribe system). We could leverage ActiveMQ or something similar, but that would be a bit more heavyweight but potentially people can develop or advanced monitoring applications around it.
Ashish
________________________________
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
Sent: Monday, December 08, 2008 12:17 PM
To: <ma...@hadoop.apache.org> hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Hadoop JobStatus
The jobid is printed out for non-silent session execution mode.
Since there's no structured interface - I had tried to have structured data emitted as key=value in the output stream. The relevant output emitted here is from:
console.printInfo("Starting Job = " + rj.getJobID() + ", Tracking URL = " + rj.getTrackingURL());
would really welcome a discussion on a better way to get structured data out from the output.
________________________________
From: Josh Ferguson [mailto:josh@besquared.net]
Sent: Monday, December 08, 2008 12:08 PM
To: <ma...@hadoop.apache.org> hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop JobStatus
When launching off hive queries using hive -e is there a way to get the job id so that I can just queue them up and go check their statuses later? What's the general pattern for queueing and monitoring without using the libraries directly?
Josh Ferguson
--
Yours,
Zheng
Re: Hadoop JobStatus
Posted by Josh Ferguson <jo...@besquared.net>.
Sure I guess that's true, it doesn't matter to me either way, it'd
just be great to have some kind of delimited logfile that we could get
information from.
Josh
On Dec 8, 2008, at 2:15 PM, Zheng Shao wrote:
> ^A is not printable. That will create confusion to users looking at
> the log right?
>
> Zheng
>
> On Mon, Dec 8, 2008 at 1:46 PM, Josh <jo...@besquared.net> wrote:
> I'm gonna throw my vote in for a structured log format. Users could
> tail it and use whatever queuing or monitoring they wish. It's also
> probably just a 30 minute project for someone already familiar with
> the code. I suggest ^A seperated key=value pairs per log line.8
>
> Josh Ferguson
>
> On Dec 8, 2008, at 1:09 PM, Ashish Thusoo <at...@facebook.com>
> wrote:
>
>> Perhaps we should just have another log4j channel for this instead
>> of debug. The consumers can then just listen on this channel and
>> take appropriate action.
>>
>> Another option would be to implement this using a message queue
>> (publish/subscribe system). We could leverage ActiveMQ or something
>> similar, but that would be a bit more heavyweight but potentially
>> people can develop or advanced monitoring applications around it.
>>
>> Ashish
>>
>> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
>> Sent: Monday, December 08, 2008 12:17 PM
>> To: hive-user@hadoop.apache.org
>> Subject: RE: Hadoop JobStatus
>>
>> The jobid is printed out for non-silent session execution mode.
>>
>>
>> Since there's no structured interface – I had tried to have
>> structured data emitted as key=value in the output stream. The
>> relevant output emitted here is from:
>>
>>
>> console.printInfo("Starting Job = " + rj.getJobID() + ",
>> Tracking URL = " + rj.getTrackingURL());
>>
>>
>> would really welcome a discussion on a better way to get structured
>> data out from the output.
>>
>>
>> From: Josh Ferguson [mailto:josh@besquared.net]
>> Sent: Monday, December 08, 2008 12:08 PM
>> To: hive-user@hadoop.apache.org
>> Subject: Hadoop JobStatus
>>
>>
>> When launching off hive queries using hive -e is there a way to get
>> the job id so that I can just queue them up and go check their
>> statuses later? What's the general pattern for queueing and
>> monitoring without using the libraries directly?
>>
>>
>> Josh Ferguson
>>
>
>
>
> --
> Yours,
> Zheng
Re: Hadoop JobStatus
Posted by Zheng Shao <zs...@gmail.com>.
^A is not printable. That will create confusion to users looking at the log
right?
Zheng
On Mon, Dec 8, 2008 at 1:46 PM, Josh <jo...@besquared.net> wrote:
> I'm gonna throw my vote in for a structured log format. Users could tail it
> and use whatever queuing or monitoring they wish. It's also probably just a
> 30 minute project for someone already familiar with the code. I suggest ^A
> seperated key=value pairs per log line.8
>
> Josh Ferguson
>
> On Dec 8, 2008, at 1:09 PM, Ashish Thusoo <at...@facebook.com> wrote:
>
> Perhaps we should just have another log4j channel for this instead of
> debug. The consumers can then just listen on this channel and take
> appropriate action.
>
> Another option would be to implement this using a message queue
> (publish/subscribe system). We could leverage ActiveMQ or something similar,
> but that would be a bit more heavyweight but potentially people can develop
> or advanced monitoring applications around it.
>
> Ashish
>
> ------------------------------
> *From:* Joydeep Sen Sarma [mailto:jssarma@facebook.com<js...@facebook.com>]
>
> *Sent:* Monday, December 08, 2008 12:17 PM
> *To:* <hi...@hadoop.apache.org>hive-user@hadoop.apache.org
> *Subject:* RE: Hadoop JobStatus
>
> The jobid is printed out for non-silent session execution mode.
>
>
>
> Since there's no structured interface – I had tried to have structured data
> emitted as key=value in the output stream. The relevant output emitted here
> is from:
>
>
>
> console.printInfo("Starting Job = " + rj.getJobID() + ", Tracking URL
> = " + rj.getTrackingURL());
>
>
>
> would really welcome a discussion on a better way to get structured data
> out from the output.
>
>
> ------------------------------
>
> *From:* Josh Ferguson [mailto:josh@besquared.net <jo...@besquared.net>]
> *Sent:* Monday, December 08, 2008 12:08 PM
> *To:* <hi...@hadoop.apache.org>hive-user@hadoop.apache.org
> *Subject:* Hadoop JobStatus
>
>
>
> When launching off hive queries using hive -e is there a way to get the job
> id so that I can just queue them up and go check their statuses later?
> What's the general pattern for queueing and monitoring without using the
> libraries directly?
>
>
>
> Josh Ferguson
>
>
--
Yours,
Zheng
Re: Hadoop JobStatus
Posted by Josh <jo...@besquared.net>.
I'm gonna throw my vote in for a structured log format. Users could
tail it and use whatever queuing or monitoring they wish. It's also
probably just a 30 minute project for someone already familiar with
the code. I suggest ^A seperated key=value pairs per log line.8
Josh Ferguson
On Dec 8, 2008, at 1:09 PM, Ashish Thusoo <at...@facebook.com> wrote:
> Perhaps we should just have another log4j channel for this instead
> of debug. The consumers can then just listen on this channel and
> take appropriate action.
>
> Another option would be to implement this using a message queue
> (publish/subscribe system). We could leverage ActiveMQ or something
> similar, but that would be a bit more heavyweight but potentially
> people can develop or advanced monitoring applications around it.
>
> Ashish
>
> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
> Sent: Monday, December 08, 2008 12:17 PM
> To: hive-user@hadoop.apache.org
> Subject: RE: Hadoop JobStatus
>
> The jobid is printed out for non-silent session execution mode.
>
>
>
> Since there’s no structured interface – I had tried to have
> structured data emitted as key=value in the output stream. The relev
> ant output emitted here is from:
>
>
>
> console.printInfo("Starting Job = " + rj.getJobID() + ",
> Tracking URL = " + rj.getTrackingURL());
>
>
>
> would really welcome a discussion on a better way to get structured
> data out from the output.
>
>
>
> From: Josh Ferguson [mailto:josh@besquared.net]
> Sent: Monday, December 08, 2008 12:08 PM
> To: hive-user@hadoop.apache.org
> Subject: Hadoop JobStatus
>
>
>
> When launching off hive queries using hive -e is there a way to get
> the job id so that I can just queue them up and go check their
> statuses later? What's the general pattern for queueing and
> monitoring without using the libraries directly?
>
>
>
> Josh Ferguson
RE: Hadoop JobStatus
Posted by Ashish Thusoo <at...@facebook.com>.
Perhaps we should just have another log4j channel for this instead of debug. The consumers can then just listen on this channel and take appropriate action.
Another option would be to implement this using a message queue (publish/subscribe system). We could leverage ActiveMQ or something similar, but that would be a bit more heavyweight but potentially people can develop or advanced monitoring applications around it.
Ashish
________________________________
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
Sent: Monday, December 08, 2008 12:17 PM
To: hive-user@hadoop.apache.org
Subject: RE: Hadoop JobStatus
The jobid is printed out for non-silent session execution mode.
Since there's no structured interface - I had tried to have structured data emitted as key=value in the output stream. The relevant output emitted here is from:
console.printInfo("Starting Job = " + rj.getJobID() + ", Tracking URL = " + rj.getTrackingURL());
would really welcome a discussion on a better way to get structured data out from the output.
________________________________
From: Josh Ferguson [mailto:josh@besquared.net]
Sent: Monday, December 08, 2008 12:08 PM
To: hive-user@hadoop.apache.org
Subject: Hadoop JobStatus
When launching off hive queries using hive -e is there a way to get the job id so that I can just queue them up and go check their statuses later? What's the general pattern for queueing and monitoring without using the libraries directly?
Josh Ferguson
RE: Hadoop JobStatus
Posted by Joydeep Sen Sarma <js...@facebook.com>.
The jobid is printed out for non-silent session execution mode.
Since there's no structured interface - I had tried to have structured data emitted as key=value in the output stream. The relevant output emitted here is from:
console.printInfo("Starting Job = " + rj.getJobID() + ", Tracking URL = " + rj.getTrackingURL());
would really welcome a discussion on a better way to get structured data out from the output.
________________________________
From: Josh Ferguson [mailto:josh@besquared.net]
Sent: Monday, December 08, 2008 12:08 PM
To: hive-user@hadoop.apache.org
Subject: Hadoop JobStatus
When launching off hive queries using hive -e is there a way to get the job id so that I can just queue them up and go check their statuses later? What's the general pattern for queueing and monitoring without using the libraries directly?
Josh Ferguson