You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Eduardo Afonso Ferreira <ea...@yahoo.com> on 2012/08/30 19:31:24 UTC
Capturing Pig action output
Hi there,
I have a pig that runs periodically by oozie via coordinator with a set frequency.
I wanted to capture the Pig script output because I need to look at some information on the results to keep track of several things.
I know I can look at the output by doing a whole bunch of clicks starting at the oozie web console as follows:
- Open oozie web console (ex.: http://localhost:11000/oozie/)
- Find and click the specific job under "Workflow Jobs"
- Select (click) the pig action in the window that pops up
- Click the magnifying glass icon on the "Console URL" field
- Click the Map of the launcher job
- Click the task ID
- Click All under "Task Logs"
My question is how can I know the exact name and location of that log file in HDFS so I can programmaticaly retrieve the file from HDFS and parse and look for what I need?
Is this something I can determine ahead of time, like pass a parameter/argument to the action/pig so that it will store the log where I want with the file name I want?
Thanks in advance for your help.
Eduardo.
Re: Capturing Pig action output
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Got it, thanks!
eferreira@eferreira-tbs-desktop:/opt/oozie$ oozie job -oozie http://localhost:11000/oozie -info 0000027-120906115500668-oozie-oozi-W@pig-node
ID : 0000027-120906115500668-oozie-oozi-W@pig-node
------------------------------------------------------------------------------------------------------------------------------------
Console URL : http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071502_69901
Error Code : -
Error Message : -
External ID : job_201208071502_69901
External Status : SUCCEEDED
Name : pig-node
Retries : 0
Tracker URI : aspendevhdp1.cnn.vgtf.net:54311
Type : pig
Started : 2012-09-06 18:01
Status : OK
Ended : 2012-09-06 18:50
------------------------------------------------------------------------------------------------------------------------------------
eferreira@eferreira-tbs-desktop:/opt/oozie$ oozie job -oozie http://localhost:11000/oozie -info 0000027-120906115500668-oozie-oozi-W@pig-node -verbose
ID : 0000027-120906115500668-oozie-oozi-W@pig-node
------------------------------------------------------------------------------------------------------------------------------------
Console URL : http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071502_69901
Error Code : -
Error Message : -
External ID : job_201208071502_69901
External Status : SUCCEEDED
Name : pig-node
Retries : 0
Tracker URI : aspendevhdp1.cnn.vgtf.net:54311
Type : pig
Started : 2012-09-06 18:01
Status : OK
Ended : 2012-09-06 18:50
External Stats :
{"JOB_GRAPH":"job_201208071502_69902,job_201208071502_69904","ACTION_TYPE":"PIG","PIG_VERSION":"0.11.0-SNAPSHOT","FEATURES":"GROUP_BY,FILTER","ERROR_MESSAGE":null,"NUMBER_JOBS":"2","RECORD_WRITTEN":"0","job_201208071502_69902":{"MAP_INPUT_RECORDS":"0","MIN_REDUCE_TIME":"113396","MULTI_STORE_COUNTERS":{},"MAX_REDUCE_TIME":"132349","NUMBER_REDUCES":"4","ERROR_MESSAGE":null,"RECORD_WRITTEN":"0","HDFS_BYTES_WRITTEN":"616","JOB_ID":"job_201208071502_69902","REDUCE_INPUT_RECORDS":"0","AVG_REDUCE_TIME":"121718","MAX_MAP_TIME":"113817","BYTES_WRITTEN":"0","Alias":"1-9,A,B,esi,filt1,keys,slots","REDUCE_OUTPUT_RECORDS":"0","SMMS_SPILL_COUNT":"0","PROACTIVE_SPILL_COUNT_RECS":"0","PROACTIVE_SPILL_COUNT_OBJECTS":"0","HADOOP_COUNTERS":null,"MIN_MAP_TIME":"927","MAP_OUTPUT_RECORDS":"0","AVG_MAP_TIME":"34318","FEATURE":"GROUP_BY,COMBINER","NUMBER_MAPS":"14"},"job_201208071502_69904":{"MAP_INPUT_RECORDS":"0","MIN_REDUCE_TIME":"27445","MULTI_STORE_COUNTERS":{},"MAX_REDU
CE_TIME":"45937","NUMBER_REDUCES":"4","ERROR_MESSAGE":null,"RECORD_WRITTEN":"0","HDFS_BYTES_WRITTEN":"0","JOB_ID":"job_201208071502_69904","REDUCE_INPUT_RECORDS":"0","AVG_REDUCE_TIME":"34823","MAX_MAP_TIME":"18786","BYTES_WRITTEN":"0","Alias":"1-10,C,E","REDUCE_OUTPUT_RECORDS":"0","SMMS_SPILL_COUNT":"0","PROACTIVE_SPILL_COUNT_RECS":"0","PROACTIVE_SPILL_COUNT_OBJECTS":"0","HADOOP_COUNTERS":null,"MIN_MAP_TIME":"1860","MAP_OUTPUT_RECORDS":"0","AVG_MAP_TIME":"13124","FEATURE":"GROUP_BY,DISTINCT","NUMBER_MAPS":"4"},"BYTES_WRITTEN":"0","HADOOP_VERSION":"0.20.2-cdh3u3","SCRIPT_ID":"6b440ae7-0ed7-4563-80a3-9d44d30e4340","PROACTIVE_SPILL_COUNT_RECORDS":"0","PROACTIVE_SPILL_COUNT_OBJECTS":"0","RETURN_CODE":"0","ERROR_CODE":"-1","SMM_SPILL_COUNT":"0","DURATION":"243348"}
External ChildIDs : job_201208071502_69902,job_201208071502_69904
------------------------------------------------------------------------------------------------------------------------------------
________________________________
From: Virag Kothari <vi...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, September 6, 2012 1:57 PM
Subject: Re: Capturing Pig action output
Also, the -info -verbose should be on pig action and not workflow job.
On 9/6/12 10:38 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>I forgot to mention that you need to install oozie share library on hdfs.
>The json-simple.jar is bundled with that package.
>Look at 'Oozie-sharelib installation' under
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/DG_QuickStart
>.
>html
>
>
>You should be able to view the stats and external Ids with -info -verbose.
>Please make sure that you are using 3.2.0 client. Command to check client
>version is 'oozie version'
>
>Thanks,
>Virag
>
>
>
>
>On 9/6/12 10:27 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hey,
>>
>>
>>I'm still interested in learning if a pre-packaged 3.2 version is
>>available out there that I can install, but I was able to move a little
>>more by adding another jar to my app, i.e. json-simple-1.1.1.jar which
>>solved the NoClassDefFoundError I experienced.
>>
>>Now I see the stats field on the oozie database (WF_ACTIONS.stats) filled
>>with a JSON of the PigStats that I'm interested in. But I still can't see
>>it when I run -info with -verbose. Am I missing something?
>>
>>Thanks.
>>Eduardo.
>>
>>
>>
>>________________________________
>> From: Eduardo Afonso Ferreira <ea...@yahoo.com>
>>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
>>Sent: Thursday, September 6, 2012 12:14 PM
>>Subject: Re: Capturing Pig action output
>>
>>Hey, Virag,
>>
>>I built and installed Oozie 3.2 from
>>http://incubator.apache.org/oozie/Downloads.html.
>>I set the property oozie.action.external.stats.write to true on my WF and
>>deployed/submitted/etc.
>>But I still don't see PigStats when I do the -info request (ex. below)
>>and I see exceptions related to org.json.simple.JSONObject
>>(NoClassDefFoundError). Maybe a build problem.
>>
>>What would be the best way of getting version 3.2 up and running? Any
>>package out there already built that we could download and install? I
>>mean, without need to build/package and look for solving all sorts of
>>dependencies.
>>
>>
>>eferreira@eferreira-tbs-desktop:~/projects/aspen-core/oozie/apps$ oozie
>>job -oozie http://localhost:11000/oozie -info
>>0000197-120905170442968-oozie-oozi-W -verbose
>>Job ID : 0000197-120905170442968-oozie-oozi-W
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>Workflow Name : video_play_counts-wf
>>App Path :
>>hdfs://aspendevhdp1.cnn.vgtf.net:54310/user/eferreira/oozie/apps/video_pl
>>a
>>y_counts
>>Status : RUNNING
>>Run : 0
>>User : eferreira
>>Group : -
>>Created : 2012-09-06 14:53
>>Started : 2012-09-06 14:53
>>Last Modified : 2012-09-06 14:53
>>Ended : -
>>CoordAction ID: 0000196-120905170442968-oozie-oozi-C@1
>>
>>Actions
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>ID Console URL Error Code Error Message External ID
>>External Status Name Retries Tracker URI Type Started
>>Status Ended
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>0000197-120905170442968-oozie-oozi-W@pig-node
>>http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071
>>5
>>02_69799 - - job_201208071502_69799 RUNNING pig-node 0
>> aspendevhdp1.cnn.vgtf.net:54311 pig 2012-09-06 14:53 RUNNING
>> -
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>
>>
>>
>>
>>
>>________________________________
>>From: Virag Kothari <vi...@yahoo-inc.com>
>>To: "oozie-users@incubator.apache.org"
>><oo...@incubator.apache.org>; Eduardo Afonso Ferreira
>><ea...@yahoo.com>
>>Sent: Thursday, August 30, 2012 2:59 PM
>>Subject: Re: Capturing Pig action output
>>
>>Hi,
>>
>>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>>be accessed through the API or EL function.
>>
>>First, the following should be set in wf configuration. This will store
>>the Pig/MR related statistics in the DB.
>><property>
>> <name>oozie.action.external.stats.write</name>
>> <value>true</value>
>> </property>
>>
>>Then, the stats and jobIds can be accessed using the verbose API
>>oozie job -info <jobId> -verbose
>>
>>Also, the hadoop job Id's can be retrieved for a Pig action through
>>El-function
>>
>>wf:actionData(<pig-action-name>)["hadoopJobs"]
>>
>>
>>Detailed docs at
>>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunc
>>t
>>i
>>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>>
>>Thanks,
>>Virag
>>
>>
>>
>>
>>
>>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com>
>>wrote:
>>
>>>Hi there,
>>>
>>>I have a pig that runs periodically by oozie via coordinator with a set
>>>frequency.
>>>I wanted to capture the Pig script output because I need to look at some
>>>information on the results to keep track of several things.
>>>I know I can look at the output by doing a whole bunch of clicks
>>>starting
>>>at the oozie web console as follows:
>>>
>>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>>- Find and click the specific job under "Workflow Jobs"
>>>- Select (click) the pig action in the window that pops up
>>>- Click the magnifying glass icon on the "Console URL" field
>>>- Click the Map of the launcher job
>>>- Click the task ID
>>>- Click All under "Task Logs"
>>>
>>>My question is how can I know the exact name and location of that log
>>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>>parse and look for what I need?
>>>
>>>Is this something I can determine ahead of time, like pass a
>>>parameter/argument to the action/pig so that it will store the log where
>>>I want with the file name I want?
>>>
>>>Thanks in advance for your help.
>>>Eduardo.
>
Re: Capturing Pig action output
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Also, the -info -verbose should be on pig action and not workflow job.
On 9/6/12 10:38 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>I forgot to mention that you need to install oozie share library on hdfs.
>The json-simple.jar is bundled with that package.
>Look at 'Oozie-sharelib installation' under
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/DG_QuickStart
>.
>html
>
>
>You should be able to view the stats and external Ids with -info -verbose.
>Please make sure that you are using 3.2.0 client. Command to check client
>version is 'oozie version'
>
>Thanks,
>Virag
>
>
>
>
>On 9/6/12 10:27 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hey,
>>
>>
>>I'm still interested in learning if a pre-packaged 3.2 version is
>>available out there that I can install, but I was able to move a little
>>more by adding another jar to my app, i.e. json-simple-1.1.1.jar which
>>solved the NoClassDefFoundError I experienced.
>>
>>Now I see the stats field on the oozie database (WF_ACTIONS.stats) filled
>>with a JSON of the PigStats that I'm interested in. But I still can't see
>>it when I run -info with -verbose. Am I missing something?
>>
>>Thanks.
>>Eduardo.
>>
>>
>>
>>________________________________
>> From: Eduardo Afonso Ferreira <ea...@yahoo.com>
>>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
>>Sent: Thursday, September 6, 2012 12:14 PM
>>Subject: Re: Capturing Pig action output
>>
>>Hey, Virag,
>>
>>I built and installed Oozie 3.2 from
>>http://incubator.apache.org/oozie/Downloads.html.
>>I set the property oozie.action.external.stats.write to true on my WF and
>>deployed/submitted/etc.
>>But I still don't see PigStats when I do the -info request (ex. below)
>>and I see exceptions related to org.json.simple.JSONObject
>>(NoClassDefFoundError). Maybe a build problem.
>>
>>What would be the best way of getting version 3.2 up and running? Any
>>package out there already built that we could download and install? I
>>mean, without need to build/package and look for solving all sorts of
>>dependencies.
>>
>>
>>eferreira@eferreira-tbs-desktop:~/projects/aspen-core/oozie/apps$ oozie
>>job -oozie http://localhost:11000/oozie -info
>>0000197-120905170442968-oozie-oozi-W -verbose
>>Job ID : 0000197-120905170442968-oozie-oozi-W
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>Workflow Name : video_play_counts-wf
>>App Path :
>>hdfs://aspendevhdp1.cnn.vgtf.net:54310/user/eferreira/oozie/apps/video_pl
>>a
>>y_counts
>>Status : RUNNING
>>Run : 0
>>User : eferreira
>>Group : -
>>Created : 2012-09-06 14:53
>>Started : 2012-09-06 14:53
>>Last Modified : 2012-09-06 14:53
>>Ended : -
>>CoordAction ID: 0000196-120905170442968-oozie-oozi-C@1
>>
>>Actions
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>ID Console URL Error Code Error Message External ID
>>External Status Name Retries Tracker URI Type Started
>>Status Ended
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>0000197-120905170442968-oozie-oozi-W@pig-node
>>http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071
>>5
>>02_69799 - - job_201208071502_69799 RUNNING pig-node 0
>> aspendevhdp1.cnn.vgtf.net:54311 pig 2012-09-06 14:53 RUNNING
>> -
>>-------------------------------------------------------------------------
>>-
>>----------------------------------------------------------
>>
>>
>>
>>
>>
>>________________________________
>>From: Virag Kothari <vi...@yahoo-inc.com>
>>To: "oozie-users@incubator.apache.org"
>><oo...@incubator.apache.org>; Eduardo Afonso Ferreira
>><ea...@yahoo.com>
>>Sent: Thursday, August 30, 2012 2:59 PM
>>Subject: Re: Capturing Pig action output
>>
>>Hi,
>>
>>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>>be accessed through the API or EL function.
>>
>>First, the following should be set in wf configuration. This will store
>>the Pig/MR related statistics in the DB.
>><property>
>> <name>oozie.action.external.stats.write</name>
>> <value>true</value>
>> </property>
>>
>>Then, the stats and jobIds can be accessed using the verbose API
>>oozie job -info <jobId> -verbose
>>
>>Also, the hadoop job Id's can be retrieved for a Pig action through
>>El-function
>>
>>wf:actionData(<pig-action-name>)["hadoopJobs"]
>>
>>
>>Detailed docs at
>>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunc
>>t
>>i
>>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>>
>>Thanks,
>>Virag
>>
>>
>>
>>
>>
>>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com>
>>wrote:
>>
>>>Hi there,
>>>
>>>I have a pig that runs periodically by oozie via coordinator with a set
>>>frequency.
>>>I wanted to capture the Pig script output because I need to look at some
>>>information on the results to keep track of several things.
>>>I know I can look at the output by doing a whole bunch of clicks
>>>starting
>>>at the oozie web console as follows:
>>>
>>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>>- Find and click the specific job under "Workflow Jobs"
>>>- Select (click) the pig action in the window that pops up
>>>- Click the magnifying glass icon on the "Console URL" field
>>>- Click the Map of the launcher job
>>>- Click the task ID
>>>- Click All under "Task Logs"
>>>
>>>My question is how can I know the exact name and location of that log
>>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>>parse and look for what I need?
>>>
>>>Is this something I can determine ahead of time, like pass a
>>>parameter/argument to the action/pig so that it will store the log where
>>>I want with the file name I want?
>>>
>>>Thanks in advance for your help.
>>>Eduardo.
>
Re: Capturing Pig action output
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Hi,
I forgot to mention that you need to install oozie share library on hdfs.
The json-simple.jar is bundled with that package.
Look at 'Oozie-sharelib installation' under
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/DG_QuickStart.
html
You should be able to view the stats and external Ids with -info -verbose.
Please make sure that you are using 3.2.0 client. Command to check client
version is 'oozie version'
Thanks,
Virag
On 9/6/12 10:27 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>Hey,
>
>
>I'm still interested in learning if a pre-packaged 3.2 version is
>available out there that I can install, but I was able to move a little
>more by adding another jar to my app, i.e. json-simple-1.1.1.jar which
>solved the NoClassDefFoundError I experienced.
>
>Now I see the stats field on the oozie database (WF_ACTIONS.stats) filled
>with a JSON of the PigStats that I'm interested in. But I still can't see
>it when I run -info with -verbose. Am I missing something?
>
>Thanks.
>Eduardo.
>
>
>
>________________________________
> From: Eduardo Afonso Ferreira <ea...@yahoo.com>
>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
>Sent: Thursday, September 6, 2012 12:14 PM
>Subject: Re: Capturing Pig action output
>
>Hey, Virag,
>
>I built and installed Oozie 3.2 from
>http://incubator.apache.org/oozie/Downloads.html.
>I set the property oozie.action.external.stats.write to true on my WF and
>deployed/submitted/etc.
>But I still don't see PigStats when I do the -info request (ex. below)
>and I see exceptions related to org.json.simple.JSONObject
>(NoClassDefFoundError). Maybe a build problem.
>
>What would be the best way of getting version 3.2 up and running? Any
>package out there already built that we could download and install? I
>mean, without need to build/package and look for solving all sorts of
>dependencies.
>
>
>eferreira@eferreira-tbs-desktop:~/projects/aspen-core/oozie/apps$ oozie
>job -oozie http://localhost:11000/oozie -info
>0000197-120905170442968-oozie-oozi-W -verbose
>Job ID : 0000197-120905170442968-oozie-oozi-W
>--------------------------------------------------------------------------
>----------------------------------------------------------
>Workflow Name : video_play_counts-wf
>App Path :
>hdfs://aspendevhdp1.cnn.vgtf.net:54310/user/eferreira/oozie/apps/video_pla
>y_counts
>Status : RUNNING
>Run : 0
>User : eferreira
>Group : -
>Created : 2012-09-06 14:53
>Started : 2012-09-06 14:53
>Last Modified : 2012-09-06 14:53
>Ended : -
>CoordAction ID: 0000196-120905170442968-oozie-oozi-C@1
>
>Actions
>--------------------------------------------------------------------------
>----------------------------------------------------------
>ID Console URL Error Code Error Message External ID
>External Status Name Retries Tracker URI Type Started
>Status Ended
>--------------------------------------------------------------------------
>----------------------------------------------------------
>0000197-120905170442968-oozie-oozi-W@pig-node
>http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_2012080715
>02_69799 - - job_201208071502_69799 RUNNING pig-node 0
> aspendevhdp1.cnn.vgtf.net:54311 pig 2012-09-06 14:53 RUNNING
> -
>--------------------------------------------------------------------------
>----------------------------------------------------------
>
>
>
>
>
>________________________________
>From: Virag Kothari <vi...@yahoo-inc.com>
>To: "oozie-users@incubator.apache.org"
><oo...@incubator.apache.org>; Eduardo Afonso Ferreira
><ea...@yahoo.com>
>Sent: Thursday, August 30, 2012 2:59 PM
>Subject: Re: Capturing Pig action output
>
>Hi,
>
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>be accessed through the API or EL function.
>
>First, the following should be set in wf configuration. This will store
>the Pig/MR related statistics in the DB.
><property>
> <name>oozie.action.external.stats.write</name>
> <value>true</value>
> </property>
>
>Then, the stats and jobIds can be accessed using the verbose API
>oozie job -info <jobId> -verbose
>
>Also, the hadoop job Id's can be retrieved for a Pig action through
>El-function
>
>wf:actionData(<pig-action-name>)["hadoopJobs"]
>
>
>Detailed docs at
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunct
>i
>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>
>Thanks,
>Virag
>
>
>
>
>
>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hi there,
>>
>>I have a pig that runs periodically by oozie via coordinator with a set
>>frequency.
>>I wanted to capture the Pig script output because I need to look at some
>>information on the results to keep track of several things.
>>I know I can look at the output by doing a whole bunch of clicks starting
>>at the oozie web console as follows:
>>
>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>- Find and click the specific job under "Workflow Jobs"
>>- Select (click) the pig action in the window that pops up
>>- Click the magnifying glass icon on the "Console URL" field
>>- Click the Map of the launcher job
>>- Click the task ID
>>- Click All under "Task Logs"
>>
>>My question is how can I know the exact name and location of that log
>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>parse and look for what I need?
>>
>>Is this something I can determine ahead of time, like pass a
>>parameter/argument to the action/pig so that it will store the log where
>>I want with the file name I want?
>>
>>Thanks in advance for your help.
>>Eduardo.
Re: Capturing Pig action output
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey,
I'm still interested in learning if a pre-packaged 3.2 version is available out there that I can install, but I was able to move a little more by adding another jar to my app, i.e. json-simple-1.1.1.jar which solved the NoClassDefFoundError I experienced.
Now I see the stats field on the oozie database (WF_ACTIONS.stats) filled with a JSON of the PigStats that I'm interested in. But I still can't see it when I run -info with -verbose. Am I missing something?
Thanks.
Eduardo.
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Thursday, September 6, 2012 12:14 PM
Subject: Re: Capturing Pig action output
Hey, Virag,
I built and installed Oozie 3.2 from http://incubator.apache.org/oozie/Downloads.html.
I set the property oozie.action.external.stats.write to true on my WF and deployed/submitted/etc.
But I still don't see PigStats when I do the -info request (ex. below) and I see exceptions related to org.json.simple.JSONObject (NoClassDefFoundError). Maybe a build problem.
What would be the best way of getting version 3.2 up and running? Any package out there already built that we could download and install? I mean, without need to build/package and look for solving all sorts of dependencies.
eferreira@eferreira-tbs-desktop:~/projects/aspen-core/oozie/apps$ oozie job -oozie http://localhost:11000/oozie -info 0000197-120905170442968-oozie-oozi-W -verbose
Job ID : 0000197-120905170442968-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : video_play_counts-wf
App Path : hdfs://aspendevhdp1.cnn.vgtf.net:54310/user/eferreira/oozie/apps/video_play_counts
Status : RUNNING
Run : 0
User : eferreira
Group : -
Created : 2012-09-06 14:53
Started : 2012-09-06 14:53
Last Modified : 2012-09-06 14:53
Ended : -
CoordAction ID: 0000196-120905170442968-oozie-oozi-C@1
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Console URL Error Code Error Message External ID External Status Name Retries Tracker URI Type Started Status Ended
------------------------------------------------------------------------------------------------------------------------------------
0000197-120905170442968-oozie-oozi-W@pig-node http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071502_69799 - - job_201208071502_69799 RUNNING pig-node 0 aspendevhdp1.cnn.vgtf.net:54311 pig 2012-09-06 14:53 RUNNING -
------------------------------------------------------------------------------------------------------------------------------------
________________________________
From: Virag Kothari <vi...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, August 30, 2012 2:59 PM
Subject: Re: Capturing Pig action output
Hi,
From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
be accessed through the API or EL function.
First, the following should be set in wf configuration. This will store
the Pig/MR related statistics in the DB.
<property>
<name>oozie.action.external.stats.write</name>
<value>true</value>
</property>
Then, the stats and jobIds can be accessed using the verbose API
oozie job -info <jobId> -verbose
Also, the hadoop job Id's can be retrieved for a Pig action through
El-function
wf:actionData(<pig-action-name>)["hadoopJobs"]
Detailed docs at
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
Thanks,
Virag
On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>Hi there,
>
>I have a pig that runs periodically by oozie via coordinator with a set
>frequency.
>I wanted to capture the Pig script output because I need to look at some
>information on the results to keep track of several things.
>I know I can look at the output by doing a whole bunch of clicks starting
>at the oozie web console as follows:
>
>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>- Find and click the specific job under "Workflow Jobs"
>- Select (click) the pig action in the window that pops up
>- Click the magnifying glass icon on the "Console URL" field
>- Click the Map of the launcher job
>- Click the task ID
>- Click All under "Task Logs"
>
>My question is how can I know the exact name and location of that log
>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>parse and look for what I need?
>
>Is this something I can determine ahead of time, like pass a
>parameter/argument to the action/pig so that it will store the log where
>I want with the file name I want?
>
>Thanks in advance for your help.
>Eduardo.
Re: Capturing Pig action output
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey, Virag,
I built and installed Oozie 3.2 from http://incubator.apache.org/oozie/Downloads.html.
I set the property oozie.action.external.stats.write to true on my WF and deployed/submitted/etc.
But I still don't see PigStats when I do the -info request (ex. below) and I see exceptions related to org.json.simple.JSONObject (NoClassDefFoundError). Maybe a build problem.
What would be the best way of getting version 3.2 up and running? Any package out there already built that we could download and install? I mean, without need to build/package and look for solving all sorts of dependencies.
eferreira@eferreira-tbs-desktop:~/projects/aspen-core/oozie/apps$ oozie job -oozie http://localhost:11000/oozie -info 0000197-120905170442968-oozie-oozi-W -verbose
Job ID : 0000197-120905170442968-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : video_play_counts-wf
App Path : hdfs://aspendevhdp1.cnn.vgtf.net:54310/user/eferreira/oozie/apps/video_play_counts
Status : RUNNING
Run : 0
User : eferreira
Group : -
Created : 2012-09-06 14:53
Started : 2012-09-06 14:53
Last Modified : 2012-09-06 14:53
Ended : -
CoordAction ID: 0000196-120905170442968-oozie-oozi-C@1
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Console URL Error Code Error Message External ID External Status Name Retries Tracker URI Type Started Status Ended
------------------------------------------------------------------------------------------------------------------------------------
0000197-120905170442968-oozie-oozi-W@pig-node http://aspendevhdp1.cnn.vgtf.net:50030/jobdetails.jsp?jobid=job_201208071502_69799 - - job_201208071502_69799 RUNNING pig-node 0 aspendevhdp1.cnn.vgtf.net:54311 pig 2012-09-06 14:53 RUNNING -
------------------------------------------------------------------------------------------------------------------------------------
________________________________
From: Virag Kothari <vi...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, August 30, 2012 2:59 PM
Subject: Re: Capturing Pig action output
Hi,
From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
be accessed through the API or EL function.
First, the following should be set in wf configuration. This will store
the Pig/MR related statistics in the DB.
<property>
<name>oozie.action.external.stats.write</name>
<value>true</value>
</property>
Then, the stats and jobIds can be accessed using the verbose API
oozie job -info <jobId> -verbose
Also, the hadoop job Id's can be retrieved for a Pig action through
El-function
wf:actionData(<pig-action-name>)["hadoopJobs"]
Detailed docs at
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
Thanks,
Virag
On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>Hi there,
>
>I have a pig that runs periodically by oozie via coordinator with a set
>frequency.
>I wanted to capture the Pig script output because I need to look at some
>information on the results to keep track of several things.
>I know I can look at the output by doing a whole bunch of clicks starting
>at the oozie web console as follows:
>
>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>- Find and click the specific job under "Workflow Jobs"
>- Select (click) the pig action in the window that pops up
>- Click the magnifying glass icon on the "Console URL" field
>- Click the Map of the launcher job
>- Click the task ID
>- Click All under "Task Logs"
>
>My question is how can I know the exact name and location of that log
>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>parse and look for what I need?
>
>Is this something I can determine ahead of time, like pass a
>parameter/argument to the action/pig so that it will store the log where
>I want with the file name I want?
>
>Thanks in advance for your help.
>Eduardo.
Re: Capturing Pig action output
Posted by Ryota Egashira <eg...@yahoo-inc.com>.
Hi, Eduardo
I didn't try this myself, but how about using java action, where you write
custom code which starts pig runner, grab std out and persist it into HDFS
for later use.
You can use distributed-cache to provide pig.jar to each compute node, so
no need to install pig.jar in whole cluster.
Thanks
Ryota
On 9/5/12 7:11 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>Hey, Kamal,
>
>That would require me to install Pig on the Hadoop/HBase cluster.
>I believe running Pig as an action seems to make better use of resources.
>In this case, Oozie runs the pig script using PigRunner.run().
>
>Anyway, I did not figure out yet how to get the output generated by
>PigRunner.run().
>
>
>I'm thinking if there's a way of having another action in my workflow
>that will retrieve that output from the FS (if I can determine how to do
>that) or if there's any EL function or variable that would retrieve that
>for me.
>
>So far, the only possibility I see to get that output is by writing a
>Java action (or whatever) that receives the launcher hadoop job ID (I
>still need to figure out how to get that hadoop job ID) and use the
>hadoop API to get the output.
>
>If somebody out there knows how to do it, would you mind sharing how you
>did it?
>
>What I really would like to see is the existence of an EL function that
>returns that output, or a way of redirecting that output to a known
>location on HDFS. Or better yet, if there were functions, variables or
>whatever that had all the information I see on that output, such as
>follows:
>
>- HadoopVersion
>- PigVersion
>- UserId,
>- StartedAt
>- FinishedAt,
>- Features
>
>- List of hadoop jobs with specific stats (JobId, Maps, Reduces,
>MaxMapTime, MinMapTime, AvgMapTime, MaxReduceTime, MinReduceTime,
>AvgReduceTime, Alias, Feature Outputs)
>- Input Information (Records Read and Source, ex. hbase table name)
>
>- Output Information (Records Written and Destination, ex. hbase table
>name)
>- Indication if it was Success or Failure
>- etc
>
>
>If I figure out how to get that output file, I can parse it and retrieve
>all the information.
>
>
>Thank you.
>Eduardo.
>
>
>
>________________________________
> From: Kamal Hakim <Ka...@aexp.com>
>To: "oozie-users@incubator.apache.org"
><oo...@incubator.apache.org>; Eduardo Afonso Ferreira
><ea...@yahoo.com>
>Sent: Tuesday, September 4, 2012 5:40 PM
>Subject: RE: Capturing Pig action output
>
>Hi Eduardo ,
>
>You could run the Pig command as an in-line script command and capture
>the output into an output file.
>
>Example:
>
>pig -f pigscript.pig > pig_log.log
>
>
>
>
>Kamal Hakim
>American Express
>Big Data Architecture
>Phone: 602-537-6819
>________________________________________
>From: Eduardo Afonso Ferreira [eafonsof@yahoo.com]
>Sent: Tuesday, September 04, 2012 02:24 PM
>To: oozie-users@incubator.apache.org
>Subject: Re: Capturing Pig action output
>
>Hey, thanks for the response.
>
>What I need is not the log file that's created by Pig when there's an
>error, you know, the one that lists stack trace and things like that.
>
>I need the output that Pig sends to stdout/stderr. This is the output
>that includes information about hadoop jobs created, started/finished
>timestamps, success/failure, etc.
>
>Here's an example of part of that output:
>
>________________________________
>
>HadoopVersion PigVersion UserId StartedAt FinishedAt
>Features
>0.20.2-cdh3u3 0.11.0-SNAPSHOT mapred 2012-09-04 17:00:56
>2012-09-04 17:06:26 GROUP_BY,DISTINCT,FILTER
>
>Success!
>
>Job Stats (time in seconds):
>JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime
>MaxReduceTime MinReduceTime AvgReduceTime Alias Feature
>Outputs
>job_201206281058_812355 14 4 20 11 16 250 235 246
> B,esi,filt1,keys,slots DISTINCT
>job_201206281058_812414 4 4 7 6 7 26 22 24
>1-4,C,E GROUP_BY,DISTINCT hbase://active_video_plays,
>
>Input(s):
>Successfully read 144190 records (5079 bytes) from:
>"hbase://events_sessions"
>
>Output(s):
>Successfully stored 20835 records in: "hbase://active_video_plays"
>
>Counters:
>Total records written : 20835
>Total bytes written : 0
>Spillable Memory Manager spill count : 0
>Total bags proactively spilled: 0
>Total records proactively spilled: 0
>________________________________
>
>
>Virag,
>
>We're currently using Oozie 2.3.2 (2.3.2-cdh3u3)and I guess the wf
>configuration you mentioned is not available on this version.
>
>
>Eduardo.
>
>
>
>
>________________________________
>From: Mona Chitnis <ch...@yahoo-inc.com>
>To: "oozie-users@incubator.apache.org"
><oo...@incubator.apache.org>; Eduardo Afonso Ferreira
><ea...@yahoo.com>
>Sent: Thursday, August 30, 2012 4:01 PM
>Subject: Re: Capturing Pig action output
>
>Hi Eduardo,
>
>The log file where the Oozie pig action's output is written to, is a local
>file in the current working directory of the map task, and not on HDFS.
>Also, passing a custom path using "pig -logfile <HDFS_PATH>" is not
>allowed. Can you try with passing another argument at the end of your
>arguments list to your pig action as a redirection to some file of choice?
>
>E.g. Trying to recreate something like the following. Note the redirection
>to 'myfile.txt' at the end
>pig -param $xyz=1000 myscript.pig &>myfile.txt
>
>I haven't tried this out myself but it will be helpful to find out.
>Otherwise, Virag's suggestion can help you access specific stats related
>information
>
>Regards,
>
>--
>Mona Chitnis
>
>
>
>
>On 8/30/12 11:59 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>
>>Hi,
>>
>>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>>be accessed through the API or EL function.
>>
>>First, the following should be set in wf configuration. This will store
>>the Pig/MR related statistics in the DB.
>><property>
>> <name>oozie.action.external.stats.write</name>
>> <value>true</value>
>> </property>
>>
>>Then, the stats and jobIds can be accessed using the verbose API
>>oozie job -info <jobId> -verbose
>>
>>Also, the hadoop job Id's can be retrieved for a Pig action through
>>El-function
>>
>>wf:actionData(<pig-action-name>)["hadoopJobs"]
>>
>>
>>Detailed docs at
>>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunc
>>t
>>i
>>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>>
>>Thanks,
>>Virag
>>
>>
>>
>>
>>
>>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com>
>>wrote:
>>
>>>Hi there,
>>>
>>>I have a pig that runs periodically by oozie via coordinator with a set
>>>frequency.
>>>I wanted to capture the Pig script output because I need to look at some
>>>information on the results to keep track of several things.
>>>I know I can look at the output by doing a whole bunch of clicks
>>>starting
>>>at the oozie web console as follows:
>>>
>>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>>- Find and click the specific job under "Workflow Jobs"
>>>- Select (click) the pig action in the window that pops up
>>>- Click the magnifying glass icon on the "Console URL" field
>>>- Click the Map of the launcher job
>>>- Click the task ID
>>>- Click All under "Task Logs"
>>>
>>>My question is how can I know the exact name and location of that log
>>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>>parse and look for what I need?
>>>
>>>Is this something I can determine ahead of time, like pass a
>>>parameter/argument to the action/pig so that it will store the log where
>>>I want with the file name I want?
>>>
>>>Thanks in advance for your help.
>>>Eduardo.
>>
>American Express made the following annotations on Tue Sep 04 2012
>14:40:20
>
>**************************************************************************
>****
>
>"This message and any attachments are solely for the intended recipient
>and may contain confidential or privileged information. If you are not
>the intended recipient, any disclosure, copying, use, or distribution of
>the information included in this message and any attachments is
>prohibited. If you have received this communication in error, please
>notify us by reply e-mail and immediately and permanently delete this
>message and any attachments. Thank you."
>
>American Express a ajouté le commentaire suivant le Tue Sep 04 2012
>14:40:20
>
>Ce courrier et toute pièce jointe qu'il contient sont réservés au seul
>destinataire indiqué et peuvent renfermer des renseignements
>confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu,
>toute divulgation, duplication, utilisation ou distribution du courrier
>ou de toute pièce jointe est interdite. Si vous avez reçu cette
>communication par erreur, veuillez nous en aviser par courrier et
>détruire immédiatement le courrier et les pièces jointes. Merci.
>
>**************************************************************************
>****
>--------------------------------------------------------------------------
>-----
Re: Capturing Pig action output
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey, Kamal,
That would require me to install Pig on the Hadoop/HBase cluster.
I believe running Pig as an action seems to make better use of resources.
In this case, Oozie runs the pig script using PigRunner.run().
Anyway, I did not figure out yet how to get the output generated by PigRunner.run().
I'm thinking if there's a way of having another action in my workflow that will retrieve that output from the FS (if I can determine how to do that) or if there's any EL function or variable that would retrieve that for me.
So far, the only possibility I see to get that output is by writing a Java action (or whatever) that receives the launcher hadoop job ID (I still need to figure out how to get that hadoop job ID) and use the hadoop API to get the output.
If somebody out there knows how to do it, would you mind sharing how you did it?
What I really would like to see is the existence of an EL function that returns that output, or a way of redirecting that output to a known location on HDFS. Or better yet, if there were functions, variables or whatever that had all the information I see on that output, such as follows:
- HadoopVersion
- PigVersion
- UserId,
- StartedAt
- FinishedAt,
- Features
- List of hadoop jobs with specific stats (JobId, Maps, Reduces, MaxMapTime, MinMapTime, AvgMapTime, MaxReduceTime, MinReduceTime, AvgReduceTime, Alias, Feature Outputs)
- Input Information (Records Read and Source, ex. hbase table name)
- Output Information (Records Written and Destination, ex. hbase table name)
- Indication if it was Success or Failure
- etc
If I figure out how to get that output file, I can parse it and retrieve all the information.
Thank you.
Eduardo.
________________________________
From: Kamal Hakim <Ka...@aexp.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Tuesday, September 4, 2012 5:40 PM
Subject: RE: Capturing Pig action output
Hi Eduardo ,
You could run the Pig command as an in-line script command and capture the output into an output file.
Example:
pig -f pigscript.pig > pig_log.log
Kamal Hakim
American Express
Big Data Architecture
Phone: 602-537-6819
________________________________________
From: Eduardo Afonso Ferreira [eafonsof@yahoo.com]
Sent: Tuesday, September 04, 2012 02:24 PM
To: oozie-users@incubator.apache.org
Subject: Re: Capturing Pig action output
Hey, thanks for the response.
What I need is not the log file that's created by Pig when there's an error, you know, the one that lists stack trace and things like that.
I need the output that Pig sends to stdout/stderr. This is the output that includes information about hadoop jobs created, started/finished timestamps, success/failure, etc.
Here's an example of part of that output:
________________________________
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u3 0.11.0-SNAPSHOT mapred 2012-09-04 17:00:56 2012-09-04 17:06:26 GROUP_BY,DISTINCT,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201206281058_812355 14 4 20 11 16 250 235 246 B,esi,filt1,keys,slots DISTINCT
job_201206281058_812414 4 4 7 6 7 26 22 24 1-4,C,E GROUP_BY,DISTINCT hbase://active_video_plays,
Input(s):
Successfully read 144190 records (5079 bytes) from: "hbase://events_sessions"
Output(s):
Successfully stored 20835 records in: "hbase://active_video_plays"
Counters:
Total records written : 20835
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
________________________________
Virag,
We're currently using Oozie 2.3.2 (2.3.2-cdh3u3)and I guess the wf configuration you mentioned is not available on this version.
Eduardo.
________________________________
From: Mona Chitnis <ch...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, August 30, 2012 4:01 PM
Subject: Re: Capturing Pig action output
Hi Eduardo,
The log file where the Oozie pig action's output is written to, is a local
file in the current working directory of the map task, and not on HDFS.
Also, passing a custom path using "pig -logfile <HDFS_PATH>" is not
allowed. Can you try with passing another argument at the end of your
arguments list to your pig action as a redirection to some file of choice?
E.g. Trying to recreate something like the following. Note the redirection
to 'myfile.txt' at the end
pig -param $xyz=1000 myscript.pig &>myfile.txt
I haven't tried this out myself but it will be helpful to find out.
Otherwise, Virag's suggestion can help you access specific stats related
information
Regards,
--
Mona Chitnis
On 8/30/12 11:59 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>be accessed through the API or EL function.
>
>First, the following should be set in wf configuration. This will store
>the Pig/MR related statistics in the DB.
><property>
> <name>oozie.action.external.stats.write</name>
> <value>true</value>
> </property>
>
>Then, the stats and jobIds can be accessed using the verbose API
>oozie job -info <jobId> -verbose
>
>Also, the hadoop job Id's can be retrieved for a Pig action through
>El-function
>
>wf:actionData(<pig-action-name>)["hadoopJobs"]
>
>
>Detailed docs at
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunct
>i
>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>
>Thanks,
>Virag
>
>
>
>
>
>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hi there,
>>
>>I have a pig that runs periodically by oozie via coordinator with a set
>>frequency.
>>I wanted to capture the Pig script output because I need to look at some
>>information on the results to keep track of several things.
>>I know I can look at the output by doing a whole bunch of clicks starting
>>at the oozie web console as follows:
>>
>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>- Find and click the specific job under "Workflow Jobs"
>>- Select (click) the pig action in the window that pops up
>>- Click the magnifying glass icon on the "Console URL" field
>>- Click the Map of the launcher job
>>- Click the task ID
>>- Click All under "Task Logs"
>>
>>My question is how can I know the exact name and location of that log
>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>parse and look for what I need?
>>
>>Is this something I can determine ahead of time, like pass a
>>parameter/argument to the action/pig so that it will store the log where
>>I want with the file name I want?
>>
>>Thanks in advance for your help.
>>Eduardo.
>
American Express made the following annotations on Tue Sep 04 2012 14:40:20
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Tue Sep 04 2012 14:40:20
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
******************************************************************************
-------------------------------------------------------------------------------
RE: Capturing Pig action output
Posted by Kamal Hakim <Ka...@aexp.com>.
Hi Eduardo ,
You could run the Pig command as an in-line script command and capture the output into an output file.
Example:
pig -f pigscript.pig > pig_log.log
Kamal Hakim
American Express
Big Data Architecture
Phone: 602-537-6819
________________________________________
From: Eduardo Afonso Ferreira [eafonsof@yahoo.com]
Sent: Tuesday, September 04, 2012 02:24 PM
To: oozie-users@incubator.apache.org
Subject: Re: Capturing Pig action output
Hey, thanks for the response.
What I need is not the log file that's created by Pig when there's an error, you know, the one that lists stack trace and things like that.
I need the output that Pig sends to stdout/stderr. This is the output that includes information about hadoop jobs created, started/finished timestamps, success/failure, etc.
Here's an example of part of that output:
________________________________
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u3 0.11.0-SNAPSHOT mapred 2012-09-04 17:00:56 2012-09-04 17:06:26 GROUP_BY,DISTINCT,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201206281058_812355 14 4 20 11 16 250 235 246 B,esi,filt1,keys,slots DISTINCT
job_201206281058_812414 4 4 7 6 7 26 22 24 1-4,C,E GROUP_BY,DISTINCT hbase://active_video_plays,
Input(s):
Successfully read 144190 records (5079 bytes) from: "hbase://events_sessions"
Output(s):
Successfully stored 20835 records in: "hbase://active_video_plays"
Counters:
Total records written : 20835
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
________________________________
Virag,
We're currently using Oozie 2.3.2 (2.3.2-cdh3u3)and I guess the wf configuration you mentioned is not available on this version.
Eduardo.
________________________________
From: Mona Chitnis <ch...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, August 30, 2012 4:01 PM
Subject: Re: Capturing Pig action output
Hi Eduardo,
The log file where the Oozie pig action's output is written to, is a local
file in the current working directory of the map task, and not on HDFS.
Also, passing a custom path using "pig -logfile <HDFS_PATH>" is not
allowed. Can you try with passing another argument at the end of your
arguments list to your pig action as a redirection to some file of choice?
E.g. Trying to recreate something like the following. Note the redirection
to 'myfile.txt' at the end
pig -param $xyz=1000 myscript.pig &>myfile.txt
I haven't tried this out myself but it will be helpful to find out.
Otherwise, Virag's suggestion can help you access specific stats related
information
Regards,
--
Mona Chitnis
On 8/30/12 11:59 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>be accessed through the API or EL function.
>
>First, the following should be set in wf configuration. This will store
>the Pig/MR related statistics in the DB.
><property>
> <name>oozie.action.external.stats.write</name>
> <value>true</value>
> </property>
>
>Then, the stats and jobIds can be accessed using the verbose API
>oozie job -info <jobId> -verbose
>
>Also, the hadoop job Id's can be retrieved for a Pig action through
>El-function
>
>wf:actionData(<pig-action-name>)["hadoopJobs"]
>
>
>Detailed docs at
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunct
>i
>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>
>Thanks,
>Virag
>
>
>
>
>
>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hi there,
>>
>>I have a pig that runs periodically by oozie via coordinator with a set
>>frequency.
>>I wanted to capture the Pig script output because I need to look at some
>>information on the results to keep track of several things.
>>I know I can look at the output by doing a whole bunch of clicks starting
>>at the oozie web console as follows:
>>
>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>- Find and click the specific job under "Workflow Jobs"
>>- Select (click) the pig action in the window that pops up
>>- Click the magnifying glass icon on the "Console URL" field
>>- Click the Map of the launcher job
>>- Click the task ID
>>- Click All under "Task Logs"
>>
>>My question is how can I know the exact name and location of that log
>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>parse and look for what I need?
>>
>>Is this something I can determine ahead of time, like pass a
>>parameter/argument to the action/pig so that it will store the log where
>>I want with the file name I want?
>>
>>Thanks in advance for your help.
>>Eduardo.
>
American Express made the following annotations on Tue Sep 04 2012 14:40:20
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Tue Sep 04 2012 14:40:20
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
******************************************************************************
-------------------------------------------------------------------------------
Re: Capturing Pig action output
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey, thanks for the response.
What I need is not the log file that's created by Pig when there's an error, you know, the one that lists stack trace and things like that.
I need the output that Pig sends to stdout/stderr. This is the output that includes information about hadoop jobs created, started/finished timestamps, success/failure, etc.
Here's an example of part of that output:
________________________________
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u3 0.11.0-SNAPSHOT mapred 2012-09-04 17:00:56 2012-09-04 17:06:26 GROUP_BY,DISTINCT,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201206281058_812355 14 4 20 11 16 250 235 246 B,esi,filt1,keys,slots DISTINCT
job_201206281058_812414 4 4 7 6 7 26 22 24 1-4,C,E GROUP_BY,DISTINCT hbase://active_video_plays,
Input(s):
Successfully read 144190 records (5079 bytes) from: "hbase://events_sessions"
Output(s):
Successfully stored 20835 records in: "hbase://active_video_plays"
Counters:
Total records written : 20835
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
________________________________
Virag,
We're currently using Oozie 2.3.2 (2.3.2-cdh3u3)and I guess the wf configuration you mentioned is not available on this version.
Eduardo.
________________________________
From: Mona Chitnis <ch...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, August 30, 2012 4:01 PM
Subject: Re: Capturing Pig action output
Hi Eduardo,
The log file where the Oozie pig action's output is written to, is a local
file in the current working directory of the map task, and not on HDFS.
Also, passing a custom path using "pig -logfile <HDFS_PATH>" is not
allowed. Can you try with passing another argument at the end of your
arguments list to your pig action as a redirection to some file of choice?
E.g. Trying to recreate something like the following. Note the redirection
to 'myfile.txt' at the end
pig -param $xyz=1000 myscript.pig &>myfile.txt
I haven't tried this out myself but it will be helpful to find out.
Otherwise, Virag's suggestion can help you access specific stats related
information
Regards,
--
Mona Chitnis
On 8/30/12 11:59 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>be accessed through the API or EL function.
>
>First, the following should be set in wf configuration. This will store
>the Pig/MR related statistics in the DB.
><property>
> <name>oozie.action.external.stats.write</name>
> <value>true</value>
> </property>
>
>Then, the stats and jobIds can be accessed using the verbose API
>oozie job -info <jobId> -verbose
>
>Also, the hadoop job Id's can be retrieved for a Pig action through
>El-function
>
>wf:actionData(<pig-action-name>)["hadoopJobs"]
>
>
>Detailed docs at
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunct
>i
>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>
>Thanks,
>Virag
>
>
>
>
>
>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hi there,
>>
>>I have a pig that runs periodically by oozie via coordinator with a set
>>frequency.
>>I wanted to capture the Pig script output because I need to look at some
>>information on the results to keep track of several things.
>>I know I can look at the output by doing a whole bunch of clicks starting
>>at the oozie web console as follows:
>>
>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>- Find and click the specific job under "Workflow Jobs"
>>- Select (click) the pig action in the window that pops up
>>- Click the magnifying glass icon on the "Console URL" field
>>- Click the Map of the launcher job
>>- Click the task ID
>>- Click All under "Task Logs"
>>
>>My question is how can I know the exact name and location of that log
>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>parse and look for what I need?
>>
>>Is this something I can determine ahead of time, like pass a
>>parameter/argument to the action/pig so that it will store the log where
>>I want with the file name I want?
>>
>>Thanks in advance for your help.
>>Eduardo.
>
Re: Capturing Pig action output
Posted by Mona Chitnis <ch...@yahoo-inc.com>.
Hi Eduardo,
The log file where the Oozie pig action's output is written to, is a local
file in the current working directory of the map task, and not on HDFS.
Also, passing a custom path using "pig -logfile <HDFS_PATH>" is not
allowed. Can you try with passing another argument at the end of your
arguments list to your pig action as a redirection to some file of choice?
E.g. Trying to recreate something like the following. Note the redirection
to 'myfile.txt' at the end
pig -param $xyz=1000 myscript.pig &>myfile.txt
I haven't tried this out myself but it will be helpful to find out.
Otherwise, Virag's suggestion can help you access specific stats related
information
Regards,
--
Mona Chitnis
On 8/30/12 11:59 AM, "Virag Kothari" <vi...@yahoo-inc.com> wrote:
>Hi,
>
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
>be accessed through the API or EL function.
>
>First, the following should be set in wf configuration. This will store
>the Pig/MR related statistics in the DB.
><property>
> <name>oozie.action.external.stats.write</name>
> <value>true</value>
> </property>
>
>Then, the stats and jobIds can be accessed using the verbose API
>oozie job -info <jobId> -verbose
>
>Also, the hadoop job Id's can be retrieved for a Pig action through
>El-function
>
>wf:actionData(<pig-action-name>)["hadoopJobs"]
>
>
>Detailed docs at
>http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunct
>i
>onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
>
>Thanks,
>Virag
>
>
>
>
>
>On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>
>>Hi there,
>>
>>I have a pig that runs periodically by oozie via coordinator with a set
>>frequency.
>>I wanted to capture the Pig script output because I need to look at some
>>information on the results to keep track of several things.
>>I know I can look at the output by doing a whole bunch of clicks starting
>>at the oozie web console as follows:
>>
>>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>>- Find and click the specific job under "Workflow Jobs"
>>- Select (click) the pig action in the window that pops up
>>- Click the magnifying glass icon on the "Console URL" field
>>- Click the Map of the launcher job
>>- Click the task ID
>>- Click All under "Task Logs"
>>
>>My question is how can I know the exact name and location of that log
>>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>>parse and look for what I need?
>>
>>Is this something I can determine ahead of time, like pass a
>>parameter/argument to the action/pig so that it will store the log where
>>I want with the file name I want?
>>
>>Thanks in advance for your help.
>>Eduardo.
>
Re: Capturing Pig action output
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Hi,
>From 3.2 onwards, counters and hadoop job ids for Pig and Map-reduce can
be accessed through the API or EL function.
First, the following should be set in wf configuration. This will store
the Pig/MR related statistics in the DB.
<property>
<name>oozie.action.external.stats.write</name>
<value>true</value>
</property>
Then, the stats and jobIds can be accessed using the verbose API
oozie job -info <jobId> -verbose
Also, the hadoop job Id's can be retrieved for a Pig action through
El-function
wf:actionData(<pig-action-name>)["hadoopJobs"]
Detailed docs at
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html. Look under "4.2.5 Hadoop EL Functions"
Thanks,
Virag
On 8/30/12 10:31 AM, "Eduardo Afonso Ferreira" <ea...@yahoo.com> wrote:
>Hi there,
>
>I have a pig that runs periodically by oozie via coordinator with a set
>frequency.
>I wanted to capture the Pig script output because I need to look at some
>information on the results to keep track of several things.
>I know I can look at the output by doing a whole bunch of clicks starting
>at the oozie web console as follows:
>
>- Open oozie web console (ex.: http://localhost:11000/oozie/)
>- Find and click the specific job under "Workflow Jobs"
>- Select (click) the pig action in the window that pops up
>- Click the magnifying glass icon on the "Console URL" field
>- Click the Map of the launcher job
>- Click the task ID
>- Click All under "Task Logs"
>
>My question is how can I know the exact name and location of that log
>file in HDFS so I can programmaticaly retrieve the file from HDFS and
>parse and look for what I need?
>
>Is this something I can determine ahead of time, like pass a
>parameter/argument to the action/pig so that it will store the log where
>I want with the file name I want?
>
>Thanks in advance for your help.
>Eduardo.