You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2012/12/05 21:45:58 UTC
[jira] [Updated] (PIG-3081) Pig progress stays at 0% for the first
job in hadoop 23
[ https://issues.apache.org/jira/browse/PIG-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-3081:
------------------------------------
Attachment: PIG-3081.patch
calculateProgress=(mapprog+reduceprog)/2
prog=calculateProgress/totalMRJobs (Pig displays progress as a percentage of
all the jobs combined together and not per job)
mapprog=0.6962389945983887, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.34811949729919434
prog=0.11603983243306477,lastProg=0.11490649978319804
mapprog=0.7032797336578369, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.35163986682891846
prog=0.11721328894297282,lastProg=0.11603983243306477
.....
mapprog=0.7190613150596619, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.35953065752983093
prog=0.11984355250994365,lastProg=0.1190514365832011
mapprog=0.725161075592041, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.3625805377960205
prog=0.12086017926534016,lastProg=0.11984355250994365
if(prog>=(lastProg+0.01)){
int perCom = (int)(prog * 100);
if(perCom!=100) {
log.info( perCom + "% complete");
}
The job progress is checked every 500ms. Since the frequency of checking is
very often and the prog is never > lastProg by more than 0.01 , it stays at 0%
and goes directly to 33% when the first job completed.
> Pig progress stays at 0% for the first job in hadoop 23
> -------------------------------------------------------
>
> Key: PIG-3081
> URL: https://issues.apache.org/jira/browse/PIG-3081
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Rohini Palaniswamy
> Fix For: 0.12
>
> Attachments: PIG-3081.patch
>
>
> We are seeing that for many scripts if there are multiple jobs in the job graph, progress stays at 0% for the first job and jumps to 33% when the first job completes. There is no intermediate progress. After that intermediate progress gets reported for the subsequent jobs. Noticed this with jobs that do filtering and order by.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira