You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Purshotam Shah (JIRA)" <ji...@apache.org> on 2014/04/17 03:31:15 UTC
[jira] [Updated] (OOZIE-1798) Limit the number of dfs/hcat calls
for latest/future evaluation
[ https://issues.apache.org/jira/browse/OOZIE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Purshotam Shah updated OOZIE-1798:
----------------------------------
Description:
{code}
while (nominalInstanceCal.compareTo(initInstance) >= 0 && !currentThread.isInterrupted()) {
ELEvaluator uriEval = getUriEvaluator(nominalInstanceCal);
String uriPath = uriEval.evaluate(uriTemplate, String.class);
if (uriHandler == null) {
URI uri = new URI(uriPath);
uriHandler = uriService.getURIHandler(uri);
uriContext = uriHandler.getContext(uri, conf, user);
}
String uriWithDoneFlag = uriHandler.getURIWithDoneFlag(uriPath, doneFlag);
if (uriHandler.exists(new URI(uriWithDoneFlag), uriContext)) {
XLog.getLog(CoordELFunctions.class)
.debug("Found latest(" + available + "): " + uriWithDoneFlag);
if (available == startOffset) {
LOG.debug("Matched latest(" + available + "): " + uriWithDoneFlag);
resolved = true;
resolvedInstances.append(DateUtils.formatDateOozieTZ(nominalInstanceCal));
resolvedURIPaths.append(uriPath);
retVal = resolvedInstances.toString();
eval.setVariable("resolved_path", resolvedURIPaths.toString());
break;
}
else if (available <= endOffset) {
LOG.debug("Matched latest(" + available + "): " + uriWithDoneFlag);
resolvedInstances.append(DateUtils.formatDateOozieTZ(nominalInstanceCal)).append(
INSTANCE_SEPARATOR);
resolvedURIPaths.append(uriPath).append(INSTANCE_SEPARATOR);
}
available--;
}
// nominalInstanceCal.add(dsTimeUnit.getCalendarUnit(), -datasetFrequency);
nominalInstanceCal = (Calendar) initInstance.clone();
instCount[0]--;
nominalInstanceCal.add(dsTimeUnit.getCalendarUnit(), instCount[0] * datasetFrequency);
// DateUtils.moveToEnd(nominalInstanceCal, getDSEndOfFlag());
}
{code}
If no partition exist, a fault coord can bring down hcat server. Other option is to support range query.
> Limit the number of dfs/hcat calls for latest/future evaluation
> ---------------------------------------------------------------
>
> Key: OOZIE-1798
> URL: https://issues.apache.org/jira/browse/OOZIE-1798
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>
> {code}
> while (nominalInstanceCal.compareTo(initInstance) >= 0 && !currentThread.isInterrupted()) {
> ELEvaluator uriEval = getUriEvaluator(nominalInstanceCal);
> String uriPath = uriEval.evaluate(uriTemplate, String.class);
> if (uriHandler == null) {
> URI uri = new URI(uriPath);
> uriHandler = uriService.getURIHandler(uri);
> uriContext = uriHandler.getContext(uri, conf, user);
> }
> String uriWithDoneFlag = uriHandler.getURIWithDoneFlag(uriPath, doneFlag);
> if (uriHandler.exists(new URI(uriWithDoneFlag), uriContext)) {
> XLog.getLog(CoordELFunctions.class)
> .debug("Found latest(" + available + "): " + uriWithDoneFlag);
> if (available == startOffset) {
> LOG.debug("Matched latest(" + available + "): " + uriWithDoneFlag);
> resolved = true;
> resolvedInstances.append(DateUtils.formatDateOozieTZ(nominalInstanceCal));
> resolvedURIPaths.append(uriPath);
> retVal = resolvedInstances.toString();
> eval.setVariable("resolved_path", resolvedURIPaths.toString());
> break;
> }
> else if (available <= endOffset) {
> LOG.debug("Matched latest(" + available + "): " + uriWithDoneFlag);
> resolvedInstances.append(DateUtils.formatDateOozieTZ(nominalInstanceCal)).append(
> INSTANCE_SEPARATOR);
> resolvedURIPaths.append(uriPath).append(INSTANCE_SEPARATOR);
> }
> available--;
> }
> // nominalInstanceCal.add(dsTimeUnit.getCalendarUnit(), -datasetFrequency);
> nominalInstanceCal = (Calendar) initInstance.clone();
> instCount[0]--;
> nominalInstanceCal.add(dsTimeUnit.getCalendarUnit(), instCount[0] * datasetFrequency);
> // DateUtils.moveToEnd(nominalInstanceCal, getDSEndOfFlag());
> }
> {code}
> If no partition exist, a fault coord can bring down hcat server. Other option is to support range query.
--
This message was sent by Atlassian JIRA
(v6.2#6252)