[jira] [Created] (SPARK-4545) If first Spark Streaming batch fails, it waits 10x batch duration before stopping

["Sean Owen (JIRA)" <ji...@apache.org>]

Sean Owen created SPARK-4545:
--------------------------------

             Summary: If first Spark Streaming batch fails, it waits 10x batch duration before stopping
                 Key: SPARK-4545
                 URL: https://issues.apache.org/jira/browse/SPARK-4545
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.1.0
            Reporter: Sean Owen
            Priority: Minor


(I'd like to track the issue raised at http://mail-archives.apache.org/mod_mbox/spark-dev/201411.mbox/%3CCAMAsSdKY=QCT0YUdrkvbVuqXdFCGp1+6g-=s71Fk8ZR4uaTK7g@mail.gmail.com%3E as a JIRA since I think it's a legitimate issue that I can take a look into, with some help.)

This bit of {{JobGenerator.stop()}} executes, since the message appears in the logs:

{code}
def haveAllBatchesBeenProcessed = {
  lastProcessedBatch != null && lastProcessedBatch.milliseconds == stopTime
}
logInfo("Waiting for jobs to be processed and checkpoints to be written")
while (!hasTimedOut && !haveAllBatchesBeenProcessed) {
  Thread.sleep(pollTime)
}

// ... 10x batch duration wait here, before seeing the next line log:

logInfo("Waited for jobs to be processed and checkpoints to be written")
{code}

I think that {{lastProcessedBatch}} is always null since no batch ever
succeeds. Of course, for all this code knows, the next batch might
succeed and so is there waiting for it. But it should proceed after
one more batch completes, even if it failed?

{{JobGenerator.onBatchCompleted}} is only called for a successful batch.
Can it be called if it fails too? I think that would fix it.

Should the condition also not be {{lastProcessedBatch.milliseconds <=
stopTime}} instead of == ?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org