[jira] [Resolved] (SAMZA-502) BootstrappingChooser never unblocks if bootstrap topic is initially empty

["Chris Riccomini (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Riccomini resolved SAMZA-502.
-----------------------------------
    Resolution: Fixed

Thanks for looking into this Yan. It looks like you're right. When bootstrapping chooser is registered, it checks:

{noformat}
    // If the offset we're starting to consume from is the same as the upcoming 
    // offset for this system stream partition, then we've already read all
    // messages in the stream, and we're at head for this system stream 
    // partition.
    checkOffset(systemStreamPartition, offset, OffsetType.UPCOMING)
{noformat}

After the SAMZA-461 patch, we default to UPCOMING if the stream is empty:

{noformat}
val nextOffset = {
  val requested = systemStreamPartitionMetadata.getOffset(offsetType)

  if (requested == null) {
    warn(s"Null offset for offset type $offsetType in $systemStreamPartition. Starting read from next possible offset.")
    systemStreamPartitionMetadata.getOffset(OffsetType.UPCOMING)
  } else requested
}
{noformat}

Doing this should trigger the bootstrapping chooser to remove the SSP from the "unbootstrapped" set during registration.

Resolving.

> BootstrappingChooser never unblocks if bootstrap topic is initially empty 
> --------------------------------------------------------------------------
>
>                 Key: SAMZA-502
>                 URL: https://issues.apache.org/jira/browse/SAMZA-502
>             Project: Samza
>          Issue Type: Bug
>          Components: container
>    Affects Versions: 0.8.0
>            Reporter: Roger Hoover
>            Assignee: Yan Fang
>            Priority: Critical
>             Fix For: 0.9.0
>
>
> To reproduce:
> - Create a job that bootstraps topic "config" and consumes topic "events"
> - Create both topics in Kafka but only populate the "events" topic
> - Start the job
> - Now populate the "config" topic
> What happens:
> - The job will consume the messages in the "config" topic but not unblock the "events" topic
> What should happen:
> * Either
> ** Samza should consider an empty bootstrap topic as "caught up" and immediately start processing the "event" topic
> ** Samza should consider an empty bootstrap topic as a fatal error and exit
> I uploaded some trace logs here in case they help: http://pastebin.com/RsdwYkRk
> In this case, the bootstrap topic is called "ops.web.parse_rules__ck_4_module" and the event topic is called "ops._raw.web_parsed"
> I'm testing using the ProcessJobFactory in case it matters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)