[jira] [Created] (SAMZA-952) Add validation for broadcast inputs configured for partitions that don't exist

["Jake Maes (JIRA)" <ji...@apache.org>]

Jake Maes created SAMZA-952:
-------------------------------

             Summary: Add validation for broadcast inputs configured for partitions that don't exist
                 Key: SAMZA-952
                 URL: https://issues.apache.org/jira/browse/SAMZA-952
             Project: Samza
          Issue Type: Bug
    Affects Versions: 0.10.0
            Reporter: Jake Maes


We had an issue at LinkedIn where the user configured a broadcast input as: 

{noformat}
task.broadcast.inputs=kafka-queueing.ads_glob#[0-9]
{noformat}

But the ads_glob topic only has 8 partitions. 

This caused the following exception in the container log:
{noformat}
2016-05-24 19:59:20 SamzaContainer [ERROR] Caught exception in process loop.
org.apache.samza.SamzaException: No metadata available for partition null.
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:412)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:380)
	at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:380)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:378)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
	at org.apache.samza.checkpoint.OffsetManager.loadDefaults(OffsetManager.scala:378)
	at org.apache.samza.checkpoint.OffsetManager.start(OffsetManager.scala:175)
	at org.apache.samza.container.SamzaContainer.startOffsetManager(SamzaContainer.scala:604)
	at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:550)
	at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
	at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
	at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
{noformat}

We should add some validation and a meaningful error message for cases when the broadcast inputs don't make sense compared to the metadata. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)