Spark 1.6.2 short circuit AND filter broken

[Patrick Woody <pa...@gmail.com>]

Hey all,

I hit a pretty nasty bug on 1.6.2 that I can't reproduce on 2.0. Here is
the code/logical plan http://pastebin.com/ULnHd1b6. I have filterPushdown
disabled, so when I call collect here it hits the Exception in my UDF
before doing a null check on the input.

I believe it is a symptom of how the DataSourceStrategy splits the
predicates and recombines since it happens at the Physical planning, but I
haven't gone deeper. This doesn't reproduce if I simply use case classes
and sqlContext.createDataFrame.

Is there going to be a 1.6.3 release where this bug can be fixed? I'm happy
to dig further and send up a PR.

Thanks!
-Pat