You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2016/11/15 15:45:00 UTC

SQL analyzer breakdown

We see the analyzer break down almost guaranteed when programs get to a
certain size or complexity. It starts complaining with messages along the
lines of "cannot find column x#255 in list of columns that includes x#255".
The workaround is to go to rdd and back. Is there a way to achieve the same
(force the analyzer to break it up in chunks that are done independently)
without the roundtrip to rdd and back? Like a checkpoint for the analyzer
so to speak.