[SPARK: org.apache.spark.util.TaskCompletionListenerException]

[Romeo Valencia <Ro...@clarivate.com>]

Hi,
I wonder if someone could help me in finding the solution to a rather vague exception that we are getting.   I am attaching the STDOUT & STDERR files when we execute spark-submit.   The exception message that we are getting is per below excerpt.

"org.apache.spark.util.TaskCompletionListenerException: org.codehaus.jackson.JsonGenerationException: Incomplete surrogate pair: first char 0xdf46, second 0x5b"

This normally happens and according to stack trace is from the code (excerpt).
..
..
GraphToTableLogger.warn("running collect on component")
val distinctComps = ss.sql("SELECT CAST(componentID AS VARCHAR) componentID FROM components_DF GROUP BY componentID")
// .repartition(repartition_size)
  .collect()
..
..


What makes it interesting is that the same dataset when re-invoking the spark-submit again will complete.
Appreciate the help in advance.
______________________________________________________________________
Thanks and best regards,

Romeo Valencia
Senior Data Engineer, IP MM-Product Management-USA |  Clarivate Analytics
Phone +1 415 278 8463  | markmonitor.com<http://www.markmonitor.com/> |  clarivate.com<http://clarivate.com/>
50 California St.  |  San Francisco, CA  94111  |  US
[cid:image003.png@01D2B3B7.8C61C350]