Spark 1.6 Streaming delay after long run

[Chan Chor Pang <ch...@indetail.co.jp>]

after upgrade from Spark 1.5 to 1.6(CDH 5.6.0 -> 5.7.1)
some of our streaming job getting delay after long run.

with a little invesgation, here is what i found.
     - the same program have no problem with Spark 1.5
     - we have two kind of streaming and only those with 
"updateStateByKey" was affected,
     - cpu usage getting higher and higher over time ( with 1core@5% at 
start and 1core@100% after a week )
     - data rate is alound 100 event/s, there is no chance for the cpu 
to work so hard.
     - process time for a batch delay from 100ms at start to 3s after a week
     - evening running the same program(for difference input data), not 
all process delay with the same scale
     - no warning or error message until it delay too much and went out 
of memory
     - process time of customer code seems have no problem
     - memory/heap usage looks normal to me

Im suspecting the problem is comming from updateStateByKey but i cant 
trace it down

any one experience the same problem?


--
BR
Peter Chan

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org