Spark Streaming job having issue with Java Flight Recorder (JFR)

[Pramod Biligiri <pr...@gmail.com>]

Hi,
Has anyone successfully used Java Flight Recorder (JFR) with Spark
Streaming on Oracle Java 8? JFR works for me on batch jobs but not with
Streaming.

I'm running my streaming job on Amazon EMR. I have enabled Java Flight
Recorder (JFR) to profile CPU usage. But at the end of the job, the JFR
output files get deleted and are not available. This is the error in the
JFR trace logs:
[jfr][DEBUG][880.590] Release chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part ref
count now 0
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part
deleted.
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr deleted.
[jfr][INFO ][880.593] Stopped recording [id=1,
name=dump-on-exit-clone-of-0, start=Wed Feb 19 17:26:06 UTC 2020, end=Wed
Feb 19 17:40:45 UTC 2020]
[jfr][ERROR][880.593] Could not copy to
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]
 oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][ERROR][880.593]
 oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][ERROR][880.593]
 oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][ERROR][880.593]
 oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][ERROR][880.593]
 oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][ERROR][880.593]    java.lang.Thread.run(Thread.java:748)
[jfr][ERROR][880.593] Could not delete
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]    oracle.jrockit.jfr
The rest of the trace can be found in the attachment (stdout.txt).
Also, I've not changed the default value of "
spark.streaming.stopGracefullyOnShutdown" (its default value is false)

The Spark driver and executor JVMs are launched with the following options:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
-XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=/tmp,maxage=2h,dumponexit=true,dumponexitpath=/tmp,loglevel=trace

Pramod