Logging from transformations in PySpark

[Michael Ravits <mi...@gmail.com>]

Hi,

I'd like to send some performance metrics from some of the transformations
to StatsD.
I understood that I should create a new connection to StatsD from each
transformation which I'm afraid would harm performance.
I've also read that there is a workaround for this in Scala by defining an
object as transient.
My question is whether that's also possible in Python with PySpark?
Specifically I'd like to lazily initialize a transient object that will be
used for sending metrics to StatsD over a local socket connection.

Thanks,
Michael