You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Brett Meyer <Br...@crowdstrike.com> on 2014/12/29 19:14:56 UTC

How to pass options to KeyConverter using PySpark

I¹m running PySpark on YARN, and I¹m reading in SequenceFiles for which I
have a custom KeyConverter class.  My KeyConverter needs to have some
configuration options passed to it, but I am unable to find a way to get the
options to that class without modifying the Spark source.  Is there a
currently built-in method by which this can be done?  The sequenceFile()
method does not take any arguments for the KeyConverter, and the
SparkContext doesn¹t seem to be accessible from my KeyConverter class, so
I¹m at a loss as to how to accomplish this.

Thanks,
Brett