Spark-submit Py-files with EMR add step?

["Afshin, Bardia" <ba...@changehealthcare.com>]

I’m writing this email to reach out to the community to demisty the py-files parameter when working with spark-submit and python projects.

Currently I have a project, say

Src/

  *   Main.py
  *   Modules/module1.py

When I zip up the src directory and submit it to spark via emr add step , the namespacing is lost.

Main.py example:
From Modules.module1 import SomeClass

My code returns and error that it cannot find this class, now this works if I goto the instance download my project, and submit it to spark from within the EMR instance via spark-submit , but not when adding it as a step in emr from external call.


Help?

Best,
Bardia
This message is confidential, intended only for the named recipient(s) and may contain information that is privileged or exempt from disclosure under applicable law. If you are not the intended recipient(s), you are notified that the dissemination, distribution, or copying of this message is strictly prohibited. If you receive this message in error or are not the named recipient(s), please notify the sender by return email and delete this message. Thank you.