You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Siddhartha Singh Sandhu <sa...@gmail.com> on 2017/07/28 19:25:22 UTC

RE: changing directories in Spark Streming

Hi,

I am saving the output of my streaming process to s3.

I want to able to change the directory of the stream as an hour passes by.

Will this work:

parsed_kf_frame.saveAsTextFiles((s3_location).format(
        datetime.datetime.today().strftime("%Y%m%d"),
datetime.datetime.today().strftime("%H"), "prefix_key"), "csv")

Thank You!

Sid.