You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Manjunath Shetty H <ma...@live.com> on 2020/03/04 08:58:04 UTC

Way to get the file name of the output when doing ORC write from dataframe

Hi,

I wanted to know if there is any way to get the output file name that `Dataframe.orc()` will write to ?. This is needed to track which file is written by which job during incremental batch jobs.

Thanks
Manjunath

Re: Way to get the file name of the output when doing ORC write from dataframe

Posted by Manjunath Shetty H <ma...@live.com>.

Or is there any way to provide a Unique file name to the ORC write function itself ?

Any suggestions will be helpful.

Regards
Manjunath Shetty
________________________________
From: Manjunath Shetty H <ma...@live.com>
Sent: Wednesday, March 4, 2020 2:28 PM
To: user <us...@spark.apache.org>
Subject: Way to get the file name of the output when doing ORC write from dataframe

Hi,

I wanted to know if there is any way to get the output file name that `Dataframe.orc()` will write to ?. This is needed to track which file is written by which job during incremental batch jobs.

Thanks
Manjunath