You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Manjunath Shetty H <ma...@live.com> on 2020/04/16 04:47:31 UTC

Spark ORC store written timestamp as column

Hi All,

Is there anyway to store the exact written timestamp in the ORC file through spark ?.
Use case something like `current_timestamp()` function in SQL. Generating in the program will not be equal to actual write time in ORC/hdfs file.

Any suggestions will be helpful.


Thanks
Manjunath

Re: Spark ORC store written timestamp as column

Posted by ZHANG Wei <we...@outlook.com>.

From what I think I understand, the OrcOutputWriter leverages orc-core
to write. I'm wondering if ORC supports the row metadata or not. If
not, maybe the org.apache.orc.Writer::addRowBatch() can be overrided to
record the metadata after RowBatch written.

-- 
Cheers,
-z

On Thu, 16 Apr 2020 04:47:31 +0000
Manjunath Shetty H <ma...@live.com> wrote:

> Hi All,
> 
> Is there anyway to store the exact written timestamp in the ORC file through spark ?.
> Use case something like `current_timestamp()` function in SQL. Generating in the program will not be equal to actual write time in ORC/hdfs file.
> 
> Any suggestions will be helpful.
> 
> 
> Thanks
> Manjunath

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org