You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Mariappan Asokan (JIRA)" <ji...@apache.org> on 2014/11/15 01:35:35 UTC
[jira] [Created] (PARQUET-134) Enhance ParquetWriter with file
creation flag
Mariappan Asokan created PARQUET-134:
----------------------------------------
Summary: Enhance ParquetWriter with file creation flag
Key: PARQUET-134
URL: https://issues.apache.org/jira/browse/PARQUET-134
Project: Parquet
Issue Type: Improvement
Reporter: Mariappan Asokan
Currently, Parquet files can be created only with "create in exclusive mode" (like {{O_CREAT|O_EXCL}} in UNIX {{open}} system call.) This may be inconvenient in a situation where user knows that overwriting an existing file is okay. This is especially true for Parquet files created on a local disk. For example, a user might want to specify a named pipe (on UNIX) as the target, encrypt the bytes on the fly by reading from the named pipe in another process, and write to disk. The named pipe file has to exist first.
I am thinking that we can have OVERWRITE and CREATE modes for now. If appending to Parquet files is supported in the future, we can add APPEND mode later. These mode flags can be defined as constants in {{ParquetWriter.java.}}
There are 7 constructors in {{ParquetWriter}} class. I am thinking of adding one more that takes the mode flag (and all the arguments of the constructor with the most arguments today.) Also, a new constructor of {{ParquetFileWriter}} will take the mode flag as an argument and the following statement
{code}
this.out = fs.create(file, false);
{code}
will be modified so that for OVERWRITE mode, the second argument to {{create()}} will be set to {{true.}}
I can submit a patch with the above changes and a test. Committers, please give your feedback with suggestions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)