You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Mariappan Asokan (JIRA)" <ji...@apache.org> on 2014/11/15 01:35:35 UTC

[jira] [Created] (PARQUET-134) Enhance ParquetWriter with file creation flag

Mariappan Asokan created PARQUET-134:
----------------------------------------

             Summary: Enhance ParquetWriter with file creation flag
                 Key: PARQUET-134
                 URL: https://issues.apache.org/jira/browse/PARQUET-134
             Project: Parquet
          Issue Type: Improvement
            Reporter: Mariappan Asokan


Currently, Parquet files can be created only with "create in exclusive mode" (like {{O_CREAT|O_EXCL}} in UNIX {{open}} system call.)  This may be inconvenient in a situation where user knows that overwriting an existing file is okay.  This is especially true for Parquet files created on a local disk.  For example, a user might want to specify a named pipe (on UNIX) as the target, encrypt the bytes on the fly by reading from the named pipe in another process, and write to disk.  The named pipe file has to exist first.

I am thinking that we can have OVERWRITE and CREATE modes for now.  If appending to Parquet files is supported in the future, we can add APPEND mode later.  These mode flags can be defined as constants in {{ParquetWriter.java.}}

There are 7 constructors in {{ParquetWriter}} class.  I am thinking of adding one more that takes the mode flag (and all the arguments of the constructor with the most arguments today.)  Also, a new constructor of {{ParquetFileWriter}} will take the mode flag as an argument and the following statement
{code}
this.out = fs.create(file, false);
{code}
will be modified so that for OVERWRITE mode, the second argument to {{create()}} will be set to {{true.}}

I can submit a patch with the above changes and a test.  Committers, please give your feedback with suggestions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)