You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by abdelhakim deneche <ad...@gmail.com> on 2015/04/21 18:11:12 UTC
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated April 21, 2015, 4:11 p.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
ParquetRecordWriter does not create a new file/folder until it's flushing data to the file.
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 3e35721
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 89837e7
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated April 27, 2015, 1:57 a.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
- changed ParquetRecordWriter to create the parquet files as soon as one record is added to the writer
- moved corresponding test to TestParquetWriterEmptyFiles
- fixed BaseTestQuery to update storage plugin in all drillbits
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 3e35721
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java b02051b
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 89837e7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriterEmptyFiles.java PRE-CREATION
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche