You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by abdelhakim deneche <ad...@gmail.com> on 2015/03/20 01:52:00 UTC
Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
Review request for drill, Aman Sinha and Jason Altekruse.
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 8bf9a92
exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java d2d97f8
exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 76328c6
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
> On March 20, 2015, 2:33 a.m., Aman Sinha wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java, line 338
> > <https://reviews.apache.org/r/32273/diff/1/?file=901031#file901031line338>
> >
> > You should also check fs.exists(path) before calling delete.
fixing it right now
- abdelhakim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/#review77163
-----------------------------------------------------------
On March 20, 2015, 12:51 a.m., abdelhakim deneche wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32273/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 12:51 a.m.)
>
>
> Review request for drill, Aman Sinha and Jason Altekruse.
>
>
> Bugs: DRILL-2408
> https://issues.apache.org/jira/browse/DRILL-2408
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 8bf9a92
> exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java d2d97f8
> exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
> exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 76328c6
>
> Diff: https://reviews.apache.org/r/32273/diff/
>
>
> Testing
> -------
>
> - Added 2 unit tests to TestParquetWriter
> - Unit tests are passing
>
>
> Thanks,
>
> abdelhakim deneche
>
>
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by Aman Sinha <as...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/#review77163
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java
<https://reviews.apache.org/r/32273/#comment125038>
You should also check fs.exists(path) before calling delete.
- Aman Sinha
On March 20, 2015, 12:51 a.m., abdelhakim deneche wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32273/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 12:51 a.m.)
>
>
> Review request for drill, Aman Sinha and Jason Altekruse.
>
>
> Bugs: DRILL-2408
> https://issues.apache.org/jira/browse/DRILL-2408
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 8bf9a92
> exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java d2d97f8
> exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
> exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 76328c6
>
> Diff: https://reviews.apache.org/r/32273/diff/
>
>
> Testing
> -------
>
> - Added 2 unit tests to TestParquetWriter
> - Unit tests are passing
>
>
> Thanks,
>
> abdelhakim deneche
>
>
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by Aman Sinha <as...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/#review77171
-----------------------------------------------------------
Ship it!
Ship It!
- Aman Sinha
On March 20, 2015, 4:33 a.m., abdelhakim deneche wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32273/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 4:33 a.m.)
>
>
> Review request for drill, Aman Sinha and Jason Altekruse.
>
>
> Bugs: DRILL-2408
> https://issues.apache.org/jira/browse/DRILL-2408
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 8bf9a92
> exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java d2d97f8
> exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
> exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 76328c6
>
> Diff: https://reviews.apache.org/r/32273/diff/
>
>
> Testing
> -------
>
> - Added 2 unit tests to TestParquetWriter
> - Unit tests are passing
>
>
> Thanks,
>
> abdelhakim deneche
>
>
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated April 27, 2015, 1:57 a.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
- changed ParquetRecordWriter to create the parquet files as soon as one record is added to the writer
- moved corresponding test to TestParquetWriterEmptyFiles
- fixed BaseTestQuery to update storage plugin in all drillbits
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 3e35721
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java b02051b
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 89837e7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriterEmptyFiles.java PRE-CREATION
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated April 21, 2015, 4:11 p.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
ParquetRecordWriter does not create a new file/folder until it's flushing data to the file.
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 3e35721
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 89837e7
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated March 31, 2015, 12:19 a.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
patch rebased on top of master
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 4a8ff5e
exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java 1d59a11
exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 288a295
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche
Re: Review Request 32273: DRILL-2408: Invalid (0 length) parquet file
created by CTAS
Posted by abdelhakim deneche <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32273/
-----------------------------------------------------------
(Updated March 20, 2015, 4:33 a.m.)
Review request for drill, Aman Sinha and Jason Altekruse.
Changes
-------
check if the file exists before deleting it
Bugs: DRILL-2408
https://issues.apache.org/jira/browse/DRILL-2408
Repository: drill-git
Description
-------
I updated ParquetRecordWriter to delete the last file created if it's empty (no records written to it). I also added two unit tests one that checks the default case where we try to create a table using a query that returns 0 rows, the second case can happen if the ParquetRecordWriter flushes it's content just after writing the last record, it will then create a new empty file.
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java 8bf9a92
exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java d2d97f8
exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 520c204
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java 76328c6
Diff: https://reviews.apache.org/r/32273/diff/
Testing
-------
- Added 2 unit tests to TestParquetWriter
- Unit tests are passing
Thanks,
abdelhakim deneche