You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jason Altekruse <al...@gmail.com> on 2014/05/02 23:39:58 UTC
Re: Review Request 20600: Drill- 400 - parquet utf8
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/
-----------------------------------------------------------
(Updated May 2, 2014, 9:39 p.m.)
Review request for drill and Jacques Nadeau.
Changes
-------
Respects the ConvertedType specified in the file rather than assuming everything should be interpreted as UTF-8.
Repository: drill-git
Description (updated)
-------
Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
Diffs (updated)
-----
exec/java-exec/pom.xml 60eeb1f
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
Diff: https://reviews.apache.org/r/20600/diff/
Testing
-------
amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
Thanks,
Jason Altekruse
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42214
-----------------------------------------------------------
Ship it!
Ship It!
- Timothy Chen
On May 2, 2014, 10:29 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 10:29 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Steven Phillips <sp...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42208
-----------------------------------------------------------
Ship it!
Ship It!
- Steven Phillips
On May 2, 2014, 10:29 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 10:29 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/
-----------------------------------------------------------
(Updated May 2, 2014, 10:29 p.m.)
Review request for drill and Jacques Nadeau.
Changes
-------
removed commented out code.
Repository: drill-git
Description
-------
Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
Diffs (updated)
-----
exec/java-exec/pom.xml 196b095
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
Diff: https://reviews.apache.org/r/20600/diff/
Testing
-------
amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
Thanks,
Jason Altekruse
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42070
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java
<https://reviews.apache.org/r/20600/#comment75816>
I just pulled it out of the previous VectorHolder interface. We should centralize this somewhere so it can be adjusted once we are gathering statistics during the reading process.
- Jason Altekruse
On May 2, 2014, 9:49 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 9:49 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
> On May 2, 2014, 9:58 p.m., Timothy Chen wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java, line 324
> > <https://reviews.apache.org/r/20600/diff/3/?file=573942#file573942line324>
> >
> > How you pick 10 and 5?
I just pulled it out of the previous VectorHolder interface. We should centralize this somewhere so it can be adjusted once we are gathering statistics during the reading process.
> On May 2, 2014, 9:58 p.m., Timothy Chen wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java, line 119
> > <https://reviews.apache.org/r/20600/diff/3/?file=573936#file573936line119>
> >
> > Not knowing all the details, but sure want to point out reset() and clear() are doing really different things. So hopfully this is intentional
This is correct. This method was previously creating an extra allocation when it should have been clearing all of the vectors. It is only every used in the cleanup method of the root level reader.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42066
-----------------------------------------------------------
On May 2, 2014, 9:49 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 9:49 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
> On May 2, 2014, 9:58 p.m., Timothy Chen wrote:
> > exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java, line 279
> > <https://reviews.apache.org/r/20600/diff/3/?file=573944#file573944line279>
> >
> > You added the test but it's marked as ignore
It relies on a binary file that isn't in the repo. We are working on a testing framework that should make it easier to write tests that generate binaries using an external tool like a local pig or hive query.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42066
-----------------------------------------------------------
On May 2, 2014, 9:49 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 9:49 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42066
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java
<https://reviews.apache.org/r/20600/#comment75810>
Not knowing all the details, but sure want to point out reset() and clear() are doing really different things. So hopfully this is intentional
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java
<https://reviews.apache.org/r/20600/#comment75809>
Remove
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java
<https://reviews.apache.org/r/20600/#comment75811>
How you pick 10 and 5?
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java
<https://reviews.apache.org/r/20600/#comment75812>
You added the test but it's marked as ignore
- Timothy Chen
On May 2, 2014, 9:49 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 9:49 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/#review42068
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java
<https://reviews.apache.org/r/20600/#comment75814>
This is correct. This method was previously creating an extra allocation when it should have been clearing all of the vectors. It is only every used in the cleanup method of the root level reader.
- Jason Altekruse
On May 2, 2014, 9:49 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20600/
> -----------------------------------------------------------
>
> (Updated May 2, 2014, 9:49 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
>
>
> Diffs
> -----
>
> exec/java-exec/pom.xml 196b095
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
> exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
>
> Diff: https://reviews.apache.org/r/20600/diff/
>
>
> Testing
> -------
>
> amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 20600: Drill- 400 - parquet utf8
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20600/
-----------------------------------------------------------
(Updated May 2, 2014, 9:49 p.m.)
Review request for drill and Jacques Nadeau.
Changes
-------
Forgot to rebase on master
Repository: drill-git
Description
-------
Drill-400 change parquet reader to place varbinary fields into VarCharVectors, allowing them to be returned by default as UTF-8 Strings. Note that this is done for parquet files with ConvertedTypes specified. This field did not exist in some older versions, so these files will still require a cast to see the data as UTF-8.
Diffs (updated)
-----
exec/java-exec/pom.xml 196b095
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BitReader.java c489d5b
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ColumnReader.java d5c88ef
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FixedByteAlignedReader.java 4f14f60
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableBitReader.java 4c060f2
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableColumnReader.java b6ae715
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/NullableFixedByteAlignedReader.java c2fc606
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/PageReadStatus.java 67262f6
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 6e17fba
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/VarLenBinaryReader.java 09d19a8
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 9ba94fa
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 73af98c
Diff: https://reviews.apache.org/r/20600/diff/
Testing
-------
amended parquet tests so they would pass with new return type. A change in value vectors actually enforced a maximum record count in a vector, so a bug was fixed in the reader that allowed for more than 65k records to be inserted into a vector.
Thanks,
Jason Altekruse