You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2018/11/26 13:05:23 UTC

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11984


Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet-common.cc
M be/src/exec/parquet-common.h
M be/src/exec/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 168 insertions(+), 70 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/1
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1540/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Dec 2018 14:29:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148:   if (e.type == parquet::Type::INT96 && convert_int96_timestamps) needs_conversion = true;
> I see your problem now as this looks suboptimal in my opinion too.
I would vote for returning to the last solution.

My problem with saving it in a member is that another member would be needed to store the physical type, because currently it is not stored whether the timestamp is int64 or int96, as this information is always available in the context when ParquetTimestampDecoder is used (often as template parameter, so without runtime cost).



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Dec 2018 14:47:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1452/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Nov 2018 13:00:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11984

to look at the new patch set (#5).

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision that
stores timestamps as nanoseconds since the Unix epoch.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 178 insertions(+), 69 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/5
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11984

to look at the new patch set (#2).

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet-common.cc
M be/src/exec/parquet-common.h
M be/src/exec/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 168 insertions(+), 70 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/2
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 5: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148: 
> I would vote for returning to the last solution.
Sure, I'm convinced. Sorry for the extra few rounds :)



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Dec 2018 19:00:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1431/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Nov 2018 13:53:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Dec 2018 15:39:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148:   if (e.type == parquet::Type::INT96 && convert_int96_timestamps) needs_conversion = true;
> Still, if you have a function to set variables based on the schema, that fo
I have added convert_int96_timestamps as a parameter to GetTimestampInfoFromSchema. If you think that it looks worse now, I can revert to the original solution or try to find a new one.



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Dec 2018 13:58:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 4: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
File fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java:

http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java@248
PS2, Line 248:     // The Parquet API is responsible for deducing logical type if only converted type
> The Parquet API we use in Java does fill the logical type if only the conve
I see, thanks!



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Nov 2018 16:06:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/11984/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11984/2//COMMIT_MSG@12
PS2, Line 12: 1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC
> Maybe you could mention that it stores the number of nanoseconds since the 
Done


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h
File be/src/exec/parquet-common.h:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@479
PS2, Line 479: LimitedRange
> Dows this "LimitedRange" part of the name add extra value? Am I right that 
There is already a function with nanosecond precision, and it can create the full range of timestamps:
FromUnixTimeNanos(time_t unix_time, int64_t nanos,
      const Timezone& local_tz);

That function could have been used here (after splitting nanoseconds to seconds and subseconds), but I think that it would slower than the specialized solution.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@507
PS2, Line 507:   /// conversion is necessary. Returns true if the schema describes a valid timestamp
> Could you please comment what the parameters are for, pls?
Done


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@509
PS2, Line 509: ProcessSchema
> I feel that something like GetTimestampInfoFromSchema might be a bit more s
Done


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@102
PS2, Line 102:   // Timestamps can be only encoded as INT64 or INT96.
> nit: You might want to adjust this comment to cover the case when this func
I am not sure if I have done what you meant. This function should be only called for TIMESTAMP SQL columns, and should return false if the Parquet column is not timestamp.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@106
PS2, Line 106: Newer solution,
> nit: No need for this part. In addition, it's not guaranteed that reading t
Some explanation about "new": LogicalType is often called "new logical type", as converted type was also some kind of logical type. I agree that in 2 years this will not be meaningful.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@122
PS2, Line 122: Older solution,
> No need for "Older solution" part of the comment.
Done


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@123
PS2, Line 123: are/were never
> nit: I think we shouldn't refer to what happened before this change as we s
This code has to handle files written by older versions of Impala/some other tools we try to be compatible with, so I think that referring to he past makes sense here.

Not specifying whether utc->local conversion is necessary was a gap in Parquet specification and was interpreted by different software differently. This will not be an issue with the new logical type, but we have to make assumptions with older files.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148:   if (e.type == parquet::Type::INT96 && convert_int96_timestamps) needs_conversion = true;
> Shouldn't this line be also included in ProcessSchema? Since you introduced
I thought that doing it this way is the lesser evil - convert_int96_timestamps is not readily available in validation logic (it depends on a flag and the writer of the file).


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc
File be/src/exec/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc@71
PS2, Line 71:   return ParquetTimestampDecoder::ProcessSchema(element, precision, needs_conversion);
> Just thinking out loud here: This ProcessSchema() is now used for 1) checki
Before this change the two functions were separate, but with the addition of logical types the code duplication became too much for me and I tried to merge the bulk of the two functions in ProcessSchema.

I think that the ideal way to do this would be to do the whole processing only once per column chunk (currently it is validated once, then processed again for stat filtering if needed, then once again to initialize the column reader). Changing this to use the same processed metadata would be a non-trivial refactor, as most of the code assumes everything needed for decoding can be simply deduced from the physical + SQL types, so handling of more complex types is a bit hacky.


http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
File fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java:

http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java@248
PS2, Line 248:     LogicalTypeAnnotation logicalType = parquetType.getLogicalTypeAnnotation();
> What happens when there is no "logical type" in a Parquet file, but only "c
The Parquet API we use in Java does fill the logical type if only the converted type is filled - every converted type can be expressed as logical type, but not vice versa.



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Nov 2018 16:30:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

(9 comments)

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h
File be/src/exec/parquet-common.h:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@479
PS2, Line 479: LimitedRange
Dows this "LimitedRange" part of the name add extra value? Am I right that we don't have any other kind of "Timestamp with Nano precision"? Then we can refer to this implementation simply as UnixTimeNanos in my opinion.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@507
PS2, Line 507:   /// conversion is necessary. Returns true if the schema describes a valid timestamp
Could you please comment what the parameters are for, pls?


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@509
PS2, Line 509: ProcessSchema
I feel that something like GetTimestampInfoFromSchema might be a bit more self-descriptive. There might be even better names :)


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@102
PS2, Line 102:   // Timestamps can be only encoded as INT64 or INT96.
nit: You might want to adjust this comment to cover the case when this function is called on non-timestamp related types e.g. string.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@106
PS2, Line 106: Newer solution,
nit: No need for this part. In addition, it's not guaranteed that reading this comment 2 years from now would reveal what "Newer solution" is referred to.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@122
PS2, Line 122: Older solution,
No need for "Older solution" part of the comment.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@123
PS2, Line 123: are/were never
nit: I think we shouldn't refer to what happened before this change as we should only talk about the current code not the previous code.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148:   if (e.type == parquet::Type::INT96 && convert_int96_timestamps) needs_conversion = true;
Shouldn't this line be also included in ProcessSchema? Since you introduced a function to set 'needs_conversion' I think it should tackle all the cases not just a subset of them.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc
File be/src/exec/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc@71
PS2, Line 71:   return ParquetTimestampDecoder::ProcessSchema(element, precision, needs_conversion);
Just thinking out loud here: This ProcessSchema() is now used for 1) checking if the timestamp in the referred element is a valid timestamp and 2) to gather information such as 'populate' and 'precision'. Have you considered splitting these 2 functionalities?



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Nov 2018 14:17:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1430/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Nov 2018 13:42:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/11984/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11984/2//COMMIT_MSG@12
PS2, Line 12: 1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC
Maybe you could mention that it stores the number of nanoseconds since the unix epoch. That's why it represents this strange time-range.


http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
File fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java:

http://gerrit.cloudera.org:8080/#/c/11984/2/fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java@248
PS2, Line 248:     LogicalTypeAnnotation logicalType = parquetType.getLogicalTypeAnnotation();
What happens when there is no "logical type" in a Parquet file, but only "converted type"?



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Nov 2018 15:10:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148: 
> I have added convert_int96_timestamps as a parameter to GetTimestampInfoFro
I see your problem now as this looks suboptimal in my opinion too.

I don't want to hold you up with this nitpicking, but what if you saved the incoming convert_int96_timestamps constructor parameter to a member of ParquetTimestampDecoder (defaulting to false)? Then you don't have to pass it as a parameter to GetTimestampInfoFromSchema() and this part of the code would look cleaner.
What do you think?



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Dec 2018 11:10:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11984

to look at the new patch set (#6).

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision that
stores timestamps as nanoseconds since the Unix epoch.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 174 insertions(+), 69 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/6
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1443/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Nov 2018 16:20:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 4:

Patch set 4 is a rebase + trivial conflict resolution.
The rebase included https://gerrit.cloudera.org/#/c/11949/, which moved parquet*.h/.cc files to a new directory.


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Nov 2018 12:28:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 6:

(1 comment)

Carry +2

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148: 
> Sure, I'm convinced. Sorry for the extra few rounds :)
No problem, this is not an urgent issue, I do not know of any released software that writes int64 nanos at the moment.



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Dec 2018 16:10:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11984

to look at the new patch set (#3).

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision that
stores timestamps as nanoseconds since the Unix epoch.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet-common.cc
M be/src/exec/parquet-common.h
M be/src/exec/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 174 insertions(+), 70 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/3
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 4:

(3 comments)

Thanks for taking care of my comments! There is one minor thing to sort out and then I'm fine with this change.

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h
File be/src/exec/parquet-common.h:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.h@479
PS2, Line 479: 
> There is already a function with nanosecond precision, and it can create th
That piece of information was missing for me. Thanks!


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc
File be/src/exec/parquet-common.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-common.cc@148
PS2, Line 148: 
> I thought that doing it this way is the lesser evil - convert_int96_timesta
Still, if you have a function to set variables based on the schema, that for me it makes sense to move everything into that function Don't see the benefit to leave the convert_int96_timestamps here. I might iss something here, though.


http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc
File be/src/exec/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/11984/2/be/src/exec/parquet-metadata-utils.cc@71
PS2, Line 71: 
> Before this change the two functions were separate, but with the addition o
Thanks for the explanation!



-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Dec 2018 11:24:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11984

to look at the new patch set (#4).

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision that
stores timestamps as nanoseconds since the Unix epoch.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
---
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 174 insertions(+), 69 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11984/4
-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 6: Code-Review+2

Patch set 6 is revert to patch set 4 + fixing a duplicated word in a comment.

Carry +2 from Gabor


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Dec 2018 16:04:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3542/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Dec 2018 11:48:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Dec 2018 11:48:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11984 )

Change subject: IMPALA-7853: Add support to read int64 NANO timestamps from Parquet
......................................................................

IMPALA-7853: Add support to read int64 NANO timestamps from Parquet

PARQUET-1387 added int64 timestamps with nanosecond precision that
stores timestamps as nanoseconds since the Unix epoch.
As 64 bits are not enough to represent the whole 1400..9999 range
of Impala timestamps, this new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary
during scanning, as every possible 64 bit value represents a valid
timestamp in Impala. This may mean that this has the potential be
the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can
be only described with new logical types in Parquet, it has no
converted type equivalent. This made implementing CREATE TABLE
LIKE PARQUET less trivial than it was for MICRO/MILLI: the type
conversion logic in ParquetHelper.java had to be rewritten to
use LogicalTypeAnnotation instead of ConvertedType.

The changes on Java side also made bumping CDH_BUILD_NUMBER
necessary.

Testing:
- added a new testfile with int64 nano timestamps
- ran core tests

Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Reviewed-on: http://gerrit.cloudera.org:8080/11984
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M bin/impala-config.sh
M common/thrift/parquet.thrift
M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
M testdata/data/README
A testdata/data/int64_timestamps_nano.parquet
M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_scanners.py
13 files changed, 174 insertions(+), 69 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/11984
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I932396d8646f43c0b9ca4a6359f164c4d8349d8f
Gerrit-Change-Number: 11984
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>