You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Michael Brown (Code Review)" <ge...@cloudera.org> on 2016/07/21 00:07:42 UTC

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Michael Brown has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/3713

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................

IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true

To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.

Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 6 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/13/3713/1
-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:

PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> I don't have particular insight into the discrepancies between impala and p
Done. There are 86 "no spaces around = in keyword args" violations in this file. I only fixed this one to keep the diff small.


PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> Also, the line is longer than 90 characters.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:

PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> I don't have particular insight into the discrepancies between impala and p
Also, the line is longer than 90 characters.


-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 3: Verified+1

verified:

- I ran disrepancy_searcher.py by hand and forced an Impala restart. Impala started with the correct option set

- short leopard run also shows Impala starting with correct option

-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 3: Code-Review+2

rebase, carry +2

-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has submitted this change and it was merged.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true

To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.

Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Reviewed-on: http://gerrit.cloudera.org:8080/3713
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Michael Brown <mi...@cloudera.com>
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 18 insertions(+), 6 deletions(-)

Approvals:
  Michael Brown: Looks good to me, approved; Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 1:

I've been using this patch locally for months.

-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "David Knupp (Code Review)" <ge...@cloudera.org>.
David Knupp has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 1: Code-Review+2

(1 comment)

Given that you've been running this for months, I'm going to assume this good enough to ship. Some formatting suggestions only.

http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:

PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
I don't have particular insight into the discrepancies between impala and postgres timestamps, but maybe this line got a little long. Might be worth storing the parts in local vars and then building the string up? If nothing else, then at least for IMPALAD_ARGS?

Also, no spaces around "=" in keyword args.


-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................


Patch Set 2: Code-Review+2

carry +2 from David

-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true

Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Hello David Knupp,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/3713

to look at the new patch set (#2).

Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................

IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true

To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.

Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 18 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/13/3713/2
-- 
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>