You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Michael Brown (Code Review)" <ge...@cloudera.org> on 2016/07/21 00:07:42 UTC
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Michael Brown has uploaded a new change for review.
http://gerrit.cloudera.org:8080/3713
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.
Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 6 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/13/3713/1
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:
PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> I don't have particular insight into the discrepancies between impala and p
Done. There are 86 "no spaces around = in keyword args" violations in this file. I only fixed this one to keep the diff small.
PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> Also, the line is longer than 90 characters.
Done
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:
PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
> I don't have particular insight into the discrepancies between impala and p
Also, the line is longer than 90 characters.
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 3: Verified+1
verified:
- I ran disrepancy_searcher.py by hand and forced an Impala restart. Impala started with the correct option set
- short leopard run also shows Impala starting with correct option
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 3: Code-Review+2
rebase, carry +2
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has submitted this change and it was merged.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.
Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Reviewed-on: http://gerrit.cloudera.org:8080/3713
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Michael Brown <mi...@cloudera.com>
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 18 insertions(+), 6 deletions(-)
Approvals:
Michael Brown: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 1:
I've been using this patch locally for months.
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "David Knupp (Code Review)" <ge...@cloudera.org>.
David Knupp has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
Given that you've been running this for months, I'm going to assume this good enough to ship. Some formatting suggestions only.
http://gerrit.cloudera.org:8080/#/c/3713/1/tests/comparison/leopard/impala_docker_env.py
File tests/comparison/leopard/impala_docker_env.py:
PS1, Line 189: "-convert_legacy_hive_parquet_utc_timestamps=true"
I don't have particular insight into the discrepancies between impala and postgres timestamps, but maybe this line got a little long. Might be worth storing the parts in local vars and then building the string up? If nothing else, then at least for IMPALAD_ARGS?
Also, no spaces around "=" in keyword args.
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
Patch Set 2: Code-Review+2
carry +2 from David
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3892: qgen: always run Impala with -convert legacy hive parquet utc timestamps=true
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Hello David Knupp,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3713
to look at the new patch set (#2).
Change subject: IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
......................................................................
IMPALA-3892: qgen: always run Impala with -convert_legacy_hive_parquet_utc_timestamps=true
To reduce discrepancies in timestamp columns between Impala and
PostgresQL, always run Impala with
-convert_legacy_hive_parquet_utc_timestamps=true . This makes triaging
correctness issues easier.
Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
---
M tests/comparison/discrepancy_searcher.py
M tests/comparison/leopard/impala_docker_env.py
2 files changed, 18 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/13/3713/2
--
To view, visit http://gerrit.cloudera.org:8080/3713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6e0a14780dd9bcc18285f2857625fd3ec1c34913
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>