[jira] [Resolved] (IMPALA-6124) test_last_ddl_time_update fails on S3

["Lars Volker (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Volker resolved IMPALA-6124.
---------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.11.0

IMPALA-6124: Fix alter table ddl updates and test

Impala would previously update the ddl time of a table when dropping a
partition but not when adding one. This change removes updates to the
ddl time when partitions are added or removed to be consistent with
Hive.

Additionally the check in the ddl update test would fail if some
operations took longer than 20 seconds. Instead, this change makes sure
that the ddl time increases as intended.

To test this change I ran test_last_ddl_time_update in exhaustive mode
and also ran a private S3 build.

Change-Id: I3126252e7709304d3e1fa4bb06a0b847180bd6cf
Reviewed-on: http://gerrit.cloudera.org:8080/8411
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins

> test_last_ddl_time_update fails on S3
> -------------------------------------
>
>                 Key: IMPALA-6124
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6124
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Infrastructure
>    Affects Versions: Impala 2.11.0
>            Reporter: Lars Volker
>            Assignee: Lars Volker
>            Priority: Blocker
>             Fix For: Impala 2.11.0
>
>
> This looks like the test took longer than 20s and thus failed. I don't understand why the two timestamps should be within 20s. My suggestion would be to change the check to something like this:
> {code:python}
> assert long(afterDdlTime) - long(beforeDdlTime) > 0
> {code}
> [~alex.behm] - You wrote the original code. Do you remember why we perform this check? If you agree with the change I suggested, I'll go ahead and prepare a patch.
> {noformat}05:53:22.493 =================================== FAILURES ===================================
> 05:53:22.493  TestLastDdlTimeUpdate.test_alter[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
> 05:53:22.493 [gw0] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/bin/../infra/python/env/bin/python
> 05:53:22.493 metadata/test_last_ddl_time_update.py:57: in test_alter
> 05:53:22.493     unique_database, TBL_NAME, True)
> 05:53:22.493 metadata/test_last_ddl_time_update.py:120: in run_test
> 05:53:22.493     assert long(afterDdlTime) - long(beforeDdlTime) <= 20
> 05:53:22.493 E   assert (1509024836L - 1509024790L) <= 20
> 05:53:22.493 E    +  where 1509024836L = long('1509024836')
> 05:53:22.493 E    +  and   1509024790L = long('1509024790')
> 05:53:22.493 ---------------------------- Captured stderr setup -----------------------------
> 05:53:22.493 -- connecting to: localhost:21000
> 05:53:22.493 SET sync_ddl=False;
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 DROP DATABASE IF EXISTS `test_alter_46f96e42` CASCADE;
> 05:53:22.493 
> 05:53:22.493 SET sync_ddl=False;
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 CREATE DATABASE `test_alter_46f96e42`;
> 05:53:22.493 
> 05:53:22.493 MainThread: Created database "test_alter_46f96e42" for test ID "metadata/test_last_ddl_time_update.py::TestLastDdlTimeUpdate::()::test_alter[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]"
> 05:53:22.493 ----------------------------- Captured stderr call -----------------------------
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 create external table test_alter_46f96e42.alter_test_tbl (i int) partitioned by (j int, s string);
> 05:53:22.493 
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 alter table test_alter_46f96e42.alter_test_tbl add partition (j=1, s='2012');
> 05:53:22.493 
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 alter table test_alter_46f96e42.alter_test_tbl add if not exists partition (j=1, s='2012');
> 05:53:22.493 
> 05:53:22.493 -- executing against localhost:21000
> 05:53:22.493 alter table test_alter_46f96e42.alter_test_tbl drop partition (j=1, s='2012');
> 05:53:22.493 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)