[jira] [Comment Edited] (PHOENIX-5018) Index mutations created by IndexTool will have wrong timestamps

["Kadir OZDEMIR (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720593#comment-16720593 ] 

Kadir OZDEMIR edited comment on PHOENIX-5018 at 12/13/18 8:57 PM:
------------------------------------------------------------------

I wrote two integration tests to check if timestamps are set correctly in UPSERT SELECT operations (see the PR). The first test uses UPSERT SELECT to insert rows to a from a data table to another data table. In this case, the destination table should not inherit the timestamps from the source table. This test passes.

The second test creates a data table and populates with rows with different timestamps. And then it creates an index table on this data table (which in turn uses UPSERT SELECT). The test checks if the timestamps of the data table are inherited by the index table. This test fails (i.e., the assert statement at line 910) as Index table rows get new timestamps based on the current wall clock. As far as I can tell, this bug is valid and needs to be fixed. [~gjacoby], [~vincentpoon], [~tdsilva]


was (Author: kozdemir):
I wrote two integration tests to check if timestamps are set correctly in UPSERT SELECT operations (see the PR). The first test uses UPSERT SELECT to insert rows to a from a data table to another data table. In this case, the destination table should not inherit the timestamps from the source table. This test passes.

The second test creates a data table and populates with rows with different timestamps. And then it creates an index table on this data table (which in turn uses UPSERT SELECT). The test checks if the timestamps of the data table are inherited by the index table. This test fails as Index table rows get new timestamps based on the current wall clock. As far as I can tell, this bug is valid and needs to be fixed. [~gjacoby], [~vincentpoon], [~tdsilva]

> Index mutations created by IndexTool will have wrong timestamps
> ---------------------------------------------------------------
>
>                 Key: PHOENIX-5018
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5018
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.14.0, 5.0.0
>            Reporter: Geoffrey Jacoby
>            Assignee: Kadir OZDEMIR
>            Priority: Major
>
> When doing a full rebuild (or initial async build) on an index using the IndexTool and PhoenixIndexImportDirectMapper, we generate the index mutations by creating an UPSERT SELECT query from the base table to the index, then taking the Mutations from it and inserting it directly into the index via an HBase HTable. 
> The timestamps of the Mutations use the default HBase behavior, which is to take the current wall clock. However, the timestamp of an index KeyValue should use the timestamp of the initial KeyValue in the base table.
> Having base table and index timestamps out of sync can cause all sorts of weird side effects, such as if the base table has data with an expired TTL that isn't expired in the index yet. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)