[jira] [Commented] (PHOENIX-3860) Implement TAL functionality for Omid

["James Taylor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382782#comment-16382782 ] 

James Taylor commented on PHOENIX-3860:
---------------------------------------

bq. Could you please refer me to this code? We also have problem with the shadow cells in this case.
The server-side code that does local index population is in UngroupedAggregateRegionObserver here:
{code}
                        } else if (buildLocalIndex) {
                            for (IndexMaintainer maintainer : indexMaintainers) {
                                if (!results.isEmpty()) {
                                    result.getKey(ptr);
                                    ValueGetter valueGetter =
                                            maintainer.createGetterFromKeyValues(
                                                ImmutableBytesPtr.copyBytesIfNecessary(ptr),
                                                results);
                                    Put put = maintainer.buildUpdateMutation(kvBuilder,
                                        valueGetter, ptr, results.get(0).getTimestamp(),
                                        env.getRegion().getRegionInfo().getStartKey(),
                                        env.getRegion().getRegionInfo().getEndKey());
                                    indexMutations.add(put);
                                }
                            }
                            result.setKeyValues(results);
{code}

bq. Do you create an index using a transaction and commit when you finish populate all the data table info to the index? What happened if the commit fails? You drop the index table, take a new fence and start the process again?
No, we don't use a transaction during the initial index population. This population of an index is done synchronously by default, but you can tack on the ASYNC keyword to the CREATE INDEX call and then a MR job will asynchronously build it. Either way, the index is only marked active (and hence used) if the population is completed successfully. If the table is transactional, then we do start and end a transaction solely for the purpose of getting a transaction ID to use for the initial population. If the population fails, then it's up to the user to issue a DROP INDEX and then call CREATE INDEX again (though it'd be nice if we did this cleanup ourselves). One more detail: as the index is being populated, we also perform the index maintenance as writes are done to the table. These mutations may lay down delete markers (our own tx delete marker) for rows that have not yet been populated. This all works out because the mutations will have a later timestamp than the timestamp of the rows created during the initial population.

For the issue with the garbage collector, can we determine in the coprocessor that the index is in a "building" state and make sure that no garbage collection is done? Or perhaps the commit table can be updated right before the index is moved to an active state?

> Implement TAL functionality for Omid
> ------------------------------------
>
>                 Key: PHOENIX-3860
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3860
>             Project: Phoenix
>          Issue Type: Sub-task
>            Reporter: Ohad Shacham
>            Assignee: Ohad Shacham
>            Priority: Major
>
> Implement TAL functionality for Omid in order to be able to use Omid as Phoenix's transaction processing engine. 
> To support the integration jira [OMID-82] was opened that encapsulates all Omid related development for Phoenix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)