[jira] [Commented] (PHOENIX-4027) Mark index as disabled during partial rebuild after configurable amount of time

["Rajeshbabu Chintaguntla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167715#comment-16167715 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-4027:
--------------------------------------------------

With this patch partial index rebuild make the index disabled forever in very easily in below situations.
1) When we write the past data with row timestamp columns
2) Sometimes any region inconsistencies introduced take more than 30 mins

When the data is huge creating index or rebuilding complete index might take hours or days. In such cases it's better to rebuild the index in intervals or batches than completely disabling. [~samarthjain] [~jamestaylor] whyt?
{noformat}
                    if (EnvironmentEdgeManager.currentTimeMillis() - Math.abs(indexDisableTimestamp) > indexDisableTimestampThreshold) {
                        /*
                         * It has been too long since the index has been disabled and any future
                         * attempts to reenable it likely will fail. So we are going to mark the
                         * index as disabled and set the index disable timestamp to 0 so that the
                         * rebuild task won't pick up this index again for rebuild.
                         */
                        try {
                            IndexUtil.updateIndexState(conn, indexTableFullName, PIndexState.DISABLE, 0l);
                            LOG.error("Unable to rebuild index " + indexTableFullName
                                    + ". Won't attempt again since index disable timestamp is older than current time by "
                                    + indexDisableTimestampThreshold
                                    + " milliseconds. Manual intervention needed to re-build the index");
                        } catch (Throwable ex) {
                            LOG.error(
                                "Unable to mark index " + indexTableFullName + " as disabled.", ex);
                        }
                        continue; // don't attempt another rebuild irrespective of whether
                                  // updateIndexState worked or not
                    }
{noformat}

> Mark index as disabled during partial rebuild after configurable amount of time
> -------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4027
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4027
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Samarth Jain
>             Fix For: 4.12.0, 4.11.1
>
>         Attachments: PHOENIX-4027_addendum.patch, PHOENIX-4027.patch
>
>
> Instead of marking an index as permanently disabled in the partial index rebuilder when a failure occurs, we should let it try again up to a configurable amount of time. The reason is that the fail-fast approach with the lower RPC timeout will continue to cause a failure until the index region can be written to. This will allow us to ride out region moves without a long RPC time out and thus without holding handler threads for long periods of time. We can base the failure on the INDEX_DISABLE_TIMESTAMP value of an index as we walk through the scan results here in MetaDataRegionObserver. :
> {code}
>                 do {
>                     results.clear();
>                     hasMore = scanner.next(results);
>                     if (results.isEmpty()) break;
>                     Result r = Result.create(results);
>                     byte[] disabledTimeStamp = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
>                         PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES);
>                     byte[] indexState = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
>                             PhoenixDatabaseMetaData.INDEX_STATE_BYTES);
>                     if (disabledTimeStamp == null || disabledTimeStamp.length == 0) {
>                         continue;
>                     }
>                     // TODO: if disabledTimeStamp - System.currentTimeMillis() > configurableAmount 
>                     // then disable the index.
> {code}
> I'd propose we allow 30 minutes to get an index back online.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)