[jira] [Commented] (PHOENIX-3585) MutableIndexIT testSplitDuringIndexScan and testIndexHalfStoreFileReader fail for transactional tables and local indexes

["Thomas D'Silva (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837216#comment-15837216 ] 

Thomas D'Silva commented on PHOENIX-3585:
-----------------------------------------

[~rajeshbabu]
The test was failing TransactionProcessor.preCompactScannerOpen runs first (because it has a higher priority) and returns an internal scanner which gets passed into the IndexHalfStoreFileReaderGenerator.preCompactScannerOpen() call as the InternalScanner so the LocalIndexStoreFileScanner is never created. 

{code}
@Override
public InternalScanner preCompactScannerOpen(ObserverContext<RegionCoprocessorEnvironment> c,
            Store store, List<? extends KeyValueScanner> scanners, ScanType scanType,
            long earliestPutTs, InternalScanner s, CompactionRequest request) throws IOException {
        if (!store.getFamily().getNameAsString()
                .startsWith(QueryConstants.LOCAL_INDEX_COLUMN_FAMILY_PREFIX)
                || s != null // -----> returns here
                || !store.hasReferences()) {
            return s;
        }
        List<StoreFileScanner> newScanners = new ArrayList<StoreFileScanner>(scanners.size());
        Scan scan = new Scan();
        scan.setMaxVersions(store.getFamily().getMaxVersions());
        boolean scanUsePread = c.getEnvironment().getConfiguration().getBoolean("hbase.storescanner.use.pread", scan.isSmall());
        for(KeyValueScanner scanner: scanners) {
            Reader reader = ((StoreFileScanner) scanner).getReaderForTesting();
            if (reader instanceof IndexHalfStoreFileReader) {
                newScanners.add(new LocalIndexStoreFileScanner(reader, reader.getScanner(
                    scan.getCacheBlocks(), scanUsePread, false), true, reader.getHFileReader()
                        .hasMVCCInfo(), store.getSmallestReadPoint()));
            } else {
                newScanners.add(((StoreFileScanner) scanner));
            }
        }
        return new StoreScanner(store, store.getScanInfo(), scan, newScanners,
            scanType, store.getSmallestReadPoint(), earliestPutTs);
    }
{code}

 [~jamestaylor] 
The TransactionProcessor creates a scan with a filter to filter out invalid data. Do you see any way we can combine the StoreScanner from the TransactionProcessor with that created in IndexHalfStoreFileReaderGenerator


{code}
protected InternalScanner createStoreScanner(RegionCoprocessorEnvironment env, String action,
                                               TransactionVisibilityState snapshot, Store store,
                                               List<? extends KeyValueScanner> scanners, ScanType type,
                                               long earliestPutTs) throws IOException {
    if (snapshot == null) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Region " + env.getRegion().getRegionNameAsString() +
                    ", no current transaction state found, defaulting to normal " + action + " scanner");
      }
      return null;
    }

    // construct a dummy transaction from the latest snapshot
    Transaction dummyTx = TxUtils.createDummyTransaction(snapshot);
    Scan scan = new Scan();
    // need to see all versions, since we filter out excludes and applications may rely on multiple versions
    scan.setMaxVersions();
    scan.setFilter(
        new IncludeInProgressFilter(dummyTx.getVisibilityUpperBound(),
            snapshot.getInvalid(),
            getTransactionFilter(dummyTx, type, null)));

    return new StoreScanner(store, store.getScanInfo(), scan, scanners,
                            type, store.getSmallestReadPoint(), earliestPutTs);
  }
{code}

> MutableIndexIT testSplitDuringIndexScan and testIndexHalfStoreFileReader fail for transactional tables and local indexes
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3585
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3585
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Thomas D'Silva
>         Attachments: diff.patch
>
>
> the tests fail if we use HDFSTransactionStateStorage instead of  InMemoryTransactionStateStorage when we create the TransactionManager in BaseTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)