You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@omid.apache.org by yonigottesman <gi...@git.apache.org> on 2018/10/10 05:53:34 UTC
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
GitHub user yonigottesman opened a pull request:
https://github.com/apache/incubator-omid/pull/46
[OMID-90] Integrate omid low latency to phoenix changes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yonigottesman/incubator-omid phoenix_ll_integration
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-omid/pull/46.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #46
----
commit 7647733a50431f2b8f5d4495310db59932c46e06
Author: Yonatan Gottesman <yo...@...>
Date: 2018-10-10T05:49:28Z
Integrate omid low latency to phoenix changes
----
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223975170
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether the transaction was invalidated
- Optional<CommitTimestamp> commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
- if (commitTimeStamp.isPresent()) {
- return commitTimeStamp.get();
+ boolean invalidatedByOther = false;
+ Optional<CommitTimestamp> commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+ if (commitTimestampFromCT.isPresent()) {
+ if (isLowLatency && !commitTimestampFromCT.get().isValid())
+ invalidatedByOther = true;
+ else
+ return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
- commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+ Optional<CommitTimestamp> commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+ // In case of LL, if found invalid ct cell, still must check sc in stage 3 then return
+ if (invalidatedByOther) {
+ assert(!commitTimestampFromCT.get().isValid());
+ return commitTimestampFromCT.get();
+ }
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous epoch (previous TSO)
- if (cellStartTimestamp < epoch) {
+ if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated = commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+ // If we are running lowLatency Omid, we could have manged to invalidate a ct entry,
+ // but the committing client already wrote to shadow cells:
--- End diff --
This can happen only in low latency mode, since in the regular mode the client keeps the commit table entry if persisting the commit was done after the tso lost its lease.
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223974301
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether the transaction was invalidated
- Optional<CommitTimestamp> commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
- if (commitTimeStamp.isPresent()) {
- return commitTimeStamp.get();
+ boolean invalidatedByOther = false;
+ Optional<CommitTimestamp> commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+ if (commitTimestampFromCT.isPresent()) {
+ if (isLowLatency && !commitTimestampFromCT.get().isValid())
+ invalidatedByOther = true;
+ else
+ return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
- commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+ Optional<CommitTimestamp> commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+ // In case of LL, if found invalid ct cell, still must check sc in stage 3 then return
+ if (invalidatedByOther) {
+ assert(!commitTimestampFromCT.get().isValid());
+ return commitTimestampFromCT.get();
+ }
+
// 4) Check the epoch and invalidate the entry
--- End diff --
or invalidate in a non latency mode
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by yonigottesman <gi...@git.apache.org>.
Github user yonigottesman closed the pull request at:
https://github.com/apache/incubator-omid/pull/46
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223975342
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -240,7 +261,8 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
CellUtil.cloneQualifier(cell),
cell.getTimestamp()),
commitCache,
- tableAccessWrapper));
+ tableAccessWrapper),
+ isLowLatency);
--- End diff --
indent
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223971768
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether the transaction was invalidated
--- End diff --
This "If the data was written at a previous epoch," can be removed.
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223976065
--- Diff: hbase-client/src/test/java/org/apache/omid/transaction/TestEndToEndScenariosWithHA.java ---
@@ -287,7 +287,7 @@ public void testScenario1() throws Exception {
// TX 2 modifies cells R1C1 & R2C2 (v2)
// TX 2 commits
// End of Test state: R1C1 & R2C2 (v2)
- @Test(timeOut = 60_000)
+ @Test(timeOut = 60_000000)
--- End diff --
Why this is needed? Due to the low_cpu mode? or you just forgot it? :)
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223974621
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether the transaction was invalidated
- Optional<CommitTimestamp> commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
- if (commitTimeStamp.isPresent()) {
- return commitTimeStamp.get();
+ boolean invalidatedByOther = false;
+ Optional<CommitTimestamp> commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+ if (commitTimestampFromCT.isPresent()) {
+ if (isLowLatency && !commitTimestampFromCT.get().isValid())
+ invalidatedByOther = true;
+ else
+ return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
- commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+ Optional<CommitTimestamp> commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+ // In case of LL, if found invalid ct cell, still must check sc in stage 3 then return
+ if (invalidatedByOther) {
+ assert(!commitTimestampFromCT.get().isValid());
+ return commitTimestampFromCT.get();
+ }
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous epoch (previous TSO)
- if (cellStartTimestamp < epoch) {
+ if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated = commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+ // If we are running lowLatency Omid, we could have manged to invalidate a ct entry,
--- End diff --
managed
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r224179421
--- Diff: hbase-client/src/test/java/org/apache/omid/transaction/TestOmidLLRaces.java ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+
+import static com.google.common.base.Charsets.UTF_8;
+import static org.apache.omid.committable.hbase.HBaseCommitTableConfig.DEFAULT_COMMIT_TABLE_CF_NAME;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+
+import com.google.common.base.Optional;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.omid.committable.CommitTable;
+import org.apache.omid.committable.hbase.KeyGenerator;
+import org.apache.omid.committable.hbase.KeyGeneratorImplementations;
+import org.apache.omid.metrics.NullMetricsProvider;
+import org.apache.omid.tso.client.OmidClientConfiguration;
+import org.apache.omid.tso.client.TSOClient;
+import org.apache.omid.tso.client.TSOFuture;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.testng.ITestContext;
+import org.testng.annotations.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+
+@Test(groups = "sharedHBase")
+public class TestOmidLLRaces extends OmidTestBase {
+
+ private static final byte[] row1 = Bytes.toBytes("test-is-committed1");
+ private static final byte[] row2 = Bytes.toBytes("test-is-committed2");
+ private static final byte[] family = Bytes.toBytes(TEST_FAMILY);
+ private static final byte[] qualifier = Bytes.toBytes("testdata");
+ private static final byte[] data1 = Bytes.toBytes("testWrite-1");
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestOmidLLRaces.class);
+ @Override
+ protected boolean isLowLatency() {
+ return true;
+ }
+
+ @Test(timeOut = 30_000)
+ public void testIsCommitted(ITestContext context) throws Exception {
+ AbstractTransactionManager tm = (AbstractTransactionManager)newTransactionManagerHBaseCommitTable(getClient(context));
+
+ Table htable = connection.getTable(TableName.valueOf(TEST_TABLE));
+ SnapshotFilterImpl snapshotFilter = new SnapshotFilterImpl(new HTableAccessWrapper(htable, htable),
+ tm.getCommitTableClient());
+ TTable table = spy(new TTable(htable, snapshotFilter, false));
+
+ HBaseTransaction t1 = (HBaseTransaction) tm.begin();
+
+ Put put = new Put(row1);
+ put.addColumn(family, qualifier, data1);
+ table.put(t1, put);
+ tm.commit(t1);
+
+ HBaseTransaction t2 = (HBaseTransaction) tm.begin();
+ put = new Put(row2);
+ put.addColumn(family, qualifier, data1);
+ table.put(t2, put);
+ table.flushCommits();
+
+ HBaseTransaction t3 = (HBaseTransaction) tm.begin();
+ put = new Put(row2);
+ put.addColumn(family, qualifier, data1);
+ table.put(t3, put);
+ tm.commit(t3);
+
+ HBaseCellId hBaseCellId1 = new HBaseCellId(table, row1, family, qualifier, t1.getStartTimestamp());
+ HBaseCellId hBaseCellId2 = new HBaseCellId(table, row2, family, qualifier, t2.getStartTimestamp());
+ HBaseCellId hBaseCellId3 = new HBaseCellId(table, row2, family, qualifier, t3.getStartTimestamp());
+
+ assertTrue(snapshotFilter.isCommitted(hBaseCellId1, 0, false), "row1 should be committed");
+ assertFalse(snapshotFilter.isCommitted(hBaseCellId2, 0, false), "row2 should not be committed for kv2");
+ assertTrue(snapshotFilter.isCommitted(hBaseCellId3, 0, false), "row2 should be committed for kv3");
+ assertTrue(tm.isLowLatency());
+ }
+
+
+ @Test(timeOut = 30_000)
+ public void testInvalidation(ITestContext context) throws Exception {
+ AbstractTransactionManager tm = (AbstractTransactionManager)newTransactionManagerHBaseCommitTable(getClient(context));
+
+ Table htable = connection.getTable(TableName.valueOf(TEST_TABLE));
+ SnapshotFilterImpl snapshotFilter = new SnapshotFilterImpl(new HTableAccessWrapper(htable, htable),
+ tm.getCommitTableClient());
+ TTable table = spy(new TTable(htable, snapshotFilter, false));
+
+ HBaseTransaction t1 = (HBaseTransaction) tm.begin();
+ Put put = new Put(row1);
+ put.addColumn(family, qualifier, data1);
+ table.put(t1, put);
+
+ HBaseTransaction t2 = (HBaseTransaction) tm.begin();
+ Get get = new Get(row1);
+ get.addColumn(family, qualifier);
+ table.get(t2,get);
+
+ //assert there is an invalidation marker:
+ Table commitTable = connection.getTable(TableName.valueOf("OMID_COMMIT_TABLE"));
+ KeyGenerator keygen = KeyGeneratorImplementations.defaultKeyGenerator();
+ byte[] row = keygen.startTimestampToKey(t1.getStartTimestamp());
+ Get getInvalidation = new Get(row);
+ getInvalidation.addColumn(Bytes.toBytes(DEFAULT_COMMIT_TABLE_CF_NAME),"IT".getBytes(UTF_8));
+ Result res = commitTable.get(getInvalidation);
+ int val = Bytes.toInt(res.getValue(Bytes.toBytes(DEFAULT_COMMIT_TABLE_CF_NAME), "IT".getBytes(UTF_8)));
+ assertTrue(val == 1);
+
+ boolean gotInvalidated = false;
+ try {
+ tm.commit(t1);
+ } catch (RollbackException e) {
+ gotInvalidated = true;
+ }
+ assertTrue(gotInvalidated);
+ tm.commit(t2);
+ Thread.sleep(1000);
+ res = commitTable.get(getInvalidation);
+ assertTrue(res.isEmpty());
+ assertTrue(tm.isLowLatency());
+ }
+
+// @Test(timeOut = 30_000000)
+// public void testReadRace(ITestContext context) throws Exception {
+//
+// final CountDownLatch waitForCommit = new CountDownLatch(1);
+// final CountDownLatch latch1 = new CountDownLatch(1);
+// final CountDownLatch latch2 = new CountDownLatch(1);
+// final Integer[] shadowCellReads = {0};
+//
+// AbstractTransactionManager tm = (AbstractTransactionManager)newTransactionManager(context);
+// Table htable = connection.getTable(TableName.valueOf(TEST_TABLE));
+// SnapshotFilterImpl snapshotFilter = spy(new SnapshotFilterImpl(new HTableAccessWrapper(htable, htable),
+// tm.getCommitTableClient()));
+// TTable table = spy(new TTable(htable, snapshotFilter, false));
+//
+// doAnswer(new Answer<Optional<CommitTable.CommitTimestamp>>() {
+// @Override
+// public Optional<CommitTable.CommitTimestamp> answer(InvocationOnMock invocation) throws Throwable {
+// shadowCellReads[0]++;
+// if (shadowCellReads[0] == 1) {
+// Optional<CommitTable.CommitTimestamp> result = (Optional<CommitTable.CommitTimestamp>) invocation.callRealMethod();
+// latch1.countDown();
+// latch2.await();
+// return result;
+// } else {
+// Optional<CommitTable.CommitTimestamp> result = (Optional<CommitTable.CommitTimestamp>) invocation.callRealMethod();
+// latch1.countDown();
+// latch2.await();
+// return result;
+// }
+// }
+// }).when(snapshotFilter).readCommitTimestampFromShadowCell(any(long.class), any(CommitTimestampLocator.class));
+//
+// doAnswer(new Answer<TSOFuture<Long>>() {
+// @Override
+// public TSOFuture<Long> answer(InvocationOnMock invocationOnMock) throws Throwable {
+//
+// TSOFuture<Long> res = (TSOFuture<Long>)invocationOnMock.callRealMethod();
+//
+// LOG.info("writer thread commit ts {}",1);
+// waitForCommit.countDown();
+// return res;
+// }
+// }).when(client).commit(any(long.class), any(Set.class));
+//
+// Thread readThread = new Thread("Read Thread") {
+// @Override
+// public void run() {
+// try {
+// waitForCommit.await();
+// HBaseTransaction t2 = (HBaseTransaction) tm.begin();
+// LOG.info("reader thread ts {}", t2.getStartTimestamp());
+// Get get = new Get(row1);
+// get.addColumn(family, qualifier);
+// table.get(t2,get);
+// tm.commit(t2);
+// } catch (IOException e) {
+// e.printStackTrace();
+// } catch (RollbackException e) {
+// e.printStackTrace();
+// } catch (InterruptedException e) {
+// e.printStackTrace();
+// }
+// }
+// };
+//
+// HBaseTransaction t1 = (HBaseTransaction) tm.begin();
+// Put put = new Put(row1);
+// put.addColumn(family, qualifier, data1);
+// table.put(t1, put);
+//
+// readThread.start();
+//
+// //latch1.await();
+//
+//
+//
+// boolean gotInvalidated = false;
+// try {
+// tm.commit(t1);
+// } catch (RollbackException e) {
+// gotInvalidated = true;
+// }
+//
+// latch2.countDown();
+// assertFalse(gotInvalidated);
+//
+//
+// assertTrue(tm.isLowLatency());
+// }
--- End diff --
Minor nit: remove commented out code (or comment back in)
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223993765
--- Diff: tso-server/src/main/java/org/apache/omid/tso/RequestProcessorSkipCT.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.tso;
+
+import com.google.inject.Inject;
+import org.apache.omid.metrics.MetricsRegistry;
+import org.jboss.netty.channel.Channel;
+
+import java.io.IOException;
+
+public class RequestProcessorSkipCT extends AbstractRequestProcessor {
+
+
+ private final ReplyProcessor replyProcessor;
+
+ private final LeaseManagement leaseManager;
+ private final Panicker panicker;
+ private final String tsoHostAndPort;
+
+ @Inject
+ RequestProcessorSkipCT(MetricsRegistry metrics,
+ TimestampOracle timestampOracle,
+ ReplyProcessor replyProcessor,
+ Panicker panicker,
+ LeaseManagement leaseManager,
+ TSOServerConfig config,
+ LowWatermarkWriter lowWatermarkWriter,
+ String tsoHostAndPort) throws IOException {
+ super(metrics, timestampOracle, panicker, config, lowWatermarkWriter);
+ this.replyProcessor = replyProcessor;
+ this.tsoHostAndPort = tsoHostAndPort;
+ requestRing = disruptor.start();
+ this.leaseManager = leaseManager;
+ this.panicker = panicker;
+ }
+
+ private void commitSuicideIfNotMaster() {
+ if (!leaseManager.stillInLeasePeriod()) {
+ panicker.panic("Replica " + tsoHostAndPort + " lost mastership whilst flushing data. Committing suicide");
+ }
+ }
+
+ @Override
+ public void forwardCommit(long startTimestamp, long commitTimestamp, Channel c, MonitoringContext monCtx) {
+ commitSuicideIfNotMaster();
--- End diff --
Add a comment that his is required since returning commit results when the TSO is not the leader might violate snapshot isolation. This is because we have to guarantee that committing transaction from previous tso has to persist all its data because a new transaction started by a new tso.
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223989230
--- Diff: transaction-client/src/main/java/org/apache/omid/transaction/AbstractTransactionManager.java ---
@@ -350,6 +356,43 @@ private void markReadOnlyTransaction(AbstractTransaction<? extends CellId> readO
}
+ private void commitLowLatencyTransaction(AbstractTransaction<? extends CellId> tx)
+ throws RollbackException, TransactionException {
+ try {
+
+ long commitTs = tsoClient.commit(tx.getStartTimestamp(), tx.getWriteSet(), tx.getConflictFreeWriteSet()).get();
+ boolean committed = commitTableWriter.atomicAddCommittedTransaction(tx.getStartTimestamp(),commitTs);
+ if (!committed) {
+ // Transaction has been invalidated by other client
+ rollback(tx);
+ commitTableClient.completeTransaction(tx.getStartTimestamp());
+ rolledbackTxsCounter.inc();
+ throw new RollbackException("Transaction " + tx.getTransactionId() + " got invalidated");
+ }
+ certifyCommitForTx(tx, commitTs);
+ updateShadowCellsAndRemoveCommitTableEntry(tx, postCommitter);
+
+ } catch (ExecutionException e) {
+ if (e.getCause() instanceof AbortException) { // TSO reports Tx conflicts as AbortExceptions in the future
+ rollback(tx);
+ rolledbackTxsCounter.inc();
+ throw new RollbackException("Conflicts detected in tx writeset", e.getCause());
+ }
+
+ if (e.getCause() instanceof ServiceUnavailableException || e.getCause() instanceof ConnectionException) {
+ errorTxsCounter.inc();
+ rollback(tx); // Rollback proactively cause it's likely that a new TSOServer is now master
--- End diff --
the leader
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223973121
--- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java ---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether the transaction was invalidated
- Optional<CommitTimestamp> commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
- if (commitTimeStamp.isPresent()) {
- return commitTimeStamp.get();
+ boolean invalidatedByOther = false;
+ Optional<CommitTimestamp> commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+ if (commitTimestampFromCT.isPresent()) {
+ if (isLowLatency && !commitTimestampFromCT.get().isValid())
+ invalidatedByOther = true;
+ else
+ return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
- commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+ Optional<CommitTimestamp> commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+ // In case of LL, if found invalid ct cell, still must check sc in stage 3 then return
--- End diff --
Please add this comment: "This check is not needed in a non low latency mode, since during post commit, the client keeps the entry in the commit table when persisting the commit was done after the TSO lost its lease."
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Posted by ohadshacham <gi...@git.apache.org>.
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223975555
--- Diff: hbase-client/src/test/java/org/apache/omid/transaction/OmidTestBase.java ---
@@ -89,6 +88,8 @@ public void beforeGroups(ITestContext context) throws Exception {
TSOServerConfig tsoConfig = new TSOServerConfig();
tsoConfig.setPort(1234);
tsoConfig.setConflictMapSize(1000);
+ tsoConfig.setLowLatency(isLowLatency());
+ tsoConfig.setWaitStrategy("LOW_CPU");
--- End diff --
nice
---