You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tephra.apache.org by gokulavasan <gi...@git.apache.org> on 2017/01/26 21:11:18 UTC
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
GitHub user gokulavasan opened a pull request:
https://github.com/apache/incubator-tephra/pull/29
TEPHRA-212 Perform writes to prune state asynchronously
JIRA : https://issues.apache.org/jira/browse/TEPHRA-212
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gokulavasan/incubator-tephra feature/tephra-212
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tephra/pull/29.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #29
----
commit 7e761b42e9720a89bed920ebe81307c7ae01eb92
Author: Gokul Gunasekaran <go...@cask.co>
Date: 2017-01-26T21:10:10Z
TEPHRA-212 Perform writes to prune state asynchronously
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98128325
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/PruneUpperBoundWriter.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Thread that will write the the prune upper bound
+ */
+public class PruneUpperBoundWriter {
+ private static final Logger LOG = LoggerFactory.getLogger(PruneUpperBoundWriter.class);
+ private static final Long PRUNE_CHECK_FREQUENCY = TimeUnit.MINUTES.toMillis(1);
+
+ private final DataJanitorState dataJanitorState;
+ private final ConcurrentMap<byte[], Long> pruneUpperBound;
--- End diff --
Since region id is constant for a co-processor instance, we can pass the region id in the constructor and use an atomic reference here, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98786420
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/PruneUpperBoundWriter.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.tephra.hbase.coprocessor.TransactionProcessor;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Thread that will write the the prune upper bound
+ */
+public class PruneUpperBoundWriter {
+ private static final Log LOG = LogFactory.getLog(TransactionProcessor.class);
+
+ private final DataJanitorState dataJanitorState;
+ private final byte[] regionName;
+ private final long pruneFlushInterval;
+ private final AtomicLong pruneUpperBound;
+ private final AtomicBoolean shouldFlush;
+
+ private Thread flushThread;
+ private long lastChecked;
+
+ public PruneUpperBoundWriter(DataJanitorState dataJanitorState, byte[] regionName, long pruneFlushInterval) {
+ this.dataJanitorState = dataJanitorState;
+ this.regionName = regionName;
+ this.pruneFlushInterval = pruneFlushInterval;
+ this.pruneUpperBound = new AtomicLong();
+ this.shouldFlush = new AtomicBoolean(false);
+ startFlushThread();
+ }
+
+ public boolean isAlive() {
+ return flushThread.isAlive();
+ }
+
+ public void persistPruneEntry(long pruneUpperBound) {
+ this.pruneUpperBound.set(pruneUpperBound);
+ this.shouldFlush.set(true);
+ }
+
+ public void stop() {
+ if (flushThread != null) {
+ flushThread.interrupt();
+ }
+ }
+
+ private void startFlushThread() {
+ flushThread = new Thread("tephra-prune-upper-bound-writer") {
+ @Override
+ public void run() {
+ while (!isInterrupted()) {
+ long now = System.currentTimeMillis();
+ if (now > (lastChecked + pruneFlushInterval)) {
+ if (shouldFlush.compareAndSet(true, false)) {
+ // should flush data
+ try {
+ dataJanitorState.savePruneUpperBoundForRegion(regionName, pruneUpperBound.get());
+ } catch (IOException ex) {
+ LOG.warn("Cannot record prune upper bound in table after compacting region " + ex.getMessage());
--- End diff --
Pass the Throwable to the logger here too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98780450
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java ---
@@ -209,6 +212,31 @@ public void destroy() {
}
}
+ /**
+ * Create the prune state table given the {@link TableName} if the table doesn't exist already.
+ *
+ * @param stateTable prune state table name
+ */
+ protected void createPruneTable(TableName stateTable) throws IOException {
+ try (Admin admin = this.connection.getAdmin()) {
+ HTableDescriptor htd = new HTableDescriptor(stateTable);
+ htd.addFamily(new HColumnDescriptor(DataJanitorState.FAMILY).setMaxVersions(1));
+ admin.createTable(htd);
--- End diff --
Since this happens on all regions on every startup, I think it would be better to check for existence first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
@poornachandra Addressed comments. Please take a look again. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
@poornachandra Please review when you get a chance. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98786621
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/PruneUpperBoundWriter.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.tephra.hbase.coprocessor.TransactionProcessor;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Thread that will write the the prune upper bound
+ */
+public class PruneUpperBoundWriter {
+ private static final Log LOG = LogFactory.getLog(TransactionProcessor.class);
+
+ private final DataJanitorState dataJanitorState;
+ private final byte[] regionName;
+ private final long pruneFlushInterval;
+ private final AtomicLong pruneUpperBound;
+ private final AtomicBoolean shouldFlush;
+
+ private Thread flushThread;
+ private long lastChecked;
+
+ public PruneUpperBoundWriter(DataJanitorState dataJanitorState, byte[] regionName, long pruneFlushInterval) {
+ this.dataJanitorState = dataJanitorState;
+ this.regionName = regionName;
+ this.pruneFlushInterval = pruneFlushInterval;
+ this.pruneUpperBound = new AtomicLong();
+ this.shouldFlush = new AtomicBoolean(false);
+ startFlushThread();
+ }
+
+ public boolean isAlive() {
+ return flushThread.isAlive();
+ }
+
+ public void persistPruneEntry(long pruneUpperBound) {
+ this.pruneUpperBound.set(pruneUpperBound);
+ this.shouldFlush.set(true);
+ }
+
+ public void stop() {
+ if (flushThread != null) {
+ flushThread.interrupt();
+ }
+ }
+
+ private void startFlushThread() {
+ flushThread = new Thread("tephra-prune-upper-bound-writer") {
+ @Override
+ public void run() {
+ while (!isInterrupted()) {
+ long now = System.currentTimeMillis();
+ if (now > (lastChecked + pruneFlushInterval)) {
+ if (shouldFlush.compareAndSet(true, false)) {
+ // should flush data
+ try {
+ dataJanitorState.savePruneUpperBoundForRegion(regionName, pruneUpperBound.get());
+ } catch (IOException ex) {
+ LOG.warn("Cannot record prune upper bound in table after compacting region " + ex.getMessage());
--- End diff --
Also it would be good to have the region name and the state table name in the exception message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
On Hold: Investigating Travis build failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98804704
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/coprocessor/TransactionProcessor.java ---
@@ -324,9 +326,14 @@ public InternalScanner preCompactScannerOpen(ObserverContext<RegionCoprocessorEn
TxConstants.TransactionPruning.DEFAULT_PRUNE_ENABLE);
String pruneTable = conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE);
- compactionState = new CompactionState(c.getEnvironment(), TableName.valueOf(pruneTable));
- LOG.debug("Automatic invalid list pruning is enabled. Compaction state will be recorded in table " +
- pruneTable);
+ long pruneFlushInterval = TimeUnit.SECONDS.toMillis(
--- End diff --
This line, and the line above getting `pruneTable` can also move into the if condition, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98782081
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/coprocessor/TransactionProcessor.java ---
@@ -324,7 +326,10 @@ public InternalScanner preCompactScannerOpen(ObserverContext<RegionCoprocessorEn
TxConstants.TransactionPruning.DEFAULT_PRUNE_ENABLE);
String pruneTable = conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE);
- compactionState = new CompactionState(c.getEnvironment(), TableName.valueOf(pruneTable));
+ long pruneFlushInterval = TimeUnit.SECONDS.toMillis(
+ conf.getLong(TxConstants.TransactionPruning.PRUNE_FLUSH_INTERVAL,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_FLUSH_INTERVAL));
+ compactionState = new CompactionState(c.getEnvironment(), TableName.valueOf(pruneTable), pruneFlushInterval);
--- End diff --
Compaction stat needs to be created only if `pruneEnable` is true, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
@poornachandra Thanks for the review. Please review the porting to compat modules when you get a chance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98801214
--- Diff: tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java ---
@@ -135,6 +138,15 @@ private void deletePruneStateTable() throws Exception {
}
}
+ private void truncatePruneStateTable() throws Exception {
+ if (hBaseAdmin.tableExists(pruneStateTable)) {
+ if (hBaseAdmin.isTableEnabled(pruneStateTable)) {
+ hBaseAdmin.disableTable(pruneStateTable);
+ }
+ hBaseAdmin.truncateTable(pruneStateTable, true);
--- End diff --
No, looks like truncateTable is enabling it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98803604
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java ---
@@ -219,13 +219,20 @@ public void destroy() {
*/
protected void createPruneTable(TableName stateTable) throws IOException {
try (Admin admin = this.connection.getAdmin()) {
+ if (admin.tableExists(stateTable)) {
+ LOG.debug("Not creating pruneStateTable {} since it already exists.",
+ stateTable.getNameWithNamespaceInclAsString());
+ return;
+ }
+
HTableDescriptor htd = new HTableDescriptor(stateTable);
htd.addFamily(new HColumnDescriptor(DataJanitorState.FAMILY).setMaxVersions(1));
admin.createTable(htd);
LOG.info("Created pruneTable {}", stateTable.getNameWithNamespaceInclAsString());
} catch (TableExistsException ex) {
- // Expected if the prune state table already exists
- LOG.debug("Not creating pruneStateTable since it already exists. {}", ex.getMessage());
+ // Expected if the prune state table is being created at the same time by another client
+ LOG.debug("Not creating pruneStateTable {} since it already exists. {}",
+ stateTable.getNameWithNamespaceInclAsString(), ex.getMessage());
--- End diff --
Log the throwable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98780134
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/PruneUpperBoundWriter.java ---
@@ -77,7 +78,7 @@ public void run() {
try {
dataJanitorState.savePruneUpperBoundForRegion(regionName, pruneUpperBound.get());
} catch (IOException ex) {
- LOG.warn("Cannot record prune upper bound in table after compacting region", ex);
+ LOG.warn("Cannot record prune upper bound in table after compacting region " + ex.getMessage());
--- End diff --
We can still pass in the Throwable to the logger.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
@poornachandra Addressed comments. Please take a look when you get a chance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98135660
--- Diff: tephra-core/src/main/java/org/apache/tephra/TxConstants.java ---
@@ -369,6 +369,12 @@
* Interval in seconds to schedule prune run.
*/
public static final String PRUNE_INTERVAL = "data.tx.prune.interval";
+
+ /**
+ * Interval in seconds to schedule flush of prune table entries to store.
+ */
+ public static final String PRUNE_FLUSH_INTERVAL = "data.tx.prune.flush.interval";
--- End diff --
Looks like this is not used anywhere. Also I don't think we need this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98779081
--- Diff: tephra-core/src/main/java/org/apache/tephra/TxConstants.java ---
@@ -369,6 +369,12 @@
* Interval in seconds to schedule prune run.
*/
public static final String PRUNE_INTERVAL = "data.tx.prune.interval";
+
+ /**
+ * Interval in seconds to schedule flush of prune table entries to store.
+ */
+ public static final String PRUNE_FLUSH_INTERVAL = "data.tx.prune.flush.interval";
--- End diff --
In the first commit, it was not being used anywhere. Since it is being used now, it is okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #29: TEPHRA-212 Perform writes to prune state asynchr...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/29
@poornachandra Addressed the final comments. Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98127853
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/PruneUpperBoundWriter.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import org.slf4j.Logger;
--- End diff --
Since this code will run in HBase co-processor, use apache commons logging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98135416
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/CompactionState.java ---
@@ -75,18 +76,34 @@ public void record(CompactionRequest request, @Nullable TransactionVisibilitySta
}
/**
+ * Stops the current {@link PruneUpperBoundWriter}.
+ */
+ public void stop() {
+ if (pruneUpperBoundWriter != null) {
+ pruneUpperBoundWriter.stop();
+ }
+ }
+
+ /**
* Persists the transaction state recorded by {@link #record(CompactionRequest, TransactionVisibilityState)}.
* This method is called after the compaction has successfully completed.
*/
public void persist() {
if (pruneUpperBound != -1) {
- try {
- dataJanitorState.savePruneUpperBoundForRegion(regionName, pruneUpperBound);
- LOG.debug(String.format("Saved prune upper bound %s for region %s", pruneUpperBound, regionNameAsString));
- } catch (IOException e) {
- LOG.warn(String.format("Cannot record prune upper bound in table %s after compacting region %s",
- stateTable, regionNameAsString), e);
- }
+ PruneUpperBoundWriter upperBoundWriter = getPruneUpperBoundWriter();
--- End diff --
There are redundant instances of class `PruneUpperBoundWriter` => `upperBoundWriter` and a`pruneUpperBoundWriter`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tephra/pull/29
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98137572
--- Diff: tephra-core/src/main/java/org/apache/tephra/TxConstants.java ---
@@ -369,6 +369,12 @@
* Interval in seconds to schedule prune run.
*/
public static final String PRUNE_INTERVAL = "data.tx.prune.interval";
+
+ /**
+ * Interval in seconds to schedule flush of prune table entries to store.
+ */
+ public static final String PRUNE_FLUSH_INTERVAL = "data.tx.prune.flush.interval";
--- End diff --
It is being used in the PruneUpperBoundWriter. Should we not have this? Then what should be the default flush interval?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #29: TEPHRA-212 Perform writes to prune state ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/29#discussion_r98786925
--- Diff: tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java ---
@@ -135,6 +138,15 @@ private void deletePruneStateTable() throws Exception {
}
}
+ private void truncatePruneStateTable() throws Exception {
+ if (hBaseAdmin.tableExists(pruneStateTable)) {
+ if (hBaseAdmin.isTableEnabled(pruneStateTable)) {
+ hBaseAdmin.disableTable(pruneStateTable);
+ }
+ hBaseAdmin.truncateTable(pruneStateTable, true);
--- End diff --
No need to re-enable the table here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---