You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Allan Yang (JIRA)" <ji...@apache.org> on 2017/01/22 08:38:26 UTC
[jira] [Updated] (HBASE-17506) started mvcc transaction is not
completed in branch-1
[ https://issues.apache.org/jira/browse/HBASE-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allan Yang updated HBASE-17506:
-------------------------------
Description:
In {{doMiniBatchMutation}}, if it is in replay and if the the nonce of the mutation is different, we append them to a different wal. But, after HBASE-14465, we start a mvcc transition in the ringbuffer's append thread. So, every time we append a wal entry, we started a mvcc transition, but we didn't complete the mvcc transition anywhere. This can block other transition of this region.
{code}
// txid should always increase, so having the one from the last call is ok.
// we use HLogKey here instead of WALKey directly to support legacy coprocessors.
walKey = new ReplayHLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, m.getClusterIds(),
currentNonceGroup, currentNonce, mvcc);
txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(), walKey,
walEdit, true);
walEdit = new WALEdit(cellCount, isInReplay);
walKey = null;
{code}
Looked at master branch, there is no such problem. It has a method named{{appendCurrentNonces}} :
{code}
private void appendCurrentNonces(final Mutation mutation, final boolean replay,
final WALEdit walEdit, final long now, final long currentNonceGroup, final long currentNonce)
throws IOException {
if (walEdit.isEmpty()) return;
if (!replay) throw new IOException("Multiple nonces per batch and not in replay");
WALKey walKey = new WALKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, mutation.getClusterIds(),
currentNonceGroup, currentNonce, mvcc, this.getReplicationScope());
this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
// Complete the mvcc transaction started down in append else it will block others
this.mvcc.complete(walKey.getWriteEntry());
}
{code}
Yes, the easiest way to fix branch-1 is to complete the writeEntry like master branch do. But is it really fine to do this?
1. :
complete the mvcc transition before waiting sync will create a disturbance of data visibility.
2.:
In what circumstance will there be different nonce and nonce group in a single wal entry? Nonce are used in append/increment. But in {{batchMuate}} ,we treat them differently and append one wal entry for each of them. So I think no test can reach this code path, that maybe why no one has found this bug(Please tell me if I'm wrong).
was:
In {{doMiniBatchMutation}}, if it is in replay and if the the nonce of the mutation is different, we append them to a different wal. But, after HBASE-14465, we start a mvcc transition in the ringbuffer's append thread. So, every time we append a wal entry, we started a mvcc transition, but we didn't complete the mvcc transition anywhere. This can block other transition of this region.
{code}
// txid should always increase, so having the one from the last call is ok.
// we use HLogKey here instead of WALKey directly to support legacy coprocessors.
walKey = new ReplayHLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, m.getClusterIds(),
currentNonceGroup, currentNonce, mvcc);
txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(), walKey,
walEdit, true);
walEdit = new WALEdit(cellCount, isInReplay);
walKey = null;
{code}
Looked at master branch, there is no such problem. It has a method named{{appendCurrentNonces}} :
{code}
private void appendCurrentNonces(final Mutation mutation, final boolean replay,
final WALEdit walEdit, final long now, final long currentNonceGroup, final long currentNonce)
throws IOException {
if (walEdit.isEmpty()) return;
if (!replay) throw new IOException("Multiple nonces per batch and not in replay");
WALKey walKey = new WALKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, mutation.getClusterIds(),
currentNonceGroup, currentNonce, mvcc, this.getReplicationScope());
this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
// Complete the mvcc transaction started down in append else it will block others
this.mvcc.complete(walKey.getWriteEntry());
}
{code}
Yes, the easiest way to fix branch-1 is to complete the writeEntry like master branch do. But is it really fine to do this?
1. Question 1:
complete the mvcc transition before waiting sync will create a disturbance of data visibility.
2.Question 2:
In what circumstance will there be different nonce and nonce group in a single wal entry? Nonce are used in append/increment. But in {{batchMuate}} ,we treat them differently and append one wal entry for each of them. So I think no test can reach this code path, that maybe why no one has found this bug(Please tell me if I'm wrong).
> started mvcc transaction is not completed in branch-1
> -----------------------------------------------------
>
> Key: HBASE-17506
> URL: https://issues.apache.org/jira/browse/HBASE-17506
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Allan Yang
> Assignee: Allan Yang
>
> In {{doMiniBatchMutation}}, if it is in replay and if the the nonce of the mutation is different, we append them to a different wal. But, after HBASE-14465, we start a mvcc transition in the ringbuffer's append thread. So, every time we append a wal entry, we started a mvcc transition, but we didn't complete the mvcc transition anywhere. This can block other transition of this region.
> {code}
> // txid should always increase, so having the one from the last call is ok.
> // we use HLogKey here instead of WALKey directly to support legacy coprocessors.
> walKey = new ReplayHLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
> this.htableDescriptor.getTableName(), now, m.getClusterIds(),
> currentNonceGroup, currentNonce, mvcc);
> txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(), walKey,
> walEdit, true);
> walEdit = new WALEdit(cellCount, isInReplay);
> walKey = null;
> {code}
> Looked at master branch, there is no such problem. It has a method named{{appendCurrentNonces}} :
> {code}
> private void appendCurrentNonces(final Mutation mutation, final boolean replay,
> final WALEdit walEdit, final long now, final long currentNonceGroup, final long currentNonce)
> throws IOException {
> if (walEdit.isEmpty()) return;
> if (!replay) throw new IOException("Multiple nonces per batch and not in replay");
> WALKey walKey = new WALKey(this.getRegionInfo().getEncodedNameAsBytes(),
> this.htableDescriptor.getTableName(), now, mutation.getClusterIds(),
> currentNonceGroup, currentNonce, mvcc, this.getReplicationScope());
> this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
> // Complete the mvcc transaction started down in append else it will block others
> this.mvcc.complete(walKey.getWriteEntry());
> }
> {code}
> Yes, the easiest way to fix branch-1 is to complete the writeEntry like master branch do. But is it really fine to do this?
> 1. :
> complete the mvcc transition before waiting sync will create a disturbance of data visibility.
> 2.:
> In what circumstance will there be different nonce and nonce group in a single wal entry? Nonce are used in append/increment. But in {{batchMuate}} ,we treat them differently and append one wal entry for each of them. So I think no test can reach this code path, that maybe why no one has found this bug(Please tell me if I'm wrong).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)