You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "James Taylor (JIRA)" <ji...@apache.org> on 2017/07/31 21:19:00 UTC
[jira] [Commented] (PHOENIX-4052) Create the correct index row
mutations for out-of-order data mutations
[ https://issues.apache.org/jira/browse/PHOENIX-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107973#comment-16107973 ]
James Taylor commented on PHOENIX-4052:
---------------------------------------
The following test, slightly different than the description above, reproduces an issue with an extra row with an invalid row key ending up in the index table:
{code}
@Test
public void testOutOfOrderDelete() throws Exception {
String tableName = generateUniqueName();
String indexName = generateUniqueName();
Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
long ts = 1000;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
Connection conn = DriverManager.getConnection(getUrl(), props);
conn.createStatement().execute("CREATE TABLE " + tableName + "(k CHAR(1) PRIMARY KEY, v VARCHAR) COLUMN_ENCODED_BYTES = 0");
conn.close();
ts = 1010;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
conn = DriverManager.getConnection(getUrl(), props);
conn.createStatement().execute("CREATE INDEX " + indexName + " ON " + tableName + "(v)");
conn.close();
ts = 1020;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
conn = DriverManager.getConnection(getUrl(), props);
conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','a')");
conn.commit();
conn.close();
ts = 1040;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
conn = DriverManager.getConnection(getUrl(), props);
conn.createStatement().execute("DELETE FROM " + tableName + " WHERE k='a'");
conn.commit();
conn.close();
ts = 1030;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
conn = DriverManager.getConnection(getUrl(), props);
conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','bbb')");
conn.commit();
conn.close();
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(tableName)));
TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(indexName)));
ts = 1050;
props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
conn = DriverManager.getConnection(getUrl(), props);
long count1 = getRowCount(conn, tableName);
long count2 = getRowCount(conn, indexName);
assertTrue("Expected table row count ( " + count1 + ") to match index row count (" + count2 + ")", count1 == count2);
conn.close();
/**
*
************ dumping T000001;hconnection-0x8f57e4c **************
a/0:/1040/DeleteFamily/vlen=0/seqid=0
a/0:V/1030/Put/vlen=3/seqid=0
a/0:V/1020/Put/vlen=1/seqid=0
a/0:_0/1030/Put/vlen=1/seqid=0
a/0:_0/1020/Put/vlen=1/seqid=0
-----------------------------------------------
************ dumping T000002;hconnection-0x8f57e4c **************
\x00a/0:_0/1040/Put/vlen=2/seqid=0
a\x00a/0:/1040/DeleteFamily/vlen=0/seqid=0
a\x00a/0:/1030/DeleteFamily/vlen=0/seqid=0
a\x00a/0:_0/1020/Put/vlen=2/seqid=0
-----------------------------------------------
*/
}
private static long getRowCount(Connection conn, String tableName) throws SQLException {
ResultSet rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX */ count(*) FROM " + tableName);
assertTrue(rs.next());
return rs.getLong(1);
}
{code}
Assertion that occurs is:
{code}
java.lang.AssertionError: Expected table row count ( 0) to match index row count (1)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.phoenix.end2end.index.OutOfOrderMutationsIT.testOutOfOrderDelete(OutOfOrderMutationsIT.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
{code}
> Create the correct index row mutations for out-of-order data mutations
> ----------------------------------------------------------------------
>
> Key: PHOENIX-4052
> URL: https://issues.apache.org/jira/browse/PHOENIX-4052
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
>
> Out-of-order processing of data rows during index maintenance causes mutable indexes to become out of sync with regard to the data table. Here's a simple example to illustrate the issue:
> # Assume table T(K,V) and index X(V,K).
> # Upsert T(A, 1) at t10. Index updates: Put X(1,A) at t10.
> # Upsert T(A, 3) at t30. Index updates: Delete X(1,A) at t29, Put X(3,A) at t30.
> # Upsert T(A,2) at t20. Index updates: Delete X(1,A) at t19, Put X(2,A) at t20, Delete X(2,A) at t29
> Ideally, we'd want to remove the Delete X(1,A) at t29 since this isn't correct in terms of timeline consistency, but we can't do that with HBase without support for deleting/undoing Delete markers.
> The above is not what is occurring. Instead, when T(A,2) comes in, the Put X(2,A) will occur at t20, but the Delete won't occur. This causes more index rows than data rows, essentially making it invalid.
> See PHOENIX-4051 as a workaround for fixing this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)