You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by JamesRTaylor <gi...@git.apache.org> on 2018/10/02 16:34:27 UTC
[GitHub] phoenix pull request #360: PHOENIX-4731 Make running transactional unit test...
GitHub user JamesRTaylor opened a pull request:
https://github.com/apache/phoenix/pull/360
PHOENIX-4731 Make running transactional unit tests for a given provider optional
Enables turning transaction tests off for Tephra and/or Omid. Please review, @ohadshacham and/or @twdsilva. This is a precursor to support of Omid - no new pom dependencies have been introduced. This also moves us to Tephra 0.15.0 which is required for the TAL changes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/phoenix parameterize-tx-provider
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/360.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #360
----
commit 1039f9d4ca0da5185d8babaa6681459e07da61ad
Author: Ohad Shacham <oh...@...>
Date: 2018-04-03T14:03:55Z
Add Omid support for Phoenix
commit a11bddcacb90d553f6c0fdab37e2c335f7f23ac4
Author: Ohad Shacham <oh...@...>
Date: 2018-05-02T11:55:47Z
Merge remote-tracking branch 'upstream/4.x-HBase-1.3' into 4.x-HBase-1.3-Omid-2
commit 45e6b0e309d037c03221de249501fa7c4bc651db
Author: Ohad Shacham <oh...@...>
Date: 2018-05-08T07:16:19Z
Remove hard coded Omid
commit 18bea3907ef73c6f842ac5410ea4623be5a36b18
Author: Ohad Shacham <oh...@...>
Date: 2018-05-08T09:28:05Z
Merge remote-tracking branch 'upstream/4.x-HBase-1.3' into 4.x-HBase-1.3-Omid-2
commit 38ab6f459e6174e150d88a146b4a35d9c1857fb6
Author: Ohad Shacham <oh...@...>
Date: 2018-05-15T08:19:05Z
some merge fixes
commit cbab9b72ee5c62d9512b9472b010f581e3866e21
Author: Ohad Shacham <oh...@...>
Date: 2018-05-22T13:04:56Z
Fix hbase config
commit 7c834994a862f5e2a0edfa560c0a1c4f047383ca
Author: Ohad Shacham <oh...@...>
Date: 2018-05-22T18:30:01Z
Partially revert the following commits.
https://github.com/ohadshacham/phoenix/commit/45e6b0e309d037c03221de249501fa7c4bc651db
https://github.com/ohadshacham/phoenix/commit/38ab6f459e6174e150d88a146b4a35d9c1857fb6
commit dc6de3ab12ad18e9dc5622521bfaf5a662e4d409
Author: Ohad Shacham <oh...@...>
Date: 2018-05-22T18:32:35Z
Merge commit '2015345a023f0adb59174443ec1328bb1399f11b' into 4.x-HBase-1.3-Omid-2
commit 9eae4abdbc770381620e23f4793acb279cec2544
Author: Ohad Shacham <oh...@...>
Date: 2018-05-22T18:42:26Z
Change back to TEPHRA what needed for testing TEPHRA
commit 0c5dfd642b3738d2d09aa19f2e6d4cbb97852c20
Author: Ohad Shacham <oh...@...>
Date: 2018-05-23T12:14:37Z
remove unnecessary changes.
commit 8d8d0e37210460aa00d387956eb013b96bd5de38
Author: Ohad Shacham <oh...@...>
Date: 2018-05-24T12:58:14Z
Remove dependency in testng
commit b1fcc46c277493b046f92358d0e9403b65815e23
Author: James Taylor <jt...@...>
Date: 2018-05-26T14:40:33Z
Include transaction provider code in transaction bytes send to server and fix misc tests
commit 3a9cd245349d05a0ab0a97dbfd26921a623567cb
Author: Ohad Shacham <oh...@...>
Date: 2018-05-29T12:54:00Z
Fix population of local index and move markPutAsCommitted to PhoenixTransactionContext.
commit 9a378cdc49b76d839ca1e1549a7a657dd7c19a45
Author: James Taylor <jt...@...>
Date: 2018-05-30T16:20:40Z
Perform local index maintainence on client side for Omid
commit 77586b857cec36c63a75bd87e9e83a375bc68bb7
Author: James Taylor <jt...@...>
Date: 2018-05-30T19:47:15Z
Fix Tephra regressions from previous commit
commit f563678cf174805bbb68b60bcf235a582cf2b2c3
Author: James Taylor <jt...@...>
Date: 2018-05-31T04:51:29Z
Fix for PartialCommitIT with Omid
commit a220bc5eaa173604c01373759938e8b2867082e8
Author: James Taylor <jt...@...>
Date: 2018-05-31T16:09:47Z
Remove unneeded PhoenixTransactionalTable
commit 57a1f11ae4aa02633e95436e7b32945d3803ac65
Author: James Taylor <jt...@...>
Date: 2018-05-31T17:13:24Z
Remove statics in OmidTransactionProcessor (but still could use more work)
commit 1e35aa2b0a3df3e5b3ebc9b6fac186da5ac0d1b8
Author: James Taylor <jt...@...>
Date: 2018-06-01T20:15:43Z
Fix asynchronous index building for Omid
commit 629e4873c212379dfab2dd12d80681ca9c2330ed
Author: James Taylor <ja...@...>
Date: 2018-06-03T06:00:05Z
Fix initial population of local indexes for Omid
commit 5a1d161566ed5a1532ed24978a48ee9c32c8ece4
Author: James Taylor <ja...@...>
Date: 2018-06-03T07:09:46Z
Fix client-side local index deletes
commit d1977fff3d78389f479963c71955b2e33e3c26c6
Author: James Taylor <ja...@...>
Date: 2018-06-03T16:33:43Z
Comment failing test with workaround
commit ec7a66072d616f740bfd669b8da56d0ac995f4a1
Author: James Taylor <ja...@...>
Date: 2018-06-12T06:08:31Z
Catch up omid2 branch with latest from 4.x-HBase-1.3
commit 64bd216c3534bc99f0803f3e442878e2bc650474
Author: James Taylor <ja...@...>
Date: 2018-06-14T17:01:26Z
Fixes and parameterization of unit tests
commit 119d5b48f5aef79e5edeade958481f8edbfdeec9
Author: James Taylor <ja...@...>
Date: 2018-06-14T17:47:55Z
Fix compilation errors
commit f925191568cfcae2373d1152aeaebfd61fa96c39
Author: James Taylor <ja...@...>
Date: 2018-06-14T18:04:21Z
Parameterize BaseViewIT and BaseIndexIT
commit a7cd47da6a8fb849358f0fd5b7b0a324d66584d2
Author: James Taylor <ja...@...>
Date: 2018-06-14T20:09:30Z
Parameterize transactional tests for both Tephra and Omid
commit aa85ed4605d242b9e3db895e2cb2cf92553ab480
Author: James Taylor <ja...@...>
Date: 2018-06-15T14:21:51Z
Parameterize transactional tests for both Tephra and Omid
commit fce08bdc61ca67b9bb3c85e9efcdb3114f39ebfd
Author: James Taylor <ja...@...>
Date: 2018-06-15T14:47:24Z
Lower parallelization to work around Tephra flakiness
commit 5af5980ab08049d74b4a8732f8b402cdddd4c741
Author: James Taylor <ja...@...>
Date: 2018-06-15T21:21:52Z
PHOENIX-4731 Make running transactional unit tests for a given provider optional
----
---
[GitHub] phoenix pull request #360: PHOENIX-4731 Make running transactional unit test...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/360#discussion_r222508350
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java ---
@@ -184,9 +184,24 @@ public boolean apply(PTable index) {
*/
public static void serialize(PTable dataTable, ImmutableBytesWritable ptr, PhoenixConnection connection) {
List<PTable> indexes = dataTable.getIndexes();
- serialize(dataTable, ptr, indexes, connection);
+ serializeServerMaintainedIndexes(dataTable, ptr, indexes, connection);
}
+ public static void serializeServerMaintainedIndexes(PTable dataTable, ImmutableBytesWritable ptr,
+ List<PTable> indexes, PhoenixConnection connection) {
+ Iterator<PTable> indexesItr = Collections.emptyListIterator();
+ boolean onlyLocalIndexes = dataTable.isImmutableRows() || dataTable.isTransactional();
+ if (onlyLocalIndexes) {
--- End diff --
Long story short, we've disallowed local indexes on Omid for now due to PHOENIX-4760. Once this is fixed, we can turn it back on. Given that index updates are transactional with data updates, there's not a great need IMHO for transactional local indexes, though. Having said that, there were a few things we needed to do for local indexes:
* Support client-side index maintenance for local indexes
* Handle the initial population (i.e. adding shadow columns) for a local index on the server side
---
[GitHub] phoenix pull request #360: PHOENIX-4731 Make running transactional unit test...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/360#discussion_r222480439
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxIndexMutationGenerator.java ---
@@ -313,7 +326,18 @@ public int compare(Cell o1, Cell o2) {
private void generateDeletes(PhoenixIndexMetaData indexMetaData,
Collection<Pair<Mutation, byte[]>> indexUpdates,
byte[] attribValue, TxTableState state) throws IOException {
- Iterable<IndexUpdate> deletes = codec.getIndexDeletes(state, indexMetaData);
+ byte[] regionStartKey = this.regionStartKey;
--- End diff --
Just for my understanding, why do we now need the start and stop key to generate the index deletes and upserts?
---
[GitHub] phoenix pull request #360: PHOENIX-4731 Make running transactional unit test...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/360#discussion_r222506752
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxIndexMutationGenerator.java ---
@@ -313,7 +326,18 @@ public int compare(Cell o1, Cell o2) {
private void generateDeletes(PhoenixIndexMetaData indexMetaData,
Collection<Pair<Mutation, byte[]>> indexUpdates,
byte[] attribValue, TxTableState state) throws IOException {
- Iterable<IndexUpdate> deletes = codec.getIndexDeletes(state, indexMetaData);
+ byte[] regionStartKey = this.regionStartKey;
--- End diff --
The region start and stop keys are used only when the index is a local index (in which case the row key is prefixed with the start key). The region end key is only used when the region start key length has a length of zero. Here's a small code snippet from IndexMaintainer.buildRowKey():
public byte[] buildRowKey(ValueGetter valueGetter, ImmutableBytesWritable rowKeyPtr, byte[] regionStartKey, byte[] regionEndKey, long ts) {
ImmutableBytesWritable ptr = new ImmutableBytesWritable();
boolean prependRegionStartKey = isLocalIndex && regionStartKey != null;
boolean isIndexSalted = !isLocalIndex && nIndexSaltBuckets > 0;
int prefixKeyLength =
prependRegionStartKey ? (regionStartKey.length != 0 ? regionStartKey.length
: regionEndKey.length) : 0;
TrustedByteArrayOutputStream stream = new TrustedByteArrayOutputStream(estimatedIndexRowKeyBytes + (prependRegionStartKey ? prefixKeyLength : 0));
DataOutput output = new DataOutputStream(stream);
try {
// For local indexes, we must prepend the row key with the start region key
if (prependRegionStartKey) {
if (regionStartKey.length == 0) {
output.write(new byte[prefixKeyLength]);
} else {
output.write(regionStartKey);
}
}
---
[GitHub] phoenix pull request #360: PHOENIX-4731 Make running transactional unit test...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/360#discussion_r222480675
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java ---
@@ -184,9 +184,24 @@ public boolean apply(PTable index) {
*/
public static void serialize(PTable dataTable, ImmutableBytesWritable ptr, PhoenixConnection connection) {
List<PTable> indexes = dataTable.getIndexes();
- serialize(dataTable, ptr, indexes, connection);
+ serializeServerMaintainedIndexes(dataTable, ptr, indexes, connection);
}
+ public static void serializeServerMaintainedIndexes(PTable dataTable, ImmutableBytesWritable ptr,
+ List<PTable> indexes, PhoenixConnection connection) {
+ Iterator<PTable> indexesItr = Collections.emptyListIterator();
+ boolean onlyLocalIndexes = dataTable.isImmutableRows() || dataTable.isTransactional();
+ if (onlyLocalIndexes) {
--- End diff --
Do we do anything special for transactional tables with local indexes?
---