You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by mjwall <gi...@git.apache.org> on 2016/06/22 16:10:33 UTC
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
GitHub user mjwall opened a pull request:
https://github.com/apache/accumulo/pull/120
ACCUMULO-4324 update data version
This PR adds a new DATA_VERSION
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mjwall/accumulo ACCUMULO-4324
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo/pull/120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #120
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #120: ACCUMULO-4324 update data version
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/120
I am trying to upgrade from 1.7.1 to 1.8.0-SNAPSHOT (with this commit merged into 1.8.0-SNAP). While doing this I am seeing the tserver and master fail to start with the following error. I am going to work through this.
```
2016-07-12 11:55:35,628 [tserver.TabletServer] ERROR: Uncaught exception in TabletServer.main, exiting
java.lang.RuntimeException: This version of accumulo (1.8.0-SNAPSHOT) is not compatible with files stored using data version 7
at org.apache.accumulo.server.Accumulo.init(Accumulo.java:182)
at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:2908)
at org.apache.accumulo.tserver.TServerExecutable.execute(TServerExecutable.java:33)
at org.apache.accumulo.start.Main$1.run(Main.java:120)
at java.lang.Thread.run(Thread.java:745)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/120#discussion_r70669365
--- Diff: server/base/src/main/java/org/apache/accumulo/server/log/WalStateManager.java ---
@@ -98,10 +102,16 @@ private String root() {
// Tablet server exists
public void initWalMarker(TServerInstance tsi) throws WalMarkerException {
byte[] data = new byte[0];
- try {
- zoo.putPersistentData(root() + "/" + tsi.toString(), data, NodeExistsPolicy.FAIL);
- } catch (KeeperException | InterruptedException e) {
- throw new WalMarkerException(e);
+ while (true) {
+ try {
+ zoo.putPersistentData(root() + "/" + tsi.toString(), data, NodeExistsPolicy.FAIL);
+ break;
+ } catch (NoNodeException e) {
+ log.info("WAL parent node does not exist (upgrade may be in progress) : " + e.getMessage());
--- End diff --
@ShawnWalker made a very nice suggestion irc. Create the WALs node instead of waiting. This breaks this invisible dependency on code in the master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #120: ACCUMULO-4324 update data version
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on the issue:
https://github.com/apache/accumulo/pull/120
PR looks fine to me. But, for this ticket, we need to make sure upgrade from 1.6.x and upgrade from 1.7.x both work. I think they both should, since we didn't modify anything requiring additional upgrade code internally, but it'd be nice to verify. I don't think we need to support upgrades from anything older than 1.6.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/accumulo/pull/120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/120#discussion_r70644711
--- Diff: server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java ---
@@ -68,7 +72,7 @@
public static final int LOGGING_TO_HDFS = 4;
public static final BitSet CAN_UPGRADE = new BitSet();
static {
- for (int i : new int[] {DATA_VERSION, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
+ for (int i : new int[] {DATA_VERSION, MOVE_TO_REPLICATION_TABLE, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
--- End diff --
bq. My gut reaction is to not worry about 1.5, but is there any fundamental problem in supporting that (is it just a testing burden)?
Yeah its just a matter of testing it. I am not completely sure, but it seems code may be in place to support that upgrade. I don't know if it works. Personally I would not want to support it w/o testing it.
bq. If we don't support a direct upgrade from 1.5, what's the fail-safe?
Could add support for it in 1.8.1 or later if someone really wants that functionality.
> Could users just bulk-import the old files into a new instance?
That's tricky, if done incorrectly can resurrect old and/or deleted data.
Also before making any definitive decisions about 1.5 and earlier, we should see if the 1.6 to 1.8 upgrade uncovers any interesting issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/120#discussion_r70645915
--- Diff: server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java ---
@@ -68,7 +72,7 @@
public static final int LOGGING_TO_HDFS = 4;
public static final BitSet CAN_UPGRADE = new BitSet();
static {
- for (int i : new int[] {DATA_VERSION, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
+ for (int i : new int[] {DATA_VERSION, MOVE_TO_REPLICATION_TABLE, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
--- End diff --
Agreed on all of your points. Want to break out things to test in a JIRA or something? I can try to help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/120#discussion_r70623846
--- Diff: server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java ---
@@ -68,7 +72,7 @@
public static final int LOGGING_TO_HDFS = 4;
public static final BitSet CAN_UPGRADE = new BitSet();
static {
- for (int i : new int[] {DATA_VERSION, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
+ for (int i : new int[] {DATA_VERSION, MOVE_TO_REPLICATION_TABLE, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
--- End diff --
I added 1.7 to the set of release we can upgrade from. Probably need to remove some (like everyone that is not tested). I think we should test the upgrade from 1.6 to 1.8 and remove all other releases. Thoughts? I can take a wack at testing 1.6 to 1.8 upgrade testing, unless someone else wants to do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #120: ACCUMULO-4324 update data version
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/120
@mjwall I rebased this on the latest 1.8 your commit is now c8aa1fc110b4fb31f78ec5a59932046d79fbe875 my follow in commit to get upgrade working is a11b28b
I also squahed my commits as I made some more changes and implemented @ShawnWalker suggestion. I tested 1.6.5 to 1.8.0-SNAP and 1.7.1 to 1.8.0-SNAP upgrade .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #120: ACCUMULO-4324 update data version
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/120#discussion_r70641365
--- Diff: server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java ---
@@ -68,7 +72,7 @@
public static final int LOGGING_TO_HDFS = 4;
public static final BitSet CAN_UPGRADE = new BitSet();
static {
- for (int i : new int[] {DATA_VERSION, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
+ for (int i : new int[] {DATA_VERSION, MOVE_TO_REPLICATION_TABLE, MOVE_TO_ROOT_TABLE, MOVE_DELETE_MARKERS, LOGGING_TO_HDFS}) {
--- End diff --
1.6 and 1.7 to 1.8 sound like they should both work to me.
My gut reaction is to not worry about 1.5, but is there any fundamental problem in supporting that (is it just a testing burden)? If we don't support a direct upgrade from 1.5, what's the fail-safe? Could users just bulk-import the old files into a new instance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #120: ACCUMULO-4324 update data version
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/120
> I think they both should, since we didn't modify anything requiring additional upgrade code internally, but it'd be nice to verify. I don't think we need to support upgrades from anything older than 1.6.
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---