You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2023/07/01 00:33:45 UTC
[GitHub] [accumulo] keith-turner opened a new pull request, #3563: Reliably refresh tablets after updating tablet files for a compaction.
keith-turner opened a new pull request, #3563:
URL: https://github.com/apache/accumulo/pull/3563
This commit modifies the compaction coordinator to reliably refresh hosted tablets in memory metadata after a compaction. The RPC request to a hosted tablet asking it to refresh happens after updating a tablets metadata for a compaction. The changes in the PR ensure that if the manager process dies after making the metadata update but before it makes the RPC that this will happen when the manager restarts.
The implementation for this relies on a new ~refresh sections in the metadata table. System compactions on hosted tablets will write a ~refresh entry before trying to update a tablets metadata. This entry allows the compaction commit process to be fault tolerant.
Javadoc was added to CompactionCoordinator.compactionCompleted() describing the overall process of committing a compaction.
This commit also refactored the refresh RPC code in bulk import so that it could also be used by compactions.
This commit also made the tablet metadata update for a compaction add scan entries for removed files when a tablet is hosted. This was done to prevent the Accumulo GC from removing compacted files that may be in use by an immediate scan.
fixes #3462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251215642
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
Review Comment:
For this the log message on the next line gives the purpose, do you think more than that is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner merged pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #3563:
URL: https://github.com/apache/accumulo/pull/3563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251215263
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
+ log.warn("Root refreshes stored in ZK at {} are getting large ({} bytes)."
+ + " Large nodes may cause problems for Zookeeper!", zpath, newJson.length());
+ }
+ return newJson.getBytes(UTF_8);
+ });
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ private Text createRow(RefreshEntry entry) {
+ return new Text(RefreshSection.getRowPrefix() + entry.getEcid().canonical());
+ }
+
+ private Mutation createAddMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+ dos.writeInt(1);
+ entry.getExtent().writeTo(dos);
+ dos.writeUTF(entry.getTserver().getHostPortSession());
+ dos.close();
+ m.at().family("").qualifier("").put(baos.toByteArray());
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ return m;
+ }
+
+ private Mutation createDeleteMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+ m.putDelete("", "");
+ return m;
+ }
+
+ private RefreshEntry decode(Map.Entry<Key,Value> entry) {
+ String row = entry.getKey().getRowData().toString();
+ Preconditions.checkArgument(row.startsWith(RefreshSection.getRowPrefix()));
+ Preconditions.checkArgument(entry.getKey().getColumnFamilyData().length() == 0);
+ Preconditions.checkArgument(entry.getKey().getColumnQualifierData().length() == 0);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(entry.getValue().get());
+ DataInputStream dis = new DataInputStream(bais)) {
+ Preconditions.checkArgument(dis.readInt() == 1);
+ var extent = KeyExtent.readFrom(dis);
+ var tserver = new TServerInstance(dis.readUTF());
+ return new RefreshEntry(
+ ExternalCompactionId.of(row.substring(RefreshSection.getRowPrefix().length())), extent,
+ tserver);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ @Override
+ public void add(Collection<RefreshEntry> entries) {
+ Objects.requireNonNull(entries);
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ // expect all of these to be the root tablet, verifying because its not stored
+ Preconditions
+ .checkArgument(entries.stream().allMatch(e -> e.getExtent().equals(RootTable.EXTENT)));
+ Consumer<Map<String,String>> mutator = map -> entries.forEach(refreshEntry -> map
+ .put(refreshEntry.getEcid().canonical(), refreshEntry.getTserver().getHostPortSession()));
+ mutateRootRefreshes(mutator);
+ } else {
+ try (BatchWriter writer = context.createBatchWriter(dataLevel.metaTable())) {
+ for (RefreshEntry entry : entries) {
+ writer.addMutation(createAddMutation(entry));
+ }
+ } catch (MutationsRejectedException | TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Collection<RefreshEntry> entries) {
+ Objects.requireNonNull(entries);
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ // expect all of these to be the root tablet, verifying because its not stored
+ Preconditions
+ .checkArgument(entries.stream().allMatch(e -> e.getExtent().equals(RootTable.EXTENT)));
+ Consumer<Map<String,String>> mutator =
+ map -> entries.forEach(refreshEntry -> map.remove(refreshEntry.getEcid().canonical()));
+ mutateRootRefreshes(mutator);
+ } else {
+ try (BatchWriter writer = context.createBatchWriter(dataLevel.metaTable())) {
+ for (RefreshEntry entry : entries) {
+ writer.addMutation(createDeleteMutation(entry));
+ }
+ } catch (MutationsRejectedException | TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ @Override
+ public Stream<RefreshEntry> list() {
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ var zooReader = context.getZooReader();
+ byte[] jsonBytes;
+ try {
+ jsonBytes =
+ zooReader.getData(context.getZooKeeperRoot() + RootTable.ZROOT_TABLET_REFRESHES);
+ return fromJson(new String(jsonBytes, UTF_8)).entrySet().stream()
+ .map(e -> new RefreshEntry(ExternalCompactionId.of(e.getKey()), RootTable.EXTENT,
+ new TServerInstance(e.getValue())));
+ } catch (KeeperException | InterruptedException e) {
+ throw new IllegalStateException(e);
+ }
+ } else {
+ Range range = RefreshSection.getRange();
+
+ Scanner scanner;
+ try {
+ scanner = context.createScanner(dataLevel.metaTable(), Authorizations.EMPTY);
+ } catch (TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ scanner.setRange(range);
+ return scanner.stream().map(this::decode);
+ }
Review Comment:
Fixed in dedaa88. Could not close the scanner in the method as a stream built from the scanner was returned. So add an onClose call to the stream to close the scanner and then closed the stream in the calling code.
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
Review Comment:
Fixed in dedaa88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1253010461
##########
core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java:
##########
@@ -43,6 +43,10 @@ public class RootTable {
* ZK path relative to the zookeeper node where the root tablet gc candidates are stored.
*/
public static final String ZROOT_TABLET_GC_CANDIDATES = ZROOT_TABLET + "/gc_candidates";
+ /*
+ * ZK path relative to the zookeeper node where the root tablet gc candidates are stored.
Review Comment:
I think this comment was a copy/paste and is not correct for this static variable
##########
test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java:
##########
@@ -730,6 +734,82 @@ public void testMetadataCompactions() throws Exception {
}
}
+ @Test
+ public void testSystemCompactionsRefresh() throws Exception {
+ // This test ensures that after a system compaction occurs that a tablet will refresh its files.
+
+ String tableName = getUniqueNames(1)[0];
+ try (final AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+
+ // configure tablet compaction iterator that filters out data not divisible by 7 and cause
+ // table to compact to one file
+
+ var ntc = new NewTableConfiguration();
+ IteratorSetting iterSetting = new IteratorSetting(100, TestFilter.class);
+ iterSetting.addOption("modulus", 7 + "");
+ ntc.attachIterator(iterSetting, EnumSet.of(IteratorScope.majc));
+ ntc.setProperties(Map.of(Property.TABLE_MAJC_RATIO.getKey(), "20"));
+
+ client.tableOperations().create(tableName, ntc);
+
+ Set<Integer> expectedData = new HashSet<>();
+
+ // Insert MAX_DATA rows
+ try (BatchWriter bw = client.createBatchWriter(tableName)) {
+ for (int i = 0; i < MAX_DATA; i++) {
+ Mutation m = new Mutation(String.format("r:%04d", i));
+ m.put("", "", "" + i);
+ bw.addMutation(m);
+
+ if (i % 75 == 0) {
+ // create many files as this will cause a system compaction
+ bw.flush();
+ client.tableOperations().flush(tableName, null, null, true);
+ }
+
+ if (i % 7 == 0) {
+ expectedData.add(i);
+ }
+ }
+ }
+
+ client.tableOperations().flush(tableName, null, null, true);
+
+ // there should be no system copmactions yet and no data should be filtered, so should see all
Review Comment:
```suggestion
// there should be no system compactions yet and no data should be filtered, so should see all
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#issuecomment-1618902100
> Curious if you have looked at https://github.com/apache/accumulo/pull/3200 and your thoughts on using ZK to signal Tablet metadata changes.
ZK is not very scalable in terms of writes (I think each write goes to the leader which then forwards it to all other ZK servers and waits for a majority to respond). This potential write bottleneck seem like an issue when considering it as an option for signaling tablet metadata refresh.
As for the tablet metadata update in the tablet server I have not had a chance to look into it in detail. Some of the issues that I know will arise for refresh off the top of my head are :
* Needs to properly synchronize with the cut over from in memory map to minor compacted file w.r.t. initiating new scan. A new scan should always use one or the other, never both or neither.
* Needs to properly synchronize the tracking of scan file entries in tablet metadata and active scans.
* Needs to properly synchronize with write ahead log recovery and tablets update around that.
* Needs to properly synchronize with a tablet that is loading or unloading.
The tablet server code has synchronization around bulk import, compactions, and splits that can be remove. I am thinking after we remove that code we can evaluate what needs are left and look into simplifying the tablets synchronization model that refresh will have to fit into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#issuecomment-1618795161
Curious if you have looked at #3200 and your thoughts on using ZK to signal Tablet metadata changes. #3200 contains a Cache, but that could be removed and it could notify listeners instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251142106
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java:
##########
@@ -593,4 +594,46 @@ default void addBulkLoadInProgressFlag(String path, long fateTxid) {
default void removeBulkLoadInProgressFlag(String path) {
throw new UnsupportedOperationException();
}
+
+ interface Refreshes {
+ static class RefreshEntry {
+ private final ExternalCompactionId ecid;
+
+ private final KeyExtent extent;
+ private final TServerInstance tserver;
+
+ public RefreshEntry(ExternalCompactionId ecid, KeyExtent extent, TServerInstance tserver) {
+ this.ecid = Objects.requireNonNull(ecid);
+ this.extent = Objects.requireNonNull(extent);
+ this.tserver = Objects.requireNonNull(tserver);
+ }
+
+ public ExternalCompactionId getEcid() {
+ return ecid;
+ }
+
+ public KeyExtent getExtent() {
+ return extent;
+ }
+
+ public TServerInstance getTserver() {
+ return tserver;
+ }
+ }
+
+ void add(Collection<RefreshEntry> entries);
+
+ void delete(Collection<RefreshEntry> entries);
+
+ Stream<RefreshEntry> list();
Review Comment:
Why not `stream()` here instead of `list()`?
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
Review Comment:
Could `262_144` be replaced with a constant with a descriptive name to make it easier to tell what it is?
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
+ log.warn("Root refreshes stored in ZK at {} are getting large ({} bytes)."
+ + " Large nodes may cause problems for Zookeeper!", zpath, newJson.length());
+ }
+ return newJson.getBytes(UTF_8);
+ });
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ private Text createRow(RefreshEntry entry) {
+ return new Text(RefreshSection.getRowPrefix() + entry.getEcid().canonical());
+ }
+
+ private Mutation createAddMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+ dos.writeInt(1);
+ entry.getExtent().writeTo(dos);
+ dos.writeUTF(entry.getTserver().getHostPortSession());
+ dos.close();
+ m.at().family("").qualifier("").put(baos.toByteArray());
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ return m;
+ }
+
+ private Mutation createDeleteMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+ m.putDelete("", "");
+ return m;
+ }
+
+ private RefreshEntry decode(Map.Entry<Key,Value> entry) {
+ String row = entry.getKey().getRowData().toString();
+ Preconditions.checkArgument(row.startsWith(RefreshSection.getRowPrefix()));
+ Preconditions.checkArgument(entry.getKey().getColumnFamilyData().length() == 0);
+ Preconditions.checkArgument(entry.getKey().getColumnQualifierData().length() == 0);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(entry.getValue().get());
+ DataInputStream dis = new DataInputStream(bais)) {
+ Preconditions.checkArgument(dis.readInt() == 1);
+ var extent = KeyExtent.readFrom(dis);
+ var tserver = new TServerInstance(dis.readUTF());
+ return new RefreshEntry(
+ ExternalCompactionId.of(row.substring(RefreshSection.getRowPrefix().length())), extent,
+ tserver);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ @Override
+ public void add(Collection<RefreshEntry> entries) {
+ Objects.requireNonNull(entries);
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ // expect all of these to be the root tablet, verifying because its not stored
+ Preconditions
+ .checkArgument(entries.stream().allMatch(e -> e.getExtent().equals(RootTable.EXTENT)));
+ Consumer<Map<String,String>> mutator = map -> entries.forEach(refreshEntry -> map
+ .put(refreshEntry.getEcid().canonical(), refreshEntry.getTserver().getHostPortSession()));
+ mutateRootRefreshes(mutator);
+ } else {
+ try (BatchWriter writer = context.createBatchWriter(dataLevel.metaTable())) {
+ for (RefreshEntry entry : entries) {
+ writer.addMutation(createAddMutation(entry));
+ }
+ } catch (MutationsRejectedException | TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Collection<RefreshEntry> entries) {
+ Objects.requireNonNull(entries);
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ // expect all of these to be the root tablet, verifying because its not stored
+ Preconditions
+ .checkArgument(entries.stream().allMatch(e -> e.getExtent().equals(RootTable.EXTENT)));
+ Consumer<Map<String,String>> mutator =
+ map -> entries.forEach(refreshEntry -> map.remove(refreshEntry.getEcid().canonical()));
+ mutateRootRefreshes(mutator);
+ } else {
+ try (BatchWriter writer = context.createBatchWriter(dataLevel.metaTable())) {
+ for (RefreshEntry entry : entries) {
+ writer.addMutation(createDeleteMutation(entry));
+ }
+ } catch (MutationsRejectedException | TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ @Override
+ public Stream<RefreshEntry> list() {
+ if (dataLevel == Ample.DataLevel.ROOT) {
+ var zooReader = context.getZooReader();
+ byte[] jsonBytes;
+ try {
+ jsonBytes =
+ zooReader.getData(context.getZooKeeperRoot() + RootTable.ZROOT_TABLET_REFRESHES);
+ return fromJson(new String(jsonBytes, UTF_8)).entrySet().stream()
+ .map(e -> new RefreshEntry(ExternalCompactionId.of(e.getKey()), RootTable.EXTENT,
+ new TServerInstance(e.getValue())));
+ } catch (KeeperException | InterruptedException e) {
+ throw new IllegalStateException(e);
+ }
+ } else {
+ Range range = RefreshSection.getRange();
+
+ Scanner scanner;
+ try {
+ scanner = context.createScanner(dataLevel.metaTable(), Authorizations.EMPTY);
+ } catch (TableNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ scanner.setRange(range);
+ return scanner.stream().map(this::decode);
+ }
Review Comment:
```suggestion
try (Scanner scanner = context.createScanner(dataLevel.metaTable(), Authorizations.EMPTY)) {
scanner.setRange(range);
return scanner.stream().map(this::decode);
} catch (TableNotFoundException e) {
throw new IllegalStateException(e);
}
}
```
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
Review Comment:
```suggestion
private static class Data {
static final int CURRENT_VERSION = 1;
```
Could add a current version variable to use throughout the code instead of hardcoding the version, in this case `1`.
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
+ log.warn("Root refreshes stored in ZK at {} are getting large ({} bytes)."
+ + " Large nodes may cause problems for Zookeeper!", zpath, newJson.length());
+ }
+ return newJson.getBytes(UTF_8);
+ });
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ private Text createRow(RefreshEntry entry) {
+ return new Text(RefreshSection.getRowPrefix() + entry.getEcid().canonical());
+ }
+
+ private Mutation createAddMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+ dos.writeInt(1);
+ entry.getExtent().writeTo(dos);
+ dos.writeUTF(entry.getTserver().getHostPortSession());
+ dos.close();
Review Comment:
```suggestion
```
Is there a reason this is closed early or can it be cleaned up automatically by the try-with-resources?
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
Review Comment:
There are a lot of places where `Preconditions.checkArgument()` is used in this class. It might be helpful to add error messages where it might be unclear what or why something is being checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251218679
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
Review Comment:
No, I think thats fine. It just seems like a weird value I guess (2^18) but I think it makes sense :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251216073
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java:
##########
@@ -593,4 +594,46 @@ default void addBulkLoadInProgressFlag(String path, long fateTxid) {
default void removeBulkLoadInProgressFlag(String path) {
throw new UnsupportedOperationException();
}
+
+ interface Refreshes {
+ static class RefreshEntry {
+ private final ExternalCompactionId ecid;
+
+ private final KeyExtent extent;
+ private final TServerInstance tserver;
+
+ public RefreshEntry(ExternalCompactionId ecid, KeyExtent extent, TServerInstance tserver) {
+ this.ecid = Objects.requireNonNull(ecid);
+ this.extent = Objects.requireNonNull(extent);
+ this.tserver = Objects.requireNonNull(tserver);
+ }
+
+ public ExternalCompactionId getEcid() {
+ return ecid;
+ }
+
+ public KeyExtent getExtent() {
+ return extent;
+ }
+
+ public TServerInstance getTserver() {
+ return tserver;
+ }
+ }
+
+ void add(Collection<RefreshEntry> entries);
+
+ void delete(Collection<RefreshEntry> entries);
+
+ Stream<RefreshEntry> list();
Review Comment:
Fixed in dedaa88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251214798
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -172,9 +198,62 @@ protected void startRunningCleaner(ScheduledThreadPoolExecutor schedExecutor) {
ThreadPools.watchNonCriticalScheduledTask(future);
}
+ private void processRefreshes(Ample.DataLevel dataLevel) {
+
+ // process batches of refresh entries to avoid reading all into memory at once
+ Iterators.partition(ctx.getAmple().refreshes(dataLevel).list().iterator(), 10000)
+ .forEachRemaining(refreshEntries -> {
+ LOG.info("Processing {} tablet refreshes for {}", refreshEntries.size(), dataLevel);
+
+ var extents =
+ refreshEntries.stream().map(RefreshEntry::getExtent).collect(Collectors.toList());
+ var tabletsMeta = new HashMap<KeyExtent,TabletMetadata>();
+ ctx.getAmple().readTablets().forTablets(extents, Optional.empty())
+ .fetch(PREV_ROW, LOCATION, SCANS).build().stream()
Review Comment:
Fixed in dedaa88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251181623
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -172,9 +198,62 @@ protected void startRunningCleaner(ScheduledThreadPoolExecutor schedExecutor) {
ThreadPools.watchNonCriticalScheduledTask(future);
}
+ private void processRefreshes(Ample.DataLevel dataLevel) {
+
+ // process batches of refresh entries to avoid reading all into memory at once
+ Iterators.partition(ctx.getAmple().refreshes(dataLevel).list().iterator(), 10000)
+ .forEachRemaining(refreshEntries -> {
+ LOG.info("Processing {} tablet refreshes for {}", refreshEntries.size(), dataLevel);
+
+ var extents =
+ refreshEntries.stream().map(RefreshEntry::getExtent).collect(Collectors.toList());
+ var tabletsMeta = new HashMap<KeyExtent,TabletMetadata>();
+ ctx.getAmple().readTablets().forTablets(extents, Optional.empty())
+ .fetch(PREV_ROW, LOCATION, SCANS).build().stream()
Review Comment:
I got a warning from my IDE that the `TabletsMetadata` object created by `.build()` is not closed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1251215955
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
+ final int version;
+
+ final Map<String,String> entries;
+
+ public Data(int version, Map<String,String> entries) {
+ this.version = version;
+ this.entries = entries;
+ }
+ }
+
+ private static String toJson(Map<String,String> entries) {
+ var data = new Data(1, Objects.requireNonNull(entries));
+ return GSON.get().toJson(data, Data.class);
+ }
+
+ private Map<String,String> fromJson(String json) {
+ Data data = GSON.get().fromJson(json, Data.class);
+ Preconditions.checkArgument(data.version == 1);
+ Objects.requireNonNull(data.entries);
+ return data.entries;
+ }
+
+ public RefreshesImpl(ServerContext context, Ample.DataLevel dataLevel) {
+ this.context = context;
+ this.dataLevel = dataLevel;
+ }
+
+ private void mutateRootRefreshes(Consumer<Map<String,String>> mutator) {
+ String zpath = context.getZooKeeperRoot() + ZROOT_TABLET_REFRESHES;
+ try {
+ context.getZooReaderWriter().mutateExisting(zpath, currVal -> {
+ String currJson = new String(currVal, UTF_8);
+ log.debug("Root refreshes before change : {}", currJson);
+ Map<String,String> entries = fromJson(currJson);
+ mutator.accept(entries);
+ String newJson = toJson(entries);
+ log.debug("Root refreshes after change : {}", newJson);
+ if (newJson.length() > 262_144) {
+ log.warn("Root refreshes stored in ZK at {} are getting large ({} bytes)."
+ + " Large nodes may cause problems for Zookeeper!", zpath, newJson.length());
+ }
+ return newJson.getBytes(UTF_8);
+ });
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ private Text createRow(RefreshEntry entry) {
+ return new Text(RefreshSection.getRowPrefix() + entry.getEcid().canonical());
+ }
+
+ private Mutation createAddMutation(RefreshEntry entry) {
+ Mutation m = new Mutation(createRow(entry));
+
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+ dos.writeInt(1);
+ entry.getExtent().writeTo(dos);
+ dos.writeUTF(entry.getTserver().getHostPortSession());
+ dos.close();
Review Comment:
I was closing it before getting the array from the byte array output stream.
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/RefreshesImpl.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.metadata;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.metadata.RootTable.ZROOT_TABLET_REFRESHES;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.RootTable;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema.RefreshSection;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.server.ServerContext;
+import org.apache.hadoop.io.Text;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+public class RefreshesImpl implements Ample.Refreshes {
+
+ private static final Logger log = LoggerFactory.getLogger(RefreshesImpl.class);
+
+ private final Ample.DataLevel dataLevel;
+ private final ServerContext context;
+
+ public static String getInitialJson() {
+ return toJson(Map.of());
+ }
+
+ // This class is used to serialize and deserialize root tablet metadata using GSon. Any changes to
+ // this class must consider persisted data.
+ private static class Data {
Review Comment:
Fixed in dedaa88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3563: Reliably refresh tablets after updating tablet files for a compaction.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3563:
URL: https://github.com/apache/accumulo/pull/3563#discussion_r1253519187
##########
core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java:
##########
@@ -43,6 +43,10 @@ public class RootTable {
* ZK path relative to the zookeeper node where the root tablet gc candidates are stored.
*/
public static final String ZROOT_TABLET_GC_CANDIDATES = ZROOT_TABLET + "/gc_candidates";
+ /*
+ * ZK path relative to the zookeeper node where the root tablet gc candidates are stored.
Review Comment:
fixed in 0c844da
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org