You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/08/04 13:04:59 UTC
[GitHub] [accumulo] EdColeman opened a new pull request #2224: Versioned Properties - refactored to address PR comments
EdColeman opened a new pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224
This replaces PR #2194 Refactored to address PR comments
This the first step in moving towards a refactored ZooKeeper property storage. Intended to replace using individual nodes in ZooKeeper to save properties to a versioned group that is stored on a ZooKeeper single node. The PR contains the changes that provide serialization / deserialization of the properties along with maintaining versioning information.
Designed to allow evolution of the storage scheme.
Provides a header that maintains schema version, data version, and timestamp.
Optional compression of the byte storage array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman edited a comment on pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman edited a comment on pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#issuecomment-929680660
The partially addresses https://github.com/apache/accumulo/issues/1454 and https://github.com/apache/accumulo/issues/1225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701932782
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
Would use of the `@Experimental` annotation alleviate your concerns about putting it in src?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703419153
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
Just curious - what's the advantage of ImmutableMap over Map.of() or Map.ofEntries()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701383398
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
Could make the `@Experimental` annotation public and use it here. It is currently only used by `Property` keys but I don't see why we couldn't use it elsewhere in places like here.
Also, I noticed this class is in the Test package. I am not sure that was intentional or not. If you put it in Test because its experimental, maybe using the annotation would make you feel better about putting it with the other codecs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683804929
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/PropSerdes.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+/**
+ * Serialize / Deserialize versioned properties into a byte array.
+ */
+public interface PropSerdes {
Review comment:
I first had encoder. I first remember serdes used in kafka for serializer and deserializer, which is what this is doing. I'm not sure I like either of the name choices. Open to suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717724234
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
I'm going to leave this for now
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: "
+ + encodingOpts.getEncodingVersion());
+ }
+
+ DataVersionInfo vMetadata = new DataVersionInfo(dis);
+
+ Map<String,String> props = decodePayload(bis, encodingOpts);
+
+ return new VersionedProperties(vMetadata.getDataVersion(), vMetadata.getTimestamp(), props);
+ }
+ }
+
+ abstract boolean checkCanDecodeVersion(final EncodingOptions encodingOpts);
+
+ /**
+ * Extracts the encoding version from the encoded byte array without fully decoding the payload.
+ * This is a convenience method if multiple encodings are present, and should only be required if
+ * upgrading / changing encodings, otherwise a single encoding should be in operation for an
+ * instance at any given time.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getEncodingVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ return new EncodingOptions(dis).getEncodingVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException("Failed to read encoding version from byte array provided",
+ ex);
+ }
+ }
+
+ /**
+ * Extracts the data version from the encoded byte array without fully decoding the payload.
+ * Normally the data version should be obtained from a fully decoded instance of the versioned
+ * properties.
+ * <p>
+ * The cost of reading the byte array from the backing store should be considered verses the
+ * additional cost of decoding - with a goal of reducing data reads from the store preferred.
+ * Generally reading from the store will be followed by some sort of usage which would require the
+ * full decode operation anyway.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getDataVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ // skip encoding metadata
+ new EncodingOptions(dis);
+ return new DataVersionInfo(dis).getDataVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException(
+ "Failed to read data version version from byte array provided", ex);
+ }
+ }
+
+ /**
+ * Decode the payload and any optional encoding specific metadata and return a map of the property
+ * name, value pairs.
+ *
+ * @param inStream
+ * an input stream
+ * @param encodingOpts
+ * the general encoding options.
+ * @return a map of properties name, value pairs.
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ abstract Map<String,String> decodePayload(final InputStream inStream,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ /**
+ * Read the property map from a data input stream as UTF strings. The input stream should be
+ * created configured by sub-classes for the output of the sub-class. If the sub-class uses an
+ * encoding other that UTF strings, they should override this method. An example would be an
+ * encoding that uses JSON to encode the map.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ Map<String,String> readMapAsUTF(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ String k = dis.readUTF();
+ String v = dis.readUTF();
+ aMap.put(k, v);
+ }
+ return aMap;
+ }
+
+ /**
+ * Write the property map to the data output stream. The underlying stream is not closed by this
+ * method.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dos
+ * a data output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ void writeMapAsUTF(final DataOutputStream dos, final Map<String,String> aMap) throws IOException {
+
+ dos.writeInt(aMap.size());
+
+ for (Map.Entry<String,String> e : aMap.entrySet()) {
+ dos.writeUTF(e.getKey());
+ dos.writeUTF(e.getValue());
+ }
+ dos.flush();
+ }
+
+ /**
+ * Helper class for reading / writing versioned properties metadata.
+ */
+ static class DataVersionInfo {
+ private final int dataVersion;
+ private final Instant timestamp;
+
Review comment:
It is used - its is a holdover an maybe overkill for just the two values - but I like the delegation so that it does not need to be handled in multiple places. Will look at using a static method like previous suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717538865
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` in this case is a little less readable in my opinion. For me, it's a bit quicker to read with each pair on its own line. Not a huge difference either way though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman edited a comment on pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman edited a comment on pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#issuecomment-929680660
The partially addresses https://github.com/apache/accumulo/issues/1454 and https://github.com/apache/accumulo/issues/1225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701918682
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: {}");
Review comment:
Looks to be missing an argument. Might need to do something like the following instead however.
```suggestion
throw new IllegalArgumentException(
"Invalid data version - cannot process the version read: " + encodingOpts);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: {}");
Review comment:
Looks to be missing an argument. Might need to do something like the following instead
```suggestion
throw new IllegalArgumentException(
"Invalid data version - cannot process the version read: " + encodingOpts);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703460030
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
Review comment:
This avoids creating an additional hashmap - it would only be one entry and not a big deal - it may turn out that this method could be refactored out as upstream changes could do the same thing and just call addOrUpdate(map) - it will depend on how many places set just one property and if this is really a convenience or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701998032
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
The `AESCryptoService` uses GCM similar to how you are using it but there are some details that differ. I don't know the imp super well but some differences that I noticed between yours and `AESCryptoService`:
- It uses `SecureRandom` to generate the key
- IV gets stored in memory so it can be incremented
- There is some Key wrapping/unwrapping
- Key gets loaded from a URI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683798345
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ public static final EncodingOptions COMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, true);
+
+ public static final EncodingOptions UNCOMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, false);
+
+ private final EncodingVersion encodingVersion;
+ private final boolean compress;
+
+ public EncodingOptions(EncodingVersion encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ public EncodingOptions(final DataInputStream dis) throws IOException {
+ encodingVersion = EncodingVersion.byId(dis.readInt());
+ compress = dis.readBoolean();
+ }
+
+ public void encode(final DataOutputStream dos) throws IOException {
+ dos.writeInt(encodingVersion.id);
+ dos.writeBoolean(compress);
+ }
+
+ public EncodingVersion getEncodingVersion() {
+ return encodingVersion;
+ }
+
+ public boolean isCompressed() {
+ return compress;
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", EncodingOptions.class.getSimpleName() + "[", "]")
+ .add("encodingVersion=" + encodingVersion).toString();
+ }
+
+ /**
+ * Provides a strong typing of the known encoding versions and allows the version id to be encoded
+ * as an integer. Adding an encoding type must be done as an addition and not change or delete
+ * previous versions or numbering to preserve compatibility.
+ */
+ public enum EncodingVersion {
Review comment:
I wanted it to be strongly typed so that it could be referenced by name rather than a "magic number" - but also wanted the serialized field to be compact and with a fast comparison, so wanted to avoid a string / name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717724234
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
I'm going to leave this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701983649
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
Review comment:
> An alternative might be addOrUpdate, or upsert?
`addOrUpdate` seems pretty intuitive to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683947077
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
+ dos.writeUTF(vProps.getTimestampISO());
Review comment:
I don't see clock time as useful for versioning or for resolving conflicts between concurrent updates, since clock times aren't that precise, aren't unique at any level of precision, and don't necessarily have meaning across different servers with different clocks. The only value I see is the human being able to read it and see what time it was on the server that stored it... but that's of limited value. I don't think that's needed. We don't do that any other place (we don't store clock time in RFile metadata, or in other ZK nodes, for example). System logs and/or ZK metadata should suffice for that kind of thing anyway.
I guess if you're finding that it is still handy, and avoids getting extra info from ZK, I'd have to try to understand how you're utilizing it at the next layer up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701978398
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
Review comment:
Its an add or an update. New key, values are added, but if the key exists the value is replaced. An alternative might be addOrUpdate, or upsert?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683781381
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ public static final EncodingOptions COMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, true);
+
+ public static final EncodingOptions UNCOMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, false);
+
+ private final EncodingVersion encodingVersion;
+ private final boolean compress;
+
+ public EncodingOptions(EncodingVersion encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ public EncodingOptions(final DataInputStream dis) throws IOException {
+ encodingVersion = EncodingVersion.byId(dis.readInt());
+ compress = dis.readBoolean();
+ }
+
+ public void encode(final DataOutputStream dos) throws IOException {
+ dos.writeInt(encodingVersion.id);
+ dos.writeBoolean(compress);
+ }
+
+ public EncodingVersion getEncodingVersion() {
+ return encodingVersion;
+ }
+
+ public boolean isCompressed() {
+ return compress;
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", EncodingOptions.class.getSimpleName() + "[", "]")
+ .add("encodingVersion=" + encodingVersion).toString();
+ }
+
+ /**
+ * Provides a strong typing of the known encoding versions and allows the version id to be encoded
+ * as an integer. Adding an encoding type must be done as an addition and not change or delete
+ * previous versions or numbering to preserve compatibility.
+ */
+ public enum EncodingVersion {
Review comment:
Not sure you need a whole class to represent a 1-up number. Can just use `int` instead of `EncodingVersion`.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
+ dos.writeUTF(vProps.getTimestampISO());
Review comment:
I'm unclear on the utility of storing a string timestamp.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/PropSerdesEncoderFactory.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+/**
+ * Create an serialization / deserialization encoder based on serialization version and additional
+ * information used in the serialization process.
+ */
+public class PropSerdesEncoderFactory {
+
+ private final PropSerdes serdes;
+
+ public PropSerdesEncoderFactory(final EncodingOptions encodingOptions) {
+
+ // currently only one version supported - this would need to be extended to support others.
Review comment:
I'm not sure we need this factory at all. It's just extra layers of code to trace through and limits comprehension. We can just instantiate the GZip implementation directly. I don't anticipate making this pluggable or anything.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
Review comment:
Is this version used to compare with the 1-up in ZK to ensure they are the same? If so, I'm not sure we need to store it inside the serialized blob, since it's only metadata about the blob. The VersionedProperty object should have it, certainly. I'm just not sure if it has any value stored inside the blob.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ public static final EncodingOptions COMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, true);
+
+ public static final EncodingOptions UNCOMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, false);
+
+ private final EncodingVersion encodingVersion;
+ private final boolean compress;
+
+ public EncodingOptions(EncodingVersion encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ public EncodingOptions(final DataInputStream dis) throws IOException {
+ encodingVersion = EncodingVersion.byId(dis.readInt());
+ compress = dis.readBoolean();
+ }
+
+ public void encode(final DataOutputStream dos) throws IOException {
+ dos.writeInt(encodingVersion.id);
+ dos.writeBoolean(compress);
+ }
+
+ public EncodingVersion getEncodingVersion() {
+ return encodingVersion;
+ }
+
+ public boolean isCompressed() {
+ return compress;
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", EncodingOptions.class.getSimpleName() + "[", "]")
+ .add("encodingVersion=" + encodingVersion).toString();
+ }
Review comment:
Not sure what this toString method is intended to be used for, but for completeness, could add compress field.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
+ dos.writeUTF(vProps.getTimestampISO());
+
+ // write the property map keys, values.
+ if (encodingOpts.isCompressed()) {
+ writeMapCompressed(bos, vProps.getAllProperties());
+ } else {
+ writeMap(dos, vProps.getAllProperties());
+ }
+
+ dos.flush();
+
+ return bos.toByteArray();
+
+ } catch (IOException ex) {
+ throw new IllegalStateException("Encountered error serializing properties", ex);
+ }
+ }
+
+ @Override
+ public VersionedProperties fromBytes(final byte[] bytes) {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ // read header - version and isCompressed.
+ EncodingOptions opts = new EncodingOptions(dis);
+
+ // read versioning information
+ int dataVersion = dis.readInt();
+ Instant timestamp = tsFormatter.parse(dis.readUTF(), Instant::from);
+
+ // read the property map keys, values
+ Map<String,String> aMap;
+ if (opts.isCompressed()) {
+ aMap = readCompressedMap(bis);
+ } else {
+ aMap = readMap(dis);
+ }
+
+ return new VersionedPropertiesImpl(dataVersion, timestamp, aMap);
+
+ } catch (IOException ex) {
+ throw new UncheckedIOException("Encountered error deserializing properties", ex);
+ }
+ }
+
+ /**
+ * Read and uncompress an input stream compressed with GZip. The input stream is not closed by
+ * this method.
+ *
+ * @param is
+ * an input stream
+ * @return a map with the property k.v pairs
+ * @throws IOException
+ * if an exception occurs reading from the stream
+ */
+ private Map<String,String> readCompressedMap(final InputStream is) throws IOException {
+
+ try (GZIPInputStream gzipIn = new GZIPInputStream(is);
+ DataInputStream dis = new DataInputStream(gzipIn)) {
+ return readMap(dis);
+ }
+ }
+
+ /**
+ * Read the property map from a data input stream
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ private Map<String,String> readMap(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ Map.Entry<String,String> e = readKV(dis);
+ aMap.put(e.getKey(), e.getValue());
+ }
+ return aMap;
+ }
+
+ /**
+ * Write the property map to the output stream, compressing the output using GZip compression. The
+ * underlying stream will be closed when this method completes.
+ *
+ * @param os
+ * an output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ private void writeMapCompressed(final OutputStream os, final Map<String,String> aMap)
+ throws IOException {
+
+ try (GZIPOutputStream gzipOut = new GZIPOutputStream(os);
+ DataOutputStream dos = new DataOutputStream(gzipOut)) {
+
+ writeMap(dos, aMap);
+
+ gzipOut.flush();
+ gzipOut.finish();
+
+ } catch (IOException ex) {
+ throw new IOException("Encountered error compressing properties", ex);
+ }
+ }
+
+ /**
+ * Write the property map to the data output stream. The underlying stream is not closed by this
+ * method.
+ *
+ * @param dos
+ * a data output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ private void writeMap(final DataOutputStream dos, final Map<String,String> aMap)
+ throws IOException {
+
+ dos.writeInt(aMap.size());
+
+ aMap.forEach((k, v) -> writeKV(k, v, dos));
+
+ dos.flush();
Review comment:
small nit. I'd clean up some of these blank lines. The more the code is spread out like that, the harder it is to fit on one screen, especially when using large fonts, making it harder to read (for me, at least)
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/PropSerdes.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+/**
+ * Serialize / Deserialize versioned properties into a byte array.
+ */
+public interface PropSerdes {
Review comment:
Not sure I like the name of this interface, but the name isn't important. I think this is a good interface, and I like how the GZip implementation extends and implements it.
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Map;
+
+public interface VersionedProperties {
Review comment:
I don't think this needs to be an interface with a separate implementation. These two can be combined and simplified to just a simple POJO container of `{ version, propMap }`.
I'm thinking something like:
```java
public class VersionedProperties {
private final long version;
private final Map<String,String> properties;
public VersionedProperties(long version, Map<String,String> properties) {
this.version = version;
this.properties = Map.copyOf(requireNonNull(properties));
}
public long getVersion() {
return version;
}
public Map<String,String> getProperties() {
return properties;
}
}
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/PropSerdesEncoderFactory.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+/**
+ * Create an serialization / deserialization encoder based on serialization version and additional
+ * information used in the serialization process.
+ */
+public class PropSerdesEncoderFactory {
+
+ private final PropSerdes serdes;
+
+ public PropSerdesEncoderFactory(final EncodingOptions encodingOptions) {
+
+ // currently only one version supported - this would need to be extended to support others.
+ if (encodingOptions.getEncodingVersion().equals(EncodingOptions.EncodingVersion.V1_0)) {
Review comment:
Should always compare enums with `==` and never `.equals()`.
```suggestion
if (encodingOptions.getEncodingVersion() == EncodingOptions.EncodingVersion.V1_0) {
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ public static final EncodingOptions COMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, true);
+
+ public static final EncodingOptions UNCOMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, false);
+
+ private final EncodingVersion encodingVersion;
+ private final boolean compress;
+
+ public EncodingOptions(EncodingVersion encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ public EncodingOptions(final DataInputStream dis) throws IOException {
+ encodingVersion = EncodingVersion.byId(dis.readInt());
+ compress = dis.readBoolean();
+ }
+
+ public void encode(final DataOutputStream dos) throws IOException {
+ dos.writeInt(encodingVersion.id);
+ dos.writeBoolean(compress);
+ }
+
+ public EncodingVersion getEncodingVersion() {
+ return encodingVersion;
+ }
+
+ public boolean isCompressed() {
+ return compress;
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", EncodingOptions.class.getSimpleName() + "[", "]")
+ .add("encodingVersion=" + encodingVersion).toString();
+ }
+
+ /**
+ * Provides a strong typing of the known encoding versions and allows the version id to be encoded
+ * as an integer. Adding an encoding type must be done as an addition and not change or delete
+ * previous versions or numbering to preserve compatibility.
+ */
+ public enum EncodingVersion {
+
+ INVALID(-1), V1_0(1);
+
+ // a unique id to identify the version
+ public final Integer id;
Review comment:
Just use primitive `int` here to avoid the unnecessary autoboxing/unboxing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701922925
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: {}");
+ }
+
+ DataVersionInfo vMetadata = new DataVersionInfo(dis);
+
+ Map<String,String> props = decodePayload(bis, encodingOpts);
+
+ return new VersionedProperties(vMetadata.getDataVersion(), vMetadata.getTimestamp(), props);
+ }
+ }
+
+ abstract boolean checkCanDecodeVersion(final EncodingOptions encodingOpts);
+
+ /**
+ * Extracts the encoding version from the encoded byte array without fully decoding the payload.
+ * This is a convenience method if multiple encodings are present, and should only be required if
+ * upgrading / changing encodings, otherwise a single encoding should be in operation for an
+ * instance at any given time.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getEncodingVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ return new EncodingOptions(dis).getEncodingVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException("Failed to read encoding version from byte array provided",
+ ex);
+ }
+ }
+
+ /**
+ * Extracts the data version from the encoded byte array without fully decoding the payload.
+ * Normally the data version should be obtained from a fully decoded instance of the versioned
+ * properties.
+ * <p>
+ * The cost of reading the byte array from the backing store should be considered verses the
+ * additional cost of decoding - with a goal of reducing data reads from the store preferred.
+ * Generally reading from the store will be followed by some sort of usage which would require the
+ * full decode operation anyway.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getDataVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ // skip encoding metadata
+ new EncodingOptions(dis);
+ return new DataVersionInfo(dis).getDataVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException(
+ "Failed to read data version version from byte array provided", ex);
+ }
+ }
+
+ /**
+ * Decode the payload and any optional encoding specific metadata and return a map of the property
+ * name, value pairs. 1
Review comment:
```suggestion
* name, value pairs.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717183367
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
Review comment:
```suggestion
this(Map.of());
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
Review comment:
An import for suggestions farther down in the file:
```suggestion
import static java.util.Objects.requireNonNull;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
Review comment:
```suggestion
try (var bis = new ByteArrayInputStream(payload)) {
// write the property map keys, values.
try (var cis = new CipherInputStream(bis, cipher);
var cdis = new DataInputStream(cis)) {
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder1 = VersionedPropEncryptCodec.codec(true,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder1.toBytes(vProps);
+
+ log.debug("len: {}, bytes: {}", encodedBytes.length, encodedBytes);
+
+ VersionedProperties decodedProps = encoder1.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ @Test
+ public void validateEncryptedValuesChange() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly and without additional processing.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ // Adding an encoding version must be done as an addition. Do not change or delete previous
+ // version numbers
+ public static final int EncodingVersion_1_0 = 1;
+ public static final int EXPERIMENTAL_CIPHER_ENCODING_1_0 = 999;
+
+ private final int encodingVersion;
+ private final boolean compress;
+
+ EncodingOptions(final int encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ /**
+ * Instantiate encoding options to use version 1.0 encoding settings.
+ *
+ * @param compress
+ * when true compress the property map.
+ * @return the encoding options.
+ */
+ public static EncodingOptions V1_0(final boolean compress) {
+ return new EncodingOptions(EncodingVersion_1_0, compress);
+ }
+
+ /**
+ * Instantiate an instance of EncodingOptions reading the values from an input stream. Typically,
+ * the stream will be obtained from reading a byte array from a data store and then creating a
+ * stream that reads from that array,
+ *
+ * @param dis
+ * a data input stream
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ public EncodingOptions(final DataInputStream dis) throws IOException {
Review comment:
Could do a named static method, rather than a constructor here. It'd be an opportunity to have a more explicit name (similar to your `V1_0` name above). Maybe `fromStream` or just `decode` to match the member method named `encode`. `write()` and `read()` also work (and would conflict less with the name of the class, which refers to encoding options for properties; a method named `encode` to encode the encoder options is a bit confusing).
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
Can avoid the Guava type by using Java's own built-in immutable types:
```suggestion
private final Map<String,String> props;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
Review comment:
Removing imports that aren't needed after unused method is removed:
```suggestion
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.CipherOutputStream;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder1 = VersionedPropEncryptCodec.codec(true,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder1.toBytes(vProps);
+
+ log.debug("len: {}, bytes: {}", encodedBytes.length, encodedBytes);
+
+ VersionedProperties decodedProps = encoder1.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ @Test
+ public void validateEncryptedValuesChange() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec codec1 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes1 = codec1.toBytes(vProps);
+
+ VersionedPropCodec codec2 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes2 = codec2.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes1);
+ log.debug("Encoded: {}", encodedBytes2);
+
+ VersionedProperties from2 = codec1.fromBytes(encodedBytes2);
+ VersionedProperties from1 = codec2.fromBytes(encodedBytes1);
+
+ assertEquals(from1.getProperties(), from2.getProperties());
+
+ VersionedPropCodec codec3 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ VersionedProperties from3 = codec3.fromBytes(encodedBytes1);
+ assertEquals(from1.getDataVersion(), from3.getDataVersion());
+ assertEquals(from1.getProperties(), from3.getProperties());
+
+ assertNotEquals(encodedBytes1, encodedBytes2);
+
+ }
+
+ private String keyGen() throws NoSuchAlgorithmException {
+ SecretKey secretKey = KeyGenerator.getInstance("AES").generateKey();
+ return Base64.getEncoder().encodeToString(secretKey.getEncoded());
+ }
Review comment:
This method isn't used and can be deleted.
```suggestion
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
Review comment:
More places var could be useful:
```suggestion
try (var bos = new ByteArrayOutputStream()) {
var cos = new CipherOutputStream(bos, cipher);
var dos = new DataOutputStream(cos);
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
Review comment:
Using var to clean up redundant type specifications in local variables:
```suggestion
var cipherProps = new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
var cipher = Cipher.getInstance("AES/GCM/NoPadding");
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
Review comment:
Removing redundant indirect reference, since Instant is immutable, and the implementation just does this anyway:
```suggestion
return timestamp;
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
Review comment:
Updating imports for suggestions further down:
```suggestion
import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
// compression friendly
var p = Map.of("accumulo.prop.key_name.1", "value1", "accumulo.prop.key_name.2", "value2",
"accumulo.prop.key_name.3", "value3", "accumulo.prop.key_name.4", "value4",
"accumulo.prop.key_name.5", "value5", "accumulo.prop.key_name.6", "value9");
VersionedProperties vProps = new VersionedProperties(aVersion, Instant.now(), p);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
Review comment:
Make immutable copy using built-ins, avoiding Guava:
```suggestion
this.timestamp = requireNonNull(timestamp);
this.props = props == null ? Map.of() : Map.copyOf(props);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param updates
+ * A map of key, values pairs.
+ * @return A new instance of this class with the properties added or updated.
+ */
+ public VersionedProperties addOrUpdate(final Map<String,String> updates) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ putAll(updates);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Delete multiple properties provided as a collection of keys.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param keys
+ * a collection of the keys that if they exist, will be removed.
+ * @return A new instance of this class.
+ */
+ public VersionedProperties remove(Collection<String> keys) {
+
+ HashMap<String,String> orig = new HashMap<>(props);
+ keys.forEach(orig::remove);
+
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(orig).build();
+
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.keySet().removeAll(keys); // removing from keySet view removes from underlying mapping
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param updates
+ * A map of key, values pairs.
+ * @return A new instance of this class with the properties added or updated.
+ */
+ public VersionedProperties addOrUpdate(final Map<String,String> updates) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ putAll(updates);
+ }
+ }).build();
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.putAll(updates);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.put(key, value);
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropCodecTest.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+
+/**
+ * Exercise the base class specific methods - most testing will occur in subclasses
+ */
+public class VersionedPropCodecTest {
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidEncodingNullArray() {
+ VersionedPropCodec.getEncodingVersion(null);
+ }
+
+ @Test
+ public void validEncoding() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ int encodingVersion = VersionedPropCodec.getEncodingVersion(bytes);
+ assertEquals(0, encodingVersion);
+ }
+
+ /**
+ * The timestamp will be invalid - this should cause a timestamp parse error that will be remapped
+ * to an IllegalArgumentException.
+ */
+ @Test(expected = IllegalArgumentException.class)
+ public void getDataVersionBadTimestamp() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ assertEquals(0, VersionedPropCodec.getDataVersion(bytes));
+ }
+
+ @Test
+ public void goPath() throws IOException {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: "
+ + encodingOpts.getEncodingVersion());
+ }
+
+ DataVersionInfo vMetadata = new DataVersionInfo(dis);
+
+ Map<String,String> props = decodePayload(bis, encodingOpts);
+
+ return new VersionedProperties(vMetadata.getDataVersion(), vMetadata.getTimestamp(), props);
+ }
+ }
+
+ abstract boolean checkCanDecodeVersion(final EncodingOptions encodingOpts);
+
+ /**
+ * Extracts the encoding version from the encoded byte array without fully decoding the payload.
+ * This is a convenience method if multiple encodings are present, and should only be required if
+ * upgrading / changing encodings, otherwise a single encoding should be in operation for an
+ * instance at any given time.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getEncodingVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ return new EncodingOptions(dis).getEncodingVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException("Failed to read encoding version from byte array provided",
+ ex);
+ }
+ }
+
+ /**
+ * Extracts the data version from the encoded byte array without fully decoding the payload.
+ * Normally the data version should be obtained from a fully decoded instance of the versioned
+ * properties.
+ * <p>
+ * The cost of reading the byte array from the backing store should be considered verses the
+ * additional cost of decoding - with a goal of reducing data reads from the store preferred.
+ * Generally reading from the store will be followed by some sort of usage which would require the
+ * full decode operation anyway.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getDataVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ // skip encoding metadata
+ new EncodingOptions(dis);
+ return new DataVersionInfo(dis).getDataVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException(
+ "Failed to read data version version from byte array provided", ex);
+ }
+ }
+
+ /**
+ * Decode the payload and any optional encoding specific metadata and return a map of the property
+ * name, value pairs.
+ *
+ * @param inStream
+ * an input stream
+ * @param encodingOpts
+ * the general encoding options.
+ * @return a map of properties name, value pairs.
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ abstract Map<String,String> decodePayload(final InputStream inStream,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ /**
+ * Read the property map from a data input stream as UTF strings. The input stream should be
+ * created configured by sub-classes for the output of the sub-class. If the sub-class uses an
+ * encoding other that UTF strings, they should override this method. An example would be an
+ * encoding that uses JSON to encode the map.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ Map<String,String> readMapAsUTF(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ String k = dis.readUTF();
+ String v = dis.readUTF();
+ aMap.put(k, v);
+ }
+ return aMap;
+ }
+
+ /**
+ * Write the property map to the data output stream. The underlying stream is not closed by this
+ * method.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dos
+ * a data output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ void writeMapAsUTF(final DataOutputStream dos, final Map<String,String> aMap) throws IOException {
+
+ dos.writeInt(aMap.size());
+
+ for (Map.Entry<String,String> e : aMap.entrySet()) {
+ dos.writeUTF(e.getKey());
+ dos.writeUTF(e.getValue());
+ }
+ dos.flush();
+ }
+
+ /**
+ * Helper class for reading / writing versioned properties metadata.
+ */
+ static class DataVersionInfo {
+ private final int dataVersion;
+ private final Instant timestamp;
+
Review comment:
This helper class doesn't appear to be used. Is this for later or leftover from a previous iteration?
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly and without additional processing.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ // Adding an encoding version must be done as an addition. Do not change or delete previous
+ // version numbers
+ public static final int EncodingVersion_1_0 = 1;
+ public static final int EXPERIMENTAL_CIPHER_ENCODING_1_0 = 999;
Review comment:
The experimental cipher encoding is only used in test. I think it could be moved there.
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropGzipCodecTest.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Exercise the {@link VersionedPropGzipCodec} class.
+ */
+public class VersionedPropGzipCodecTest {
+
+ private static final Logger log = LoggerFactory.getLogger(VersionedPropGzipCodecTest.class);
+
+ @Test
+ public void roundTripUncompressed() throws IOException {
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
Review comment:
Can inline `Instant.now()` and use `Map.of()` to define these maps and inline them for easier readability (as I did in previous comments, but am too lazy to do for this class! :smiley_cat: )
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropCodecTest.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+
+/**
+ * Exercise the base class specific methods - most testing will occur in subclasses
+ */
+public class VersionedPropCodecTest {
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidEncodingNullArray() {
+ VersionedPropCodec.getEncodingVersion(null);
+ }
+
+ @Test
+ public void validEncoding() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ int encodingVersion = VersionedPropCodec.getEncodingVersion(bytes);
+ assertEquals(0, encodingVersion);
+ }
+
+ /**
+ * The timestamp will be invalid - this should cause a timestamp parse error that will be remapped
+ * to an IllegalArgumentException.
+ */
+ @Test(expected = IllegalArgumentException.class)
Review comment:
Could use `assertThrows()` to ensure the specific line throws the desired exception (can also optionally do checks on the resulting exception).
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
I'd keep this in test. The experimental annotation is only useful on public API, and really then, only to generate docs. There's no value in making it more visible to expand its scope. Encrypting the properties in ZK is a neat idea, but quite a bit out of scope of the main work to improve property storage in ZK, and I think it's better if we keep this stuff in test for now.
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
Review comment:
Redundant flush (close on next line implies flush):
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717183367
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
Review comment:
```suggestion
this(Map.of());
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
Review comment:
An import for suggestions farther down in the file:
```suggestion
import static java.util.Objects.requireNonNull;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
Review comment:
```suggestion
try (var bis = new ByteArrayInputStream(payload)) {
// write the property map keys, values.
try (var cis = new CipherInputStream(bis, cipher);
var cdis = new DataInputStream(cis)) {
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder1 = VersionedPropEncryptCodec.codec(true,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder1.toBytes(vProps);
+
+ log.debug("len: {}, bytes: {}", encodedBytes.length, encodedBytes);
+
+ VersionedProperties decodedProps = encoder1.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ @Test
+ public void validateEncryptedValuesChange() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly and without additional processing.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ // Adding an encoding version must be done as an addition. Do not change or delete previous
+ // version numbers
+ public static final int EncodingVersion_1_0 = 1;
+ public static final int EXPERIMENTAL_CIPHER_ENCODING_1_0 = 999;
+
+ private final int encodingVersion;
+ private final boolean compress;
+
+ EncodingOptions(final int encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ /**
+ * Instantiate encoding options to use version 1.0 encoding settings.
+ *
+ * @param compress
+ * when true compress the property map.
+ * @return the encoding options.
+ */
+ public static EncodingOptions V1_0(final boolean compress) {
+ return new EncodingOptions(EncodingVersion_1_0, compress);
+ }
+
+ /**
+ * Instantiate an instance of EncodingOptions reading the values from an input stream. Typically,
+ * the stream will be obtained from reading a byte array from a data store and then creating a
+ * stream that reads from that array,
+ *
+ * @param dis
+ * a data input stream
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ public EncodingOptions(final DataInputStream dis) throws IOException {
Review comment:
Could do a named static method, rather than a constructor here. It'd be an opportunity to have a more explicit name (similar to your `V1_0` name above). Maybe `fromStream` or just `decode` to match the member method named `encode`. `write()` and `read()` also work (and would conflict less with the name of the class, which refers to encoding options for properties; a method named `encode` to encode the encoder options is a bit confusing).
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
Can avoid the Guava type by using Java's own built-in immutable types:
```suggestion
private final Map<String,String> props;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
Review comment:
Removing imports that aren't needed after unused method is removed:
```suggestion
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.CipherOutputStream;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder1 = VersionedPropEncryptCodec.codec(true,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder1.toBytes(vProps);
+
+ log.debug("len: {}, bytes: {}", encodedBytes.length, encodedBytes);
+
+ VersionedProperties decodedProps = encoder1.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ @Test
+ public void validateEncryptedValuesChange() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec codec1 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes1 = codec1.toBytes(vProps);
+
+ VersionedPropCodec codec2 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes2 = codec2.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes1);
+ log.debug("Encoded: {}", encodedBytes2);
+
+ VersionedProperties from2 = codec1.fromBytes(encodedBytes2);
+ VersionedProperties from1 = codec2.fromBytes(encodedBytes1);
+
+ assertEquals(from1.getProperties(), from2.getProperties());
+
+ VersionedPropCodec codec3 = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ VersionedProperties from3 = codec3.fromBytes(encodedBytes1);
+ assertEquals(from1.getDataVersion(), from3.getDataVersion());
+ assertEquals(from1.getProperties(), from3.getProperties());
+
+ assertNotEquals(encodedBytes1, encodedBytes2);
+
+ }
+
+ private String keyGen() throws NoSuchAlgorithmException {
+ SecretKey secretKey = KeyGenerator.getInstance("AES").generateKey();
+ return Base64.getEncoder().encodeToString(secretKey.getEncoded());
+ }
Review comment:
This method isn't used and can be deleted.
```suggestion
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
Review comment:
More places var could be useful:
```suggestion
try (var bos = new ByteArrayOutputStream()) {
var cos = new CipherOutputStream(bos, cipher);
var dos = new DataOutputStream(cos);
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
Review comment:
Using var to clean up redundant type specifications in local variables:
```suggestion
var cipherProps = new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
var cipher = Cipher.getInstance("AES/GCM/NoPadding");
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
Review comment:
Removing redundant indirect reference, since Instant is immutable, and the implementation just does this anyway:
```suggestion
return timestamp;
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
Review comment:
Updating imports for suggestions further down:
```suggestion
import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
// compression friendly
var p = Map.of("accumulo.prop.key_name.1", "value1", "accumulo.prop.key_name.2", "value2",
"accumulo.prop.key_name.3", "value3", "accumulo.prop.key_name.4", "value4",
"accumulo.prop.key_name.5", "value5", "accumulo.prop.key_name.6", "value9");
VersionedProperties vProps = new VersionedProperties(aVersion, Instant.now(), p);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
Review comment:
Make immutable copy using built-ins, avoiding Guava:
```suggestion
this.timestamp = requireNonNull(timestamp);
this.props = props == null ? Map.of() : Map.copyOf(props);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param updates
+ * A map of key, values pairs.
+ * @return A new instance of this class with the properties added or updated.
+ */
+ public VersionedProperties addOrUpdate(final Map<String,String> updates) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ putAll(updates);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Delete multiple properties provided as a collection of keys.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param keys
+ * a collection of the keys that if they exist, will be removed.
+ * @return A new instance of this class.
+ */
+ public VersionedProperties remove(Collection<String> keys) {
+
+ HashMap<String,String> orig = new HashMap<>(props);
+ keys.forEach(orig::remove);
+
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(orig).build();
+
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.keySet().removeAll(keys); // removing from keySet view removes from underlying mapping
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param updates
+ * A map of key, values pairs.
+ * @return A new instance of this class with the properties added or updated.
+ */
+ public VersionedProperties addOrUpdate(final Map<String,String> updates) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ putAll(updates);
+ }
+ }).build();
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.putAll(updates);
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
Review comment:
Avoid unnecessary complexity, since the constructor will make an immutable copy anyway and this is easier to understand and maintain:
```suggestion
var updated = new HashMap<>(props);
updated.put(key, value);
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropCodecTest.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+
+/**
+ * Exercise the base class specific methods - most testing will occur in subclasses
+ */
+public class VersionedPropCodecTest {
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidEncodingNullArray() {
+ VersionedPropCodec.getEncodingVersion(null);
+ }
+
+ @Test
+ public void validEncoding() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ int encodingVersion = VersionedPropCodec.getEncodingVersion(bytes);
+ assertEquals(0, encodingVersion);
+ }
+
+ /**
+ * The timestamp will be invalid - this should cause a timestamp parse error that will be remapped
+ * to an IllegalArgumentException.
+ */
+ @Test(expected = IllegalArgumentException.class)
+ public void getDataVersionBadTimestamp() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ assertEquals(0, VersionedPropCodec.getDataVersion(bytes));
+ }
+
+ @Test
+ public void goPath() throws IOException {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
```suggestion
var vProps = new VersionedProperties(aVersion, Instant.now(), Map.of("k1", "v1"));
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: "
+ + encodingOpts.getEncodingVersion());
+ }
+
+ DataVersionInfo vMetadata = new DataVersionInfo(dis);
+
+ Map<String,String> props = decodePayload(bis, encodingOpts);
+
+ return new VersionedProperties(vMetadata.getDataVersion(), vMetadata.getTimestamp(), props);
+ }
+ }
+
+ abstract boolean checkCanDecodeVersion(final EncodingOptions encodingOpts);
+
+ /**
+ * Extracts the encoding version from the encoded byte array without fully decoding the payload.
+ * This is a convenience method if multiple encodings are present, and should only be required if
+ * upgrading / changing encodings, otherwise a single encoding should be in operation for an
+ * instance at any given time.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getEncodingVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ return new EncodingOptions(dis).getEncodingVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException("Failed to read encoding version from byte array provided",
+ ex);
+ }
+ }
+
+ /**
+ * Extracts the data version from the encoded byte array without fully decoding the payload.
+ * Normally the data version should be obtained from a fully decoded instance of the versioned
+ * properties.
+ * <p>
+ * The cost of reading the byte array from the backing store should be considered verses the
+ * additional cost of decoding - with a goal of reducing data reads from the store preferred.
+ * Generally reading from the store will be followed by some sort of usage which would require the
+ * full decode operation anyway.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getDataVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ // skip encoding metadata
+ new EncodingOptions(dis);
+ return new DataVersionInfo(dis).getDataVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException(
+ "Failed to read data version version from byte array provided", ex);
+ }
+ }
+
+ /**
+ * Decode the payload and any optional encoding specific metadata and return a map of the property
+ * name, value pairs.
+ *
+ * @param inStream
+ * an input stream
+ * @param encodingOpts
+ * the general encoding options.
+ * @return a map of properties name, value pairs.
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ abstract Map<String,String> decodePayload(final InputStream inStream,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ /**
+ * Read the property map from a data input stream as UTF strings. The input stream should be
+ * created configured by sub-classes for the output of the sub-class. If the sub-class uses an
+ * encoding other that UTF strings, they should override this method. An example would be an
+ * encoding that uses JSON to encode the map.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ Map<String,String> readMapAsUTF(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ String k = dis.readUTF();
+ String v = dis.readUTF();
+ aMap.put(k, v);
+ }
+ return aMap;
+ }
+
+ /**
+ * Write the property map to the data output stream. The underlying stream is not closed by this
+ * method.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dos
+ * a data output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ void writeMapAsUTF(final DataOutputStream dos, final Map<String,String> aMap) throws IOException {
+
+ dos.writeInt(aMap.size());
+
+ for (Map.Entry<String,String> e : aMap.entrySet()) {
+ dos.writeUTF(e.getKey());
+ dos.writeUTF(e.getValue());
+ }
+ dos.flush();
+ }
+
+ /**
+ * Helper class for reading / writing versioned properties metadata.
+ */
+ static class DataVersionInfo {
+ private final int dataVersion;
+ private final Instant timestamp;
+
Review comment:
This helper class doesn't appear to be used. Is this for later or leftover from a previous iteration?
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly and without additional processing.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ // Adding an encoding version must be done as an addition. Do not change or delete previous
+ // version numbers
+ public static final int EncodingVersion_1_0 = 1;
+ public static final int EXPERIMENTAL_CIPHER_ENCODING_1_0 = 999;
Review comment:
The experimental cipher encoding is only used in test. I think it could be moved there.
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropGzipCodecTest.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Exercise the {@link VersionedPropGzipCodec} class.
+ */
+public class VersionedPropGzipCodecTest {
+
+ private static final Logger log = LoggerFactory.getLogger(VersionedPropGzipCodecTest.class);
+
+ @Test
+ public void roundTripUncompressed() throws IOException {
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
Review comment:
Can inline `Instant.now()` and use `Map.of()` to define these maps and inline them for easier readability (as I did in previous comments, but am too lazy to do for this class! :smiley_cat: )
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropCodecTest.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Test;
+
+/**
+ * Exercise the base class specific methods - most testing will occur in subclasses
+ */
+public class VersionedPropCodecTest {
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidEncodingNullArray() {
+ VersionedPropCodec.getEncodingVersion(null);
+ }
+
+ @Test
+ public void validEncoding() {
+ // length so that array reads do not error
+ byte[] bytes = new byte[100];
+ int encodingVersion = VersionedPropCodec.getEncodingVersion(bytes);
+ assertEquals(0, encodingVersion);
+ }
+
+ /**
+ * The timestamp will be invalid - this should cause a timestamp parse error that will be remapped
+ * to an IllegalArgumentException.
+ */
+ @Test(expected = IllegalArgumentException.class)
Review comment:
Could use `assertThrows()` to ensure the specific line throws the desired exception (can also optionally do checks on the resulting exception).
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
I'd keep this in test. The experimental annotation is only useful on public API, and really then, only to generate docs. There's no value in making it more visible to expand its scope. Encrypting the properties in ZK is a neat idea, but quite a bit out of scope of the main work to improve property storage in ZK, and I think it's better if we keep this stuff in test for now.
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
Review comment:
Redundant flush (close on next line implies flush):
```suggestion
```
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` has the nice property of being immutable, so you don't have to worry that it's modified anywhere in the test. You can format it to get one pair per line and suppress formatting by wrapping it with `// @formatter:off` and `// @formatter:on`, but it's probably not worth it. I actually prefer two columns per line, as long as both key and value fit (and they do here) because I don't actually care about the data and want to see the subsequent lines of code sooner. But, either way is fine with me. It's definitely not important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703451421
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
None? But it strongly enforces the instance to be immutable. The goal was to prevent making an internal error that could change the state. Instances are very likely to be shared by multiple threads, it was important to make sure that intermediate updates before being commuted to ZooKeeper were not visible until committed, retrieved so updates are consistent across threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701957771
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
I can go either way. The one sticking point for me is how to treat the secret - maybe you have some guidance from your other work with encryption? It may be external to these changes, but in the scope of making it an option (even if experimental) I think we would need to provide something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703413185
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one.
+ * <p>
+ * Because instances of this class are immutable, this method creates a new copy of the
+ * properties. Other processes will continue to see original values retrieved from the data store.
+ * Other processes will receive an update when the instance is encoded and stored in the data
+ * store and then retrieved with the normal store update mechanisms.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties addOrUpdate(final String key, final String value) {
+ ImmutableMap<String,String> updated =
Review comment:
You could just call the other addOrUpdate method here, making a map with a single k,v pair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683949071
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/PropSerdes.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+/**
+ * Serialize / Deserialize versioned properties into a byte array.
+ */
+public interface PropSerdes {
Review comment:
In general, I think a "serialization" interface can be presumed to have a corresponding deserialization action coupled to it (a one-way serializer isn't super useful). So, I don't find it necessary to force both words into the name. I'd probably just call it a PropSerializer (or perhaps MapSerializer). Similarly, other possible names could be based on "converter" or "encoder". Or, since you named the package "codec", you could just call this "PropertyCodec", with "encode" and "decode" method names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701974478
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
Review comment:
Actually not so sure. "update" does make sense when a property is being replaced/updated but not so sure when a new prop is being added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701941244
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
Review comment:
```suggestion
* properties allows for comparison of properties and can be used to ensure that values being
* written to the backend store have not changed. This metadata should be written / appear early in
* the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
```
The missing 'u' in values pushed these to new lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman merged pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman merged pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#issuecomment-929680660
The partially addresses https://github.com/apache/accumulo/issues/1454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#issuecomment-929680660
The partially addresses https://github.com/apache/accumulo/issues/1454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683384659
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
+ dos.writeUTF(vProps.getTimestampISO());
+
+ // write the property map keys, values.
+ if (encodingOpts.isCompressed()) {
+ writeMapCompressed(bos, vProps.getAllProperties());
+ } else {
+ writeMap(dos, vProps.getAllProperties());
+ }
+
+ dos.flush();
+
+ return bos.toByteArray();
+
+ } catch (IOException ex) {
+ throw new IllegalStateException("Encountered error serializing properties", ex);
+ }
+ }
+
+ @Override
+ public VersionedProperties fromBytes(final byte[] bytes) {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ // read header - version and isCompressed.
+ EncodingOptions opts = new EncodingOptions(dis);
+
+ // read versioning information
+ int dataVersion = dis.readInt();
+ Instant timestamp = tsFormatter.parse(dis.readUTF(), Instant::from);
+
+ // read the property map keys, values
+ Map<String,String> aMap;
+ if (opts.isCompressed()) {
+ aMap = readCompressedMap(bis);
+ } else {
+ aMap = readMap(dis);
+ }
+
+ return new VersionedPropertiesImpl(dataVersion, timestamp, aMap);
+
+ } catch (IOException ex) {
+ throw new UncheckedIOException("Encountered error deserializing properties", ex);
+ }
+ }
+
+ /**
+ * Read and uncompress an input stream compressed with GZip. The input stream is not closed by
+ * this method.
+ *
+ * @param is
+ * an input stream
+ * @return a map with the property k.v pairs
+ * @throws IOException
+ * if an exception occurs reading from the stream
+ */
+ private Map<String,String> readCompressedMap(final InputStream is) throws IOException {
+
+ try (GZIPInputStream gzipIn = new GZIPInputStream(is);
+ DataInputStream dis = new DataInputStream(gzipIn)) {
+ return readMap(dis);
+ }
+ }
+
+ /**
+ * Read the property map from a data input stream
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ private Map<String,String> readMap(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ Map.Entry<String,String> e = readKV(dis);
Review comment:
`readKV` is only called from this location. You could just put the method body here instead of making a new method. Same comment applies to `writeKV`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701442578
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodec.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.EncodingOptions.EXPERIMENTAL_CIPHER_ENCODING_1_0;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.KeySpec;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.SecretKeySpec;
+
+/**
+ * EXPERIMENTAL - demonstrates using an alternate encoding scheme. The sample is completely
+ * functional, however, certain elements such as password / key handling may not be suitable for
+ * production. The encoding version is EXPERIMENTAL_CIPHER_ENCODING_1_0.
+ * <p>
+ * This codec uses AES algorithm in GCM mode for encryption to encode the property map that is
+ * stored in the external store.
+ */
+public class VersionedPropEncryptCodec extends VersionedPropCodec {
Review comment:
I put in test until we discuss if it is even a desirable to provide as an option. It was really useful to drive the design because it required additional metadata - if it is acceptable and moved into the code base, there would be small changes required (like a property for codec selection) - those would be worthwhile to do, but I deferred until they would actually be used.
The code should also receive additional scrutiny to check that I'm not just providing an illusion of increased confidentially because of a coding or misapplication of the encryption API.
In test, it seemed clear enough to show intent and how additional codecs could be developed is desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman merged pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman merged pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683802264
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/GzipPropEncoding.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.time.Instant;
+import java.util.AbstractMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+/**
+ * Initial property encoding that (optionally) uses gzip to compress the property map. The encoding
+ * version supported is EncodingVersion.V1_0.
+ */
+public class GzipPropEncoding implements PropSerdes {
+
+ private final EncodingOptions encodingOpts;
+
+ public GzipPropEncoding(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ @Override
+ public byte[] toBytes(final VersionedProperties vProps) {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write header - version id, isCompressed
+ encodingOpts.encode(dos);
+
+ // write updated property versioning info (data version, time stamp)
+ dos.writeInt(vProps.getNextVersion());
+ dos.writeUTF(vProps.getTimestampISO());
Review comment:
The timestamp seemed attractive when looking at the data - it provides a hint when the data was initially serialized. I also had thought about ways that having a timestamp could be used for deconfliction of concurrent updates - you would then know the order that the operations were serialized - but never took it further thinking something might be possible.
The versioned properties should have a timestamp - but that could be time derived from the zNode modification time if we don't want to serialize it.
I can drop the serialization, but it did seem like it was handy having that info available without needing extra info from zookeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703552250
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
If multiple threads reference an instance of VersionedProperties, and one thread calls `addOrUpdate` or `remove`, how does the other thread see the change when these two methods return a new object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701947609
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
Review comment:
```suggestion
* instances of this class are immutable, this creates a new copy of the properties. Other
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, the creates a new copy of the properties. Other
Review comment:
```suggestion
* Because instances of this class are immutable, this creates a new copy of the properties. Other
```
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ put(key, value);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Add or update multiple properties. If a property already exists it is overwritten.
+ * <p>
+ * Because instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param updates
+ * A map of key, values pairs.
+ * @return A new instance of this class with the properties added or updated.
+ */
+ public VersionedProperties update(final Map<String,String> updates) {
+ ImmutableMap<String,String> updated =
+ ImmutableMap.<String,String>builder().putAll(new HashMap<>() {
+ {
+ putAll(props);
+ putAll(updates);
+ }
+ }).build();
+ return new VersionedProperties(dataVersion, Instant.now(), updated);
+ }
+
+ /**
+ * Delete multiple properties provided as a collection of keys.
+ * <p>
+ * Because instances of this class are immutable, the creates a new copy of the properties. Other
Review comment:
```suggestion
* Because instances of this class are immutable, this creates a new copy of the properties. Other
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717903959
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedPropCodec.java
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.apache.accumulo.server.conf.codec.VersionedProperties.tsFormatter;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.time.format.DateTimeParseException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Abstract class to provide encoding / decoding of versioned properties. This class handles the
+ * serialization of the metadata and subclasses are required to implement
+ * {@link #encodePayload(OutputStream, VersionedProperties, EncodingOptions)} and
+ * {@link #decodePayload(InputStream, EncodingOptions)} to handle any specific implementation
+ * metadata (optional) and the property map according to the encoding scheme of the subclass.
+ * <p>
+ * The basic encoding format:
+ * <ul>
+ * <li>encoding metadata - specifies codec to be used</li>
+ * <li>version metadata - specifies property versioning information</li>
+ * <li>codec specific metadata (optional)</li>
+ * <li>the property map</li>
+ * </ul>
+ *
+ */
+public abstract class VersionedPropCodec {
+
+ private final EncodingOptions encodingOpts;
+
+ public VersionedPropCodec(final EncodingOptions encodingOpts) {
+ this.encodingOpts = encodingOpts;
+ }
+
+ /**
+ * The general encoding options that apply to all encodings.
+ *
+ * @return the general options.
+ */
+ public EncodingOptions getEncodingOpts() {
+ return encodingOpts;
+ }
+
+ /**
+ * Serialize the versioned properties. The version information on the properties is updated if the
+ * data is successfully serialized.
+ *
+ * @param vProps
+ * the versioned properties.
+ * @return a byte array with the serialized properties.
+ */
+ public byte[] toBytes(final VersionedProperties vProps) throws IOException {
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(bos)) {
+
+ // write encoding metadata
+ encodingOpts.encode(dos);
+
+ // write version metadata
+ DataVersionInfo vMetadata =
+ new DataVersionInfo(vProps.getNextVersion(), vProps.getTimestamp());
+ vMetadata.write(dos);
+
+ // delegate property encoding to sub-class
+ encodePayload(bos, vProps, encodingOpts);
+
+ return bos.toByteArray();
+ }
+ }
+
+ /**
+ * Encode the properties and optionally any specific encoding metadata that is necessary to decode
+ * the payload with the scheme chosen.
+ *
+ * @param out
+ * an output stream
+ * @param vProps
+ * the versioned properties
+ * @param encodingOpts
+ * the general encoding options.
+ * @throws IOException
+ * if an error occurs writing to the underlying output stream.
+ */
+ abstract void encodePayload(final OutputStream out, final VersionedProperties vProps,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ public VersionedProperties fromBytes(final byte[] bytes) throws IOException {
+
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+
+ EncodingOptions encodingOpts = new EncodingOptions(dis);
+
+ if (!checkCanDecodeVersion(encodingOpts)) {
+ throw new IllegalArgumentException(
+ "Invalid data version - cannot process the version read: "
+ + encodingOpts.getEncodingVersion());
+ }
+
+ DataVersionInfo vMetadata = new DataVersionInfo(dis);
+
+ Map<String,String> props = decodePayload(bis, encodingOpts);
+
+ return new VersionedProperties(vMetadata.getDataVersion(), vMetadata.getTimestamp(), props);
+ }
+ }
+
+ abstract boolean checkCanDecodeVersion(final EncodingOptions encodingOpts);
+
+ /**
+ * Extracts the encoding version from the encoded byte array without fully decoding the payload.
+ * This is a convenience method if multiple encodings are present, and should only be required if
+ * upgrading / changing encodings, otherwise a single encoding should be in operation for an
+ * instance at any given time.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getEncodingVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ return new EncodingOptions(dis).getEncodingVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException("Failed to read encoding version from byte array provided",
+ ex);
+ }
+ }
+
+ /**
+ * Extracts the data version from the encoded byte array without fully decoding the payload.
+ * Normally the data version should be obtained from a fully decoded instance of the versioned
+ * properties.
+ * <p>
+ * The cost of reading the byte array from the backing store should be considered verses the
+ * additional cost of decoding - with a goal of reducing data reads from the store preferred.
+ * Generally reading from the store will be followed by some sort of usage which would require the
+ * full decode operation anyway.
+ *
+ * @param bytes
+ * serialized encoded versioned property byte array.
+ * @return the encoding version used to serialize the properties.
+ */
+ public static int getDataVersion(final byte[] bytes) {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bis)) {
+ // skip encoding metadata
+ new EncodingOptions(dis);
+ return new DataVersionInfo(dis).getDataVersion();
+ } catch (NullPointerException | IOException ex) {
+ throw new IllegalArgumentException(
+ "Failed to read data version version from byte array provided", ex);
+ }
+ }
+
+ /**
+ * Decode the payload and any optional encoding specific metadata and return a map of the property
+ * name, value pairs.
+ *
+ * @param inStream
+ * an input stream
+ * @param encodingOpts
+ * the general encoding options.
+ * @return a map of properties name, value pairs.
+ * @throws IOException
+ * if an exception occurs reading from the input stream.
+ */
+ abstract Map<String,String> decodePayload(final InputStream inStream,
+ final EncodingOptions encodingOpts) throws IOException;
+
+ /**
+ * Read the property map from a data input stream as UTF strings. The input stream should be
+ * created configured by sub-classes for the output of the sub-class. If the sub-class uses an
+ * encoding other that UTF strings, they should override this method. An example would be an
+ * encoding that uses JSON to encode the map.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dis
+ * a data input stream
+ * @return the property map
+ * @throws IOException
+ * if an exception occurs reading from the stream.
+ */
+ Map<String,String> readMapAsUTF(DataInputStream dis) throws IOException {
+
+ Map<String,String> aMap = new HashMap<>();
+ int items = dis.readInt();
+
+ for (int i = 0; i < items; i++) {
+ String k = dis.readUTF();
+ String v = dis.readUTF();
+ aMap.put(k, v);
+ }
+ return aMap;
+ }
+
+ /**
+ * Write the property map to the data output stream. The underlying stream is not closed by this
+ * method.
+ * <p>
+ * The handling the properties as UTF strings is one implementation. Subclasses can implement
+ * different mechanism if desired, one example might be using a JSON implementation to encode /
+ * decode the properties.
+ *
+ * @param dos
+ * a data output stream
+ * @param aMap
+ * the property map of k, v string pairs.
+ * @throws IOException
+ * if an exception occurs.
+ */
+ void writeMapAsUTF(final DataOutputStream dos, final Map<String,String> aMap) throws IOException {
+
+ dos.writeInt(aMap.size());
+
+ for (Map.Entry<String,String> e : aMap.entrySet()) {
+ dos.writeUTF(e.getKey());
+ dos.writeUTF(e.getValue());
+ }
+ dos.flush();
+ }
+
+ /**
+ * Helper class for reading / writing versioned properties metadata.
+ */
+ static class DataVersionInfo {
+ private final int dataVersion;
+ private final Instant timestamp;
+
Review comment:
It is used - its is a holdover an maybe overkill for just the two values - but I like the delegation so that it does not need to be handled in multiple places. Will look at using a static method like previous suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717869966
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` has the nice property of being immutable, so you don't have to worry that it's modified anywhere in the test. You can format it to get one pair per line and suppress formatting by wrapping it with `// @formatter:off` and `// @formatter:on`, but it's probably not worth it. I actually prefer two columns per line, as long as both key and value fit (and they do here) because I don't actually care about the data and want to see the subsequent lines of code sooner. But, either way is fine with me. It's definitely not important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701971051
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
+
+ /**
+ * Instantiate an initial instance with default version info and empty map.
+ */
+ public VersionedProperties() {
+ this(NO_VERSION, Instant.now(), Collections.emptyMap());
+ }
+
+ /**
+ * Instantiate an initial instance with default version info and provided property map.
+ *
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(Map<String,String> props) {
+ this(NO_VERSION, Instant.now(), props);
+ }
+
+ /**
+ * Instantiate an instance and set the initial properties to the provided values.
+ *
+ * @param dataVersion
+ * version info with data version and timestamp.
+ * @param timestamp
+ * timestamp of this version.
+ * @param props
+ * optional map of initial property key, value pairs. The properties are assumed to have
+ * been previously validated (if required)
+ */
+ public VersionedProperties(final int dataVersion, final Instant timestamp,
+ final Map<String,String> props) {
+ this.dataVersion = dataVersion;
+ this.timestamp = timestamp;
+ if (Objects.nonNull(props)) {
+ this.props = new ImmutableMap.Builder<String,String>().putAll(props).build();
+ } else {
+ this.props = new ImmutableMap.Builder<String,String>().build();
+ }
+ }
+
+ /**
+ * Get an unmodifiable map with all property key,value pairs.
+ *
+ * @return An unmodifiable view of the property key, value pairs.
+ */
+ public Map<String,String> getProperties() {
+ return props;
+ }
+
+ /**
+ * Get the current data version. The version should match the node version of the stored data. The
+ * value should be used on data writes as the expected version. If the data write fails do to an
+ * unexpected version, it signals that the node version has changed since the instance was
+ * instantiated and encoded.
+ *
+ * @return 0 for initial version, otherwise the data version when the properties were serialized.
+ */
+ public int getDataVersion() {
+ return Math.max(dataVersion, 0);
+ }
+
+ /**
+ * Calculates the version that should be stored when serialized. The serialized version, when
+ * stored, should match the version that will be assigned. This way, data reading the serialized
+ * version can compare the stored version with the node version at any time to detect if the node
+ * version has been updated.
+ * <p>
+ * The initialization of the data version to a negative value allows this value to be calculated
+ * correctly for the first serialization. On the first store, the expected version will be 0.
+ *
+ * @return the next version number that should be serialized, or 0 if this is the initial version.
+ */
+ public int getNextVersion() {
+ return Math.max(dataVersion + 1, 0);
+ }
+
+ /**
+ * The timestamp of the instance when created or last modified.
+ *
+ * @return the timestamp of the instance.
+ */
+ public Instant getTimestamp() {
+ return Instant.from(timestamp);
+ }
+
+ /**
+ * The timestamp formatted as an ISO 8601 string with format of
+ * {@code YYYY-MM-DDTHH:mm:ss.SSSSSSZ}
+ *
+ * @return a formatted timestamp string.
+ */
+ public String getTimestampISO() {
+ return tsFormatter.format(timestamp);
+ }
+
+ /**
+ * Update a single property. If a property already exists it is overwritten.
+ * <p>
+ * It is much more efficient to add multiple properties at a time rather than one by one. Because
+ * instances of this class are immutable, the creates a new copy of the properties. Other
+ * processes will continue to see original values retrieved from the data store. Other processes
+ * will be updated when the instance is encoded and stored in the data store.
+ *
+ * @param key
+ * the property name.
+ * @param value
+ * the property value.
+ * @return A new instance of this class with the property added or updated.
+ */
+ public VersionedProperties update(final String key, final String value) {
Review comment:
I'm wondering if update() could be renamed to something that implies that the properties are being added. "update" could mean other things too. Not a big deal just thought I would bring it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717538865
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` in this case is a little less readable in my opinion.
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` in this case is a little less readable in my opinion. For me, it's a bit quicker to read with each pair on its own line. Not a huge difference either way though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r717538865
##########
File path: server/base/src/test/java/org/apache/accumulo/server/conf/codec/VersionedPropEncryptCodecTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.KeyGenerator;
+import javax.crypto.SecretKey;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VersionedPropEncryptCodecTest {
+
+ private final Logger log = LoggerFactory.getLogger(VersionedPropEncryptCodecTest.class);
+
+ /**
+ * Perform a round trip - encode, decode set of operations.
+ *
+ * @throws Exception
+ * an exception is a test failure.
+ */
+ @Test
+ public void roundTripSample() throws Exception {
+
+ // set-up sample "secret" key - for testing only.
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropEncryptCodec.GCMCipherParams cipherProps =
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt);
+
+ Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+ cipher.init(Cipher.ENCRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+
+ byte[] payload;
+
+ try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
+
+ CipherOutputStream cos = new CipherOutputStream(bos, cipher);
+
+ DataOutputStream dos = new DataOutputStream(cos);
+
+ dos.writeUTF("A");
+ dos.writeUTF("B");
+ dos.writeUTF("C");
+
+ cos.flush();
+ cos.close();
+
+ payload = bos.toByteArray();
+
+ log.debug("Output: {}", payload);
+
+ }
+
+ cipher.init(Cipher.DECRYPT_MODE, cipherProps.getSecretKey(), cipherProps.getParameterSpec());
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(payload)) {
+
+ // write the property map keys, values.
+ try (CipherInputStream cis = new CipherInputStream(bis, cipher);
+
+ DataInputStream cdis = new DataInputStream(cis)) {
+
+ assertEquals("A", cdis.readUTF());
+ assertEquals("B", cdis.readUTF());
+ assertEquals("C", cdis.readUTF());
+ }
+ }
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryption() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ Map<String,String> p = new HashMap<>();
+ p.put("k1", "v1");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
+
+ final char[] pass = {'a', 'b', 'c'};
+ final byte[] salt = {1, 2, 3};
+
+ VersionedPropCodec encoder = VersionedPropEncryptCodec.codec(false,
+ new VersionedPropEncryptCodec.GCMCipherParams(pass, salt));
+
+ byte[] encodedBytes = encoder.toBytes(vProps);
+
+ log.debug("Encoded: {}", encodedBytes);
+
+ VersionedProperties decodedProps = encoder.fromBytes(encodedBytes);
+
+ log.debug("Decoded: {}", decodedProps.print(true));
+
+ assertEquals(vProps.getProperties(), decodedProps.getProperties());
+
+ // validate that the expected node version matches original version.
+ assertEquals(aVersion, vProps.getDataVersion());
+
+ // validate encoded version incremented.
+ assertEquals(aVersion + 1, decodedProps.getDataVersion());
+
+ assertEquals("encoded version should be 1 up", aVersion + 1, decodedProps.getDataVersion());
+ assertEquals("version written should be the source next version", vProps.getNextVersion(),
+ decodedProps.getDataVersion());
+ assertEquals("the next version in decoded should be +2", aVersion + 2,
+ decodedProps.getNextVersion());
+
+ assertTrue("timestamp should be now or earlier",
+ vProps.getTimestamp().compareTo(Instant.now()) <= 0);
+
+ }
+
+ /**
+ * Validate versioning with something other than default.
+ */
+ @Test
+ public void roundTripEncryptionCompressed() throws Exception {
+
+ int aVersion = 13;
+ Instant now = Instant.now();
+
+ // compression friendly
+ Map<String,String> p = new HashMap<>();
+ p.put("accumulo.prop.key_name.1", "value1");
+ p.put("accumulo.prop.key_name.2", "value2");
+ p.put("accumulo.prop.key_name.3", "value3");
+ p.put("accumulo.prop.key_name.4", "value4");
+ p.put("accumulo.prop.key_name.5", "value5");
+ p.put("accumulo.prop.key_name.6", "value9");
+
+ VersionedProperties vProps = new VersionedProperties(aVersion, now, p);
Review comment:
`Map.of` in this case is a little less readable in my opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701908453
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly and without additional processing.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ // Adding an encoding version must be done as an addition. Do not change or delete previous
+ // version numbers
+ public static final int EncodingVersion_1_0 = 1;
+ public static final int EXPERIMENTAL_CIPHER_ENCODING_1_0 = 999;
+
+ private final int encodingVersion;
+ private final boolean compress;
+
+ EncodingOptions(final int encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ /**
+ * Instantiate encoding options to use version 1.0 encoding settings.
+ *
+ * @param compress
+ * when true compress the property map.
+ * @return the encoding options.
+ */
+ public static EncodingOptions V1_0(final boolean compress) {
+ return new EncodingOptions(EncodingVersion_1_0, compress);
+ }
+
+ /**
+ * Instantiate an instance of EncodingOptions reading the values from an input stream. Typically,
+ * the stream will be obtained from reading a byte array from a data store and then creating a
+ * stream the reads from that array,
Review comment:
```suggestion
* stream that reads from that array,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703419153
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
Just curious - what's the advantage of ImmutableMap over Map.of() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r703597457
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that values being
+ * written to the backend store have not changed. This metadata should be written / appear early in
+ * the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
+ * unnecessary.
+ * <p>
+ * Note: Avoid using -1 because that has significance in ZooKeeper - writing a ZooKeeper node with a
+ * version of -1 disables the ZooKeeper expected version checking and just overwrites the node.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class VersionedProperties {
+
+ public static final DateTimeFormatter tsFormatter =
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneId.from(ZoneOffset.UTC));
+ // flag value for initialization - on store both the version and next version should be 0.
+ private static final int NO_VERSION = -2;
+ private final int dataVersion;
+ private final Instant timestamp;
+ private final ImmutableMap<String,String> props;
Review comment:
The changes are propagated through ZooKeeper. On a data change, the watcher will fire, which the propagates the notification to those interested - that code is in the next layer up. This way, all processes on the "local" server and the tserver(s) and other processes only see "atomic" changes in properties - with ZooKeeper being the truth.
Otherwise, say something in the manager was updating properties - until those changes are committed to ZooKeeper, all master processes will have the same, original values. Once stored, those processes will get updated values on next read from the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r701941244
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/VersionedProperties.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Version properties maintain a {@code Map<String,String>}; of property k,v pairs along with
+ * versioning information metadata.
+ * <p>
+ * The metadata used to verify cached values match stored values. Storing the metadata with the
+ * properties allows for comparison of properties and can be used to ensure that vales being written
+ * to the backend store have not changed. This metadata should be written / appear early in the
+ * encoded bytes and be uncompressed so that decisions can be made that may make deserialization
Review comment:
```suggestion
* properties allows for comparison of properties and can be used to ensure that values being
* written to the backend store have not changed. This metadata should be written / appear early in
* the encoded bytes and be uncompressed so that decisions can be made that may make deserialization
```
The missing 'u' in values pushed these to new lines after formatting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #2224: Versioned Properties - refactored to address PR comments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2224:
URL: https://github.com/apache/accumulo/pull/2224#discussion_r683943792
##########
File path: server/base/src/main/java/org/apache/accumulo/server/conf/codec/EncodingOptions.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server.conf.codec;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.StringJoiner;
+
+/**
+ * Serialization metadata to allow for evolution of the encoding used for property storage. This
+ * info is expected to be stored first in the serialization and uncompressed so that the handling of
+ * subsequent fields and data can be processed correctly.
+ * <p>
+ * Instances of this class are immutable.
+ */
+public class EncodingOptions {
+
+ public static final EncodingOptions COMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, true);
+
+ public static final EncodingOptions UNCOMPRESSED_V1 =
+ new EncodingOptions(EncodingOptions.EncodingVersion.V1_0, false);
+
+ private final EncodingVersion encodingVersion;
+ private final boolean compress;
+
+ public EncodingOptions(EncodingVersion encodingVersion, final boolean compress) {
+ this.encodingVersion = encodingVersion;
+ this.compress = compress;
+ }
+
+ public EncodingOptions(final DataInputStream dis) throws IOException {
+ encodingVersion = EncodingVersion.byId(dis.readInt());
+ compress = dis.readBoolean();
+ }
+
+ public void encode(final DataOutputStream dos) throws IOException {
+ dos.writeInt(encodingVersion.id);
+ dos.writeBoolean(compress);
+ }
+
+ public EncodingVersion getEncodingVersion() {
+ return encodingVersion;
+ }
+
+ public boolean isCompressed() {
+ return compress;
+ }
+
+ @Override
+ public String toString() {
+ return new StringJoiner(", ", EncodingOptions.class.getSimpleName() + "[", "]")
+ .add("encodingVersion=" + encodingVersion).toString();
+ }
+
+ /**
+ * Provides a strong typing of the known encoding versions and allows the version id to be encoded
+ * as an integer. Adding an encoding type must be done as an addition and not change or delete
+ * previous versions or numbering to preserve compatibility.
+ */
+ public enum EncodingVersion {
Review comment:
What about just using constants for convenience names? Also, an integer might work better in future if you have an implementation that is compatible with a range of previous versions. Checking ranges for integers might be easier in that case than enumerating a set of versions by name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org