You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "frankgh (via GitHub)" <gi...@apache.org> on 2024/02/05 19:52:34 UTC

[PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

frankgh opened a new pull request, #38:
URL: https://github.com/apache/cassandra-analytics/pull/38

   This commit adds the ability to use the newly supported in Cassandra Sidecar XXhash32 digest algorithm. The commit allows for backwards compatibility to perform MD5 checksumming, but it now defaults to XXHash32.
   
   A new Writer option is added:
   
   ```
   .option(WriterOptions.DIGEST_TYPE.name(), "XXHASH32") // or
   .option(WriterOptions.DIGEST_TYPE.name(), "MD5")
   ```
   
   This option defaults to XXHash32, when not provided, but it can be configured to use the legacy MD5 algorithm.
   
   Path by Francisco Guerrero; Reviewed by TBD for CASSANDRA-19369


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#discussion_r1482390314


##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/BulkSparkConf.java:
##########
@@ -128,12 +128,13 @@ public class BulkSparkConf implements Serializable
     protected boolean useOpenSsl;
     protected int ringRetryCount;
     protected final Set<String> blockedInstances;
+    protected final DigestTypeOption digestTypeOption;
 
     public BulkSparkConf(SparkConf conf, Map<String, String> options)
     {
         this.conf = conf;
         Optional<Integer> sidecarPortFromOptions = MapUtils.getOptionalInt(options, WriterOptions.SIDECAR_PORT.name(), "sidecar port");
-        this.userProvidedSidecarPort = sidecarPortFromOptions.isPresent() ? sidecarPortFromOptions.get() : getOptionalInt(SIDECAR_PORT).orElse(-1);
+        this.userProvidedSidecarPort = sidecarPortFromOptions.orElseGet(() -> getOptionalInt(SIDECAR_PORT).orElse(-1));

Review Comment:
   nit: ternary operator reads better than orElseGet



##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/common/Digest.java:
##########
@@ -19,25 +19,13 @@
 
 package org.apache.cassandra.spark.common;
 
-import java.security.MessageDigest;
-import java.util.Base64;
-
-public final class MD5Hash
+/**
+ * Interface that represents a checksum digest

Review Comment:
   nit: checksum kind of equals to digest. It is repeat like "I have two animal cats". Checksum can be removed



##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/DigestTypeOption.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.spark.bulkwriter;
+
+import org.apache.cassandra.spark.utils.DigestProvider;
+import org.apache.cassandra.spark.utils.MD5DigestProvider;
+import org.apache.cassandra.spark.utils.XXHash32DigestProvider;
+
+/**
+ * Represents the user-provided digest type configuration to be used to validate SSTable files during bulk writes
+ */
+public enum DigestTypeOption
+{
+    /**
+     * Represents an MD5 digest type option. This option is supported for legacy reasons, but its use
+     * is strongly discouraged.
+     */
+    MD5
+    {
+        @Override
+        DigestProvider provider()
+        {
+            return new MD5DigestProvider();
+        }
+    },
+
+    /**
+     * Represents an xxhash32 digest type option
+     */
+    XXHASH32
+    {
+        @Override
+        DigestProvider provider()
+        {
+            return new XXHash32DigestProvider();
+        }
+    };
+
+    /**
+     * @return the provider for the configured digest type
+     */
+    abstract DigestProvider provider();

Review Comment:
   How about extracting out as an interface? And implement the interface in the enum. Potentially, one cannot change the source and supply a different impl.



##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/RecordWriter.java:
##########
@@ -400,4 +403,23 @@ private StreamSession createStreamSession(TaskContext taskContext)
         LOGGER.info("[{}] Creating stream session for range={}", taskContext.partitionId(), tokenRange);
         return new StreamSession(writerContext, getStreamId(taskContext), tokenRange, failureHandler);
     }
+
+    /**
+     * Functional interface that helps with supplying {@link SSTableWriter} instances.
+     */
+    public interface SSTableWriterSupplier

Review Comment:
   nit: supplier by convention takes no parameter. It is the factory here. SSTableWriterFactory wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "frankgh (via GitHub)" <gi...@apache.org>.
frankgh closed pull request #38: CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables
URL: https://github.com/apache/cassandra-analytics/pull/38


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "frankgh (via GitHub)" <gi...@apache.org>.
frankgh commented on code in PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#discussion_r1482954608


##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/DigestTypeOption.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.spark.bulkwriter;
+
+import org.apache.cassandra.spark.utils.DigestProvider;
+import org.apache.cassandra.spark.utils.MD5DigestProvider;
+import org.apache.cassandra.spark.utils.XXHash32DigestProvider;
+
+/**
+ * Represents the user-provided digest type configuration to be used to validate SSTable files during bulk writes
+ */
+public enum DigestTypeOption implements DigestTypeProvider

Review Comment:
   open to suggestions for the interface name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "frankgh (via GitHub)" <gi...@apache.org>.
frankgh commented on code in PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#discussion_r1482923367


##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/BulkSparkConf.java:
##########
@@ -128,12 +128,13 @@ public class BulkSparkConf implements Serializable
     protected boolean useOpenSsl;
     protected int ringRetryCount;
     protected final Set<String> blockedInstances;
+    protected final DigestTypeOption digestTypeOption;
 
     public BulkSparkConf(SparkConf conf, Map<String, String> options)
     {
         this.conf = conf;
         Optional<Integer> sidecarPortFromOptions = MapUtils.getOptionalInt(options, WriterOptions.SIDECAR_PORT.name(), "sidecar port");
-        this.userProvidedSidecarPort = sidecarPortFromOptions.isPresent() ? sidecarPortFromOptions.get() : getOptionalInt(SIDECAR_PORT).orElse(-1);
+        this.userProvidedSidecarPort = sidecarPortFromOptions.orElseGet(() -> getOptionalInt(SIDECAR_PORT).orElse(-1));

Review Comment:
   I actually prefer the orElseGet approach here. But I will revert since it's not related to this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "frankgh (via GitHub)" <gi...@apache.org>.
frankgh commented on PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#issuecomment-1942964282

   Closed via https://github.com/apache/cassandra-analytics/commit/dc0e79b9c483562ec0920d69e886715eb329c426


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "frankgh (via GitHub)" <gi...@apache.org>.
frankgh commented on code in PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#discussion_r1480589895


##########
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/QuoteIdentifiersWriteTest.java:
##########
@@ -76,58 +59,31 @@ class QuoteIdentifiersWriteTest extends SharedClusterSparkIntegrationTestBase
                   uniqueTestTableFullName("keyspace"), // keyspace is a reserved word
                   uniqueTestTableFullName(TEST_KEYSPACE, "QuOtEd_TaBlE"),
                   new QualifiedName(TEST_KEYSPACE, "table"));  // table is a reserved word
+    Dataset<Row> df;
 
     @ParameterizedTest(name = "{index} => table={0}")
     @MethodSource("testInputs")
     void testQuoteIdentifiersBulkWrite(QualifiedName tableName)
     {
-        SparkSession spark = getOrCreateSparkSession();
-        SparkContext sc = spark.sparkContext();
-        JavaSparkContext javaSparkContext = JavaSparkContext.fromSparkContext(sc);
-        SQLContext sql = spark.sqlContext();
-
-        int parallelism = sc.defaultParallelism();
-        JavaRDD<Row> rows = genDataset(javaSparkContext, ROW_COUNT, parallelism);
-        Dataset<Row> df = sql.createDataFrame(rows, writeSchema());
-
         bulkWriterDataFrameWriter(df, tableName).option(WriterOptions.QUOTE_IDENTIFIERS.name(), "true")
                                                 .save();
-        validateWrites(tableName, rows);
+        validateWrites(cluster, tableName, df);

Review Comment:
   consolidated this code in a helper class 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [PR] CASSANDRA-19369 Use XXHash32 for digest calculation of SSTables [cassandra-analytics]

Posted by "JeetKunDoug (via GitHub)" <gi...@apache.org>.
JeetKunDoug commented on code in PR #38:
URL: https://github.com/apache/cassandra-analytics/pull/38#discussion_r1484658554


##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/BulkSparkConf.java:
##########
@@ -128,12 +128,13 @@ public class BulkSparkConf implements Serializable
     protected boolean useOpenSsl;
     protected int ringRetryCount;
     protected final Set<String> blockedInstances;
+    protected final DigestTypeOption digestTypeOption;
 
     public BulkSparkConf(SparkConf conf, Map<String, String> options)
     {
         this.conf = conf;
         Optional<Integer> sidecarPortFromOptions = MapUtils.getOptionalInt(options, WriterOptions.SIDECAR_PORT.name(), "sidecar port");
-        this.userProvidedSidecarPort = sidecarPortFromOptions.isPresent() ? sidecarPortFromOptions.get() : getOptionalInt(SIDECAR_PORT).orElse(-1);
+        this.userProvidedSidecarPort = sidecarPortFromOptions.orElseGet(() -> getOptionalInt(SIDECAR_PORT).orElse(-1));

Review Comment:
   Agree with @frankgh here... I prefer `orElseGet` when using optionals.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org