You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "lukasz-antoniak (via GitHub)" <gi...@apache.org> on 2025/03/11 12:09:42 UTC
[PR] Support CQL duration type [cassandra-analytics]
lukasz-antoniak opened a new pull request, #102:
URL: https://github.com/apache/cassandra-analytics/pull/102
Adds support for CQL `duration` type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1995977546
##########
cassandra-analytics-common/src/main/java/org/apache/cassandra/bridge/type/InternalDuration.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.bridge.type;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Analytics-internal Duration type to bridge CQL Duration type.
+ */
+public class InternalDuration implements Serializable
+{
+ private static final long serialVersionUID = -6867844786318937949L;
+ public final int months;
+ public final int days;
+ public final long nanoseconds;
+
+ public InternalDuration(int months, int days, long nanoseconds)
+ {
+ this.months = months;
+ this.days = days;
+ this.nanoseconds = nanoseconds;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o)
+ {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass())
+ {
+ return false;
+ }
+ InternalDuration that = (InternalDuration) o;
+ return months == that.months &&
+ days == that.days &&
+ nanoseconds == that.nanoseconds;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(months, days, nanoseconds);
+ }
+
+ @Override
+ public String toString()
+ {
+ StringBuilder builder = new StringBuilder();
+ if (this.months < 0 || this.days < 0 || this.nanoseconds < 0L)
+ {
+ builder.append('-');
+ }
Review Comment:
CQL duration validate all non-zero fields are all negative or all positive. Since this is consuming from data that is validated by Cassandra already, I think this condition is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1994137469
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Disregard the renaming suggestion. I just refreshed and found out `CqlDuration` is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2725427736
Ah. Just realized one thing. Can you update `changes.txt`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1993301580
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
FWIW, I cannot easily move logic of `Duration` to `SparkDuration`, because Cassandra serializer expects `org.apache.cassandra.cql3.Duration`. To achieve it, I would need to add Cassandra dependency to `cassandra-analytics-spark-converter` module. Is that OK?
I am not sure if `toSparkSqlType()` shall return `org.apache.cassandra.cql.Duration`, as this is C* type. I guess I might have misunderstood your suggestion.
```
@Override
public Object toSparkSqlType(@NotNull Object value, boolean isFrozen)
{
CalendarInterval cl = (CalendarInterval) value;
return Duration.newInstance(cl.months, cl.days, cl.microseconds * 1000);
}
```
```
Caused by: java.lang.ClassCastException: class org.apache.cassandra.cql3.Duration cannot be cast to class org.apache.spark.unsafe.types.CalendarInterval (org.apache.cassandra.cql3.Duration and org.apache.spark.unsafe.types.CalendarInterval are in unnamed module of loader 'app')
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getInterval(rows.scala:49)
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getInterval$(rows.scala:49)
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getInterval(rows.scala:195)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
```
I have tried to exclude Spark dependency from `cassandra-four-zero-types` module in various ways. Best I could come up with, was to introduce a POJO `CqlDuration` that is able to map internally to `CalendarInterval`. See commit: https://github.com/apache/cassandra-analytics/pull/102/commits/df40c73b4e73728d8455609526314f01e845df13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1990340830
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Could you not pull in Spark here? I think you can work around by moving your `CalendarIntervalSerializer` class to `SparkDuration` and performing the conversion in `toSparkSqlType`. An effort was made last year to decouple the types from Spark, so bulk reads can be performed outside of Spark and also because CDC does not necessarily run in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1994134904
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
For the write-path, the converters are located at a different place. Check out `org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter`.
They should be consolidated, but it is what it is as of today.
A nit on the class name. How about calling it `InternalDuration` instead of `DurationWrapper`? Since it does not wrap anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2716025092
Thanks for adding this! My one request is to keep the Spark conversion code in `SparkDuration` to keep `Duration` decoupled from Spark.;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1994136552
##########
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/data/types/Duration.java:
##########
@@ -32,8 +46,148 @@ public String name()
}
@Override
- public boolean isSupported()
+ public AbstractType<?> dataType()
+ {
+ return DurationType.instance;
+ }
+
+ public <T> TypeSerializer<T> serializer()
+ {
+ return (TypeSerializer<T>) CalendarIntervalSerializer.INSTANCE;
Review Comment:
I think we can continue using `DurationSerializer.instance`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2725014629
@yifan-c, thank you very much for the review! I did not notice your commit and was doing changes myself, but then pushed your stuff to my branch. I think I have addressed point 4, so feel free to review PR once you have time. Can you also check new implementation of `testDuration()` test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2732131007
Updated CI link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2731189520
The changes look good to me. However, the CI link you shared does not open. It gives me 404. Can you double check and shared it at https://issues.apache.org/jira/browse/CASSANALYTICS-18?
Re: squashing. You do not need to squash them. For this repo, the convention is to merge from the PR page directly, using the `Squash and merge` button. (There is one single trunk. so why the trouble of the manual work :p)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1994136552
##########
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/data/types/Duration.java:
##########
@@ -32,8 +46,148 @@ public String name()
}
@Override
- public boolean isSupported()
+ public AbstractType<?> dataType()
+ {
+ return DurationType.instance;
+ }
+
+ public <T> TypeSerializer<T> serializer()
+ {
+ return (TypeSerializer<T>) CalendarIntervalSerializer.INSTANCE;
Review Comment:
~I think we can continue using `DurationSerializer.instance`.~
_Update_
I tried out `DurationSerializer.instance` and other necessary changes. It does not work for nested type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1993301580
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
FWIW, I cannot easily move logic of `Duration` to `SparkDuration`, because Cassandra serializer expects `org.apache.cassandra.cql3.Duration`. To achieve it, I would need to add Cassandra dependency to `cassandra-analytics-spark-converter` module. Is that OK?
I am not sure if `toSparkSqlType()` shall return `org.apache.cassandra.cql.Duration`, as this is C* type. I guess I might have misunderstood your intention.
```
@Override
public Object toSparkSqlType(@NotNull Object value, boolean isFrozen)
{
CalendarInterval cl = (CalendarInterval) value;
return Duration.newInstance(cl.months, cl.days, cl.microseconds * 1000);
}
```
```
Caused by: java.lang.ClassCastException: class org.apache.cassandra.cql3.Duration cannot be cast to class org.apache.spark.unsafe.types.CalendarInterval (org.apache.cassandra.cql3.Duration and org.apache.spark.unsafe.types.CalendarInterval are in unnamed module of loader 'app')
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getInterval(rows.scala:49)
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getInterval$(rows.scala:49)
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getInterval(rows.scala:195)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
```
I have tried to exclude Spark dependency from `cassandra-four-zero-types` module in various ways. Best I could come up with, was to introduce a POJO `CqlDuration` that is able to map internally to `CalendarInterval`. See commit: https://github.com/apache/cassandra-analytics/pull/102/commits/df40c73b4e73728d8455609526314f01e845df13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1994061550
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Clear. Where shall I convert `CalendarInterval` (generated by Spark) to `DurationWrapper` to correctly support write-path from Spark to Cassandra. We do have `TypeConverter`, but it is used only in the opposite direction (deserialization).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1993917649
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Could you convert it into a an intermediate JDK type from `Duration`? Even if it's a simple wrapper like:
```
public class DurationWrapper {
public int months;
public int days;
public long microseconds;
}
```
And then convert `DurationWrapper ` to `CalendarInterval` in `SparkDuration`? It just means we can consume Duration without the Spark dependency (and in CDC).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1993920716
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Or even `java.time.Duration` if that works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2729179651
@jberragan, removed the dependency line. Sorry for missing that before. Pipeline link added. I had to move creation of `CalendarInterval` to `cassandra-analytics-spark-converter` module, because API changes between Spark 2 and 3.
@yifan-c, I have reviewed changes from this PR and they look good to me. Can you have final look? Shall I squash your commit with mine or leave 3 commits in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2719276986
Hi @lukasz-antoniak , thank you for the patch! Can you create a JIRA ticket at https://issues.apache.org/jira/projects/CASSANALYTICS/ if not yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c merged PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on code in PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#discussion_r1996292460
##########
cassandra-four-zero-types/build.gradle:
##########
@@ -33,6 +33,7 @@ dependencies {
compileOnly project(":cassandra-analytics-common")
compileOnly(project(path: ':cassandra-four-zero', configuration: 'shadow'))
compileOnly "com.esotericsoftware:kryo-shaded:${kryoVersion}"
+ compileOnly(group: "${sparkGroupId}", name: "spark-core_${scalaMajorVersion}", version: "${project.rootProject.sparkVersion}")
Review Comment:
Could you remember to remove this dependency, it seems to compile without it now, after moving to `SparkDuration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "jberragan (via GitHub)" <gi...@apache.org>.
jberragan commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2725787934
If you could remember to remove this line:
https://github.com/apache/cassandra-analytics/pull/102/files#diff-a22af2fd084620cc2bec9d43f77ea8d8c050f1c59b316f9970f6b3e2b9d1ec10R36
Otherwise I'm +1 (nb)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [PR] CASSANALYTICS-18: Support CQL duration type [cassandra-analytics]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #102:
URL: https://github.com/apache/cassandra-analytics/pull/102#issuecomment-2725423783
Did you setup circleCI for the project? If so, please comment the CI link in the JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org