You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cboumalh (via GitHub)" <gi...@apache.org> on 2026/02/16 19:20:44 UTC
[PR] [SPARK-55558][SQL] Add Support for Tuple/Theta Set Operations [spark]
cboumalh opened a new pull request, #54338:
URL: https://github.com/apache/spark/pull/54338
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR adds support for set operations (union, intersection, difference) between Apache DataSketches TupleSketch and ThetaSketch objects. Specifically, it adds the following new SQL functions:
New Functions:
**tuple_union_theta_double** - Unions a TupleSketch (with double summaries) with a ThetaSketch
**tuple_union_theta_integer** - Unions a TupleSketch (with integer summaries) with a ThetaSketch
**tuple_intersection_theta_double** - Intersects a TupleSketch (with double summaries) with a ThetaSketch
**tuple_intersection_theta_integer** - Intersects a TupleSketch (with integer summaries) with a ThetaSketch
**tuple_difference_theta_double** - Subtracts a ThetaSketch from a TupleSketch (with double summaries)
**tuple_difference_theta_integer** - Subtracts a ThetaSketch from a TupleSketch (with integer summaries)
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR extends the existing TupleSketch functionality to support interoperability with ThetaSketch objects. Previously, users could only perform set operations between two TupleSketches or two ThetaSketches, but not between different sketch types. This enhancement allows users to:
1. Combine cardinality estimates from TupleSketch (which tracks summaries) with ThetaSketch (cardinality-only)
2. Perform set operations across different sketch types in the same query
3. Migrate from ThetaSketch to TupleSketch incrementally by supporting mixed operations
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. This PR adds six new SQL functions for performing set operations between TupleSketch and ThetaSketch objects:
```
-- Union a TupleSketch with a ThetaSketch
SELECT tuple_sketch_estimate_double(
tuple_union_theta_double(
tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2)))
FROM table;
-- Intersect a TupleSketch with a ThetaSketch
SELECT tuple_sketch_estimate_double(
tuple_intersection_theta_double(
tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2)))
FROM table;
-- Subtract a ThetaSketch from a TupleSketch
SELECT tuple_sketch_estimate_double(
tuple_difference_theta_double(
tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2)))
FROM table;
```
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Comprehensive SQL tests were added in [sql-tests/inputs/tuplesketch.sql](vscode-webview://1e9ujo6kk2dbu8vc5clrjpeshok5asvo9j08ij02fatnep06rg21/sql/core/src/test/resources/sql-tests/inputs/tuplesketch.sql).
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Sonnet 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #54338:
URL: https://github.com/apache/spark/pull/54338#discussion_r2828290196
##########
docs/sql-ref-sketch-aggregates.md:
##########
@@ -748,6 +751,138 @@ FROM VALUES (1, 10.0, 4, 40.0), (2, 20.0, 4, 40.0), (3, 30.0, 5, 50.0), (4, 40.0
---
+### tuple_union_theta_*
+
+Merges a Tuple sketch with a Theta sketch using union (scalar function). This combines distinct keys from both sketches, with Theta sketch entries assigned a default summary value.
Review Comment:
yep, I added those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3915889978
cc @dtenedor @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3946909759
LGTM, reviewed al the tests again, thanks again for working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3916679592
I asked a question offline, we can think about tradeoffs between adding new functionality from these functions versus increasing complexity in the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #54338:
URL: https://github.com/apache/spark/pull/54338#discussion_r2828318948
##########
sql/core/src/test/resources/sql-tests/inputs/tuplesketch.sql:
##########
@@ -1013,6 +1146,46 @@ SELECT tuple_sketch_estimate_integer(
lgNomEntries => 14))
FROM t_int_int_1_5_through_7_11;
+-- Test tuple_union_theta_double with named parameters - only required params
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - setting only lgNomEntries
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2),
+ lgNomEntries => 14))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - setting only mode
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2),
+ mode => 'max'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - different order
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ mode => 'min',
+ lgNomEntries => 15,
+ second => theta_sketch_agg(key2),
+ first => tuple_sketch_agg_double(key1, val1)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer with named parameters - setting only lgNomEntries
Review Comment:
yep I added more SQL tests for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3927769804
@dtenedor Thanks for taking a look! I addressed your comments. Please let me know if there is anything I missed 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3922238092
@cboumalh do you want to sync the latest master branch contents and push a commit to resolve the merge conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3922236646
We talked offline, it looks like there is a valid use case for this. We can proceed with the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #54338:
URL: https://github.com/apache/spark/pull/54338#discussion_r2828310445
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala:
##########
@@ -97,6 +97,78 @@ case class TupleDifferenceInteger(left: Expression, right: Expression)
}
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
+ Datasketches ThetaSketch from a TupleSketch with double summary data type using a TupleSketch
+ AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch. """,
+ examples = """
+ Examples:
+ > SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (5, 5.0D, 4), (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 1) tab(col1, val1, col2);
+ 2.0
+ """,
+ group = "sketch_funcs",
+ since = "4.2.0")
+// scalastyle:on line.size.limit
+case class TupleDifferenceThetaDouble(left: Expression, right: Expression)
Review Comment:
The second byte of every sketch represents the type of sketch it is. I added a check in both the theta and tuple sketches deserialization (heapify, wrap) functions with their corresponding errors. This way this error will come up in every function call where the wrong sketch type is passed in, not just the ones defined in this PR. Let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #54338:
URL: https://github.com/apache/spark/pull/54338#issuecomment-3946911785
Merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #54338:
URL: https://github.com/apache/spark/pull/54338#discussion_r2823742380
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -3904,6 +3904,213 @@ class DataFrameAggregateSuite extends QueryTest
assert(estimate == 1.0)
}
+ test("SPARK-55558: tuple_difference_theta_double basic functionality") {
+ val df1 = Seq((1, 1.5), (2, 2.5), (3, 3.5), (5, 5.5)).toDF("key", "summary")
+ val df2 = Seq(1, 2, 4).toDF("value")
+
+ val tupleSketchDf = df1.agg(tuple_sketch_agg_double($"key", $"summary").alias("tuple_sketch"))
+ val thetaSketchDf = df2.agg(theta_sketch_agg($"value").alias("theta_sketch"))
+
+ val joined = tupleSketchDf.crossJoin(thetaSketchDf)
+
+ // Test difference (keys in tuple_sketch but not in theta_sketch: 3 and 5)
+ val difference = joined
+ .select(tuple_difference_theta_double($"tuple_sketch", $"theta_sketch"))
+ .collect()(0)(0)
+ assert(difference != null)
+ assert(difference.asInstanceOf[Array[Byte]].length > 0)
+
+ // Test with column names
+ val difference2 = joined
+ .select(tuple_difference_theta_double("tuple_sketch", "theta_sketch"))
+ .collect()(0)(0)
+ assert(difference2 != null)
+
+ // Verify estimate from difference
+ val estimate = joined
+ .select(tuple_sketch_estimate_double(
+ tuple_difference_theta_double($"tuple_sketch", $"theta_sketch")))
+ .collect()(0)(0)
+ assert(estimate == 2.0)
Review Comment:
We're checking the cardinality estimates here; can we also check the summary values for correctness?
##########
sql/core/src/test/resources/sql-tests/inputs/tuplesketch.sql:
##########
@@ -1013,6 +1146,46 @@ SELECT tuple_sketch_estimate_integer(
lgNomEntries => 14))
FROM t_int_int_1_5_through_7_11;
+-- Test tuple_union_theta_double with named parameters - only required params
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - setting only lgNomEntries
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2),
+ lgNomEntries => 14))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - setting only mode
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ first => tuple_sketch_agg_double(key1, val1),
+ second => theta_sketch_agg(key2),
+ mode => 'max'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double with named parameters - different order
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ mode => 'min',
+ lgNomEntries => 15,
+ second => theta_sketch_agg(key2),
+ first => tuple_sketch_agg_double(key1, val1)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer with named parameters - setting only lgNomEntries
Review Comment:
Other test ideas:
What happens when one or both inputs are empty sketches? e.g.:
* Union of a non-empty TupleSketch with an empty ThetaSketch
* Intersection of a TupleSketch with an empty ThetaSketch (should return empty)
* Difference of a TupleSketch from an empty ThetaSketch (should return the TupleSketch)
Can we exercise null inputs for one or both arguments?
Can we validate that after intersection, the summaries should be preserved from the tuple sketch side (since theta entries get identity defaults)? After difference, remaining entries should retain their original summaries.
##########
sql/core/src/test/resources/sql-tests/inputs/tuplesketch.sql:
##########
@@ -371,6 +371,139 @@ SELECT tuple_sketch_estimate_integer(
tuple_sketch_agg_integer(key2, val2, 12, 'sum')))
FROM t_int_int_1_5_through_7_11;
+-- Test tuple_union_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double function with LongType key sketches and explicit lgNomEntries parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1, 15),
+ theta_sketch_agg(key2), 15))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_union_theta_double function with StringType key sketches and explicit lgNomEntries parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2), 14))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_union_theta_double with lgNomEntries and mode parameters
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 12, 'sum'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_union_theta_integer(
+ tuple_sketch_agg_integer(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer with lgNomEntries and mode parameters
+SELECT tuple_sketch_estimate_integer(
+ tuple_union_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 12, 'sum'))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with LongType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1, 5),
+ theta_sketch_agg(key2)))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_intersection_theta_double function with StringType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_intersection_theta_double with mode parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1, 12, 'min'),
+ theta_sketch_agg(key2), 'min'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_intersection_theta_integer(
+ tuple_sketch_agg_integer(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_integer with mode parameter
+SELECT tuple_sketch_estimate_integer(
+ tuple_intersection_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 'sum'))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with LongType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_difference_theta_double function with StringType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_difference_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_difference_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
Review Comment:
Can we also test other supported aggregation modes: "max", "min", "alwaysone"?
##########
docs/sql-ref-sketch-aggregates.md:
##########
@@ -748,6 +751,138 @@ FROM VALUES (1, 10.0, 4, 40.0), (2, 20.0, 4, 40.0), (3, 30.0, 5, 50.0), (4, 40.0
---
+### tuple_union_theta_*
+
+Merges a Tuple sketch with a Theta sketch using union (scalar function). This combines distinct keys from both sketches, with Theta sketch entries assigned a default summary value.
Review Comment:
The default values are identity elements for each mode (0.0 for sum, +Inf for min, -Inf for max, 1.0 for alwaysone). This is correct but non-obvious. Can these docs and the `@ExpressionDescription` explicitly state what default value Theta entries receive for each mode?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala:
##########
@@ -97,6 +97,78 @@ case class TupleDifferenceInteger(left: Expression, right: Expression)
}
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
+ Datasketches ThetaSketch from a TupleSketch with double summary data type using a TupleSketch
+ AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch. """,
+ examples = """
+ Examples:
+ > SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (5, 5.0D, 4), (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 1) tab(col1, val1, col2);
+ 2.0
+ """,
+ group = "sketch_funcs",
+ since = "4.2.0")
+// scalastyle:on line.size.limit
+case class TupleDifferenceThetaDouble(left: Expression, right: Expression)
Review Comment:
If a user passes these two binaries in the wrong order (theta first, tuple second), they'll get a confusing deserialization error rather than a clear type mismatch. This is an inherent limitation of the binary type approach but could be mitigated with a better error message. Is there any way we could exercise this in testing and possibly catch a specific Datasketches exception and convert that to a Spark error class that helps the user understand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #54338:
URL: https://github.com/apache/spark/pull/54338#discussion_r2828312682
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -3904,6 +3904,213 @@ class DataFrameAggregateSuite extends QueryTest
assert(estimate == 1.0)
}
+ test("SPARK-55558: tuple_difference_theta_double basic functionality") {
+ val df1 = Seq((1, 1.5), (2, 2.5), (3, 3.5), (5, 5.5)).toDF("key", "summary")
+ val df2 = Seq(1, 2, 4).toDF("value")
+
+ val tupleSketchDf = df1.agg(tuple_sketch_agg_double($"key", $"summary").alias("tuple_sketch"))
+ val thetaSketchDf = df2.agg(theta_sketch_agg($"value").alias("theta_sketch"))
+
+ val joined = tupleSketchDf.crossJoin(thetaSketchDf)
+
+ // Test difference (keys in tuple_sketch but not in theta_sketch: 3 and 5)
+ val difference = joined
+ .select(tuple_difference_theta_double($"tuple_sketch", $"theta_sketch"))
+ .collect()(0)(0)
+ assert(difference != null)
+ assert(difference.asInstanceOf[Array[Byte]].length > 0)
+
+ // Test with column names
+ val difference2 = joined
+ .select(tuple_difference_theta_double("tuple_sketch", "theta_sketch"))
+ .collect()(0)(0)
+ assert(difference2 != null)
+
+ // Verify estimate from difference
+ val estimate = joined
+ .select(tuple_sketch_estimate_double(
+ tuple_difference_theta_double($"tuple_sketch", $"theta_sketch")))
+ .collect()(0)(0)
+ assert(estimate == 2.0)
Review Comment:
added
##########
sql/core/src/test/resources/sql-tests/inputs/tuplesketch.sql:
##########
@@ -371,6 +371,139 @@ SELECT tuple_sketch_estimate_integer(
tuple_sketch_agg_integer(key2, val2, 12, 'sum')))
FROM t_int_int_1_5_through_7_11;
+-- Test tuple_union_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double function with LongType key sketches and explicit lgNomEntries parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1, 15),
+ theta_sketch_agg(key2), 15))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_union_theta_double function with StringType key sketches and explicit lgNomEntries parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2), 14))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_union_theta_double with lgNomEntries and mode parameters
+SELECT tuple_sketch_estimate_double(
+ tuple_union_theta_double(
+ tuple_sketch_agg_double(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 12, 'sum'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_union_theta_integer(
+ tuple_sketch_agg_integer(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_union_theta_integer with lgNomEntries and mode parameters
+SELECT tuple_sketch_estimate_integer(
+ tuple_union_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 12, 'sum'))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with LongType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1, 5),
+ theta_sketch_agg(key2)))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_intersection_theta_double function with StringType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_intersection_theta_double with mode parameter
+SELECT tuple_sketch_estimate_double(
+ tuple_intersection_theta_double(
+ tuple_sketch_agg_double(key1, val1, 12, 'min'),
+ theta_sketch_agg(key2), 'min'))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_intersection_theta_integer(
+ tuple_sketch_agg_integer(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_intersection_theta_integer with mode parameter
+SELECT tuple_sketch_estimate_integer(
+ tuple_intersection_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
+ theta_sketch_agg(key2), 'sum'))
+FROM t_int_int_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with IntegerType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_int_double_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with LongType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_long_double_1_5_through_7_11;
+
+-- Test tuple_difference_theta_double function with DoubleType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_double_double_1_1_1_4_through_1_5_1_8;
+
+-- Test tuple_difference_theta_double function with StringType key sketches
+SELECT tuple_sketch_estimate_double(
+ tuple_difference_theta_double(
+ tuple_sketch_agg_double(key1, val1),
+ theta_sketch_agg(key2)))
+FROM t_string_double_a_d_through_e_h;
+
+-- Test tuple_difference_theta_integer function with IntegerType key sketches
+SELECT tuple_sketch_estimate_integer(
+ tuple_difference_theta_integer(
+ tuple_sketch_agg_integer(key1, val1, 12, 'sum'),
Review Comment:
yep added these!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55558][SQL] Add support for Tuple/Theta set operations [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #54338: [SPARK-55558][SQL] Add support for Tuple/Theta set operations
URL: https://github.com/apache/spark/pull/54338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org