You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hv...@apache.org on 2024/08/23 18:09:44 UTC
(spark) branch master updated: [SPARK-49027][CONNECT][SQL] Share Column API between Class and Connect
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e95618f4e7df [SPARK-49027][CONNECT][SQL] Share Column API between Class and Connect
e95618f4e7df is described below
commit e95618f4e7df3ba32e704d413ecc33d4ba316b4e
Author: Herman van Hovell <he...@databricks.com>
AuthorDate: Fri Aug 23 14:09:30 2024 -0400
[SPARK-49027][CONNECT][SQL] Share Column API between Class and Connect
### What changes were proposed in this pull request?
This PR moves the sql Column to sql/api. A (foreseen) consequence of this is that we need to move the Spark Connect Scala Client to the shared API. This makes the PR quite large.
### Why are the changes needed?
We want to create a Scala Client interface that is shared between Classic and Connect.
### Does this PR introduce _any_ user-facing change?
Connect does no support untyped Scala UDFs until we figure out a solution for the conf system.
### How was this patch tested?
Mostly existing tests. I had to change a couple of tests on the connect side:
- UDAF related tests used a feature that was not supposed to work in the first place. You could use a typed aggregator on an untyped Dataset (Row). I have added a test for the scenario where you use an untyped aggregator on an untyped dataset.
- In some cases the shared Column API generates slightly different structures (more/less arguments, different names, or different structure). The verbatims used by `PlanGenerationTestSuite` and `ProtoToParsedPlanTestSuite` were updated to reflect this. The protos that had non trivial changes were renamed with the `_orphaned` suffix, this to make sure we keep testing plans produced by older clients.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47839 from hvanhovell/SPARK-49027.
Authored-by: Herman van Hovell <he...@databricks.com>
Signed-off-by: Herman van Hovell <he...@databricks.com>
---
.../main/scala/org/apache/spark/sql/Column.scala | 1514 ----
.../apache/spark/sql/DataFrameNaFunctions.scala | 1 +
.../apache/spark/sql/DataFrameStatFunctions.scala | 1 +
.../org/apache/spark/sql/DataFrameWriterV2.scala | 1 +
.../main/scala/org/apache/spark/sql/Dataset.scala | 57 +-
.../apache/spark/sql/KeyValueGroupedDataset.scala | 32 +-
.../org/apache/spark/sql/MergeIntoWriter.scala | 80 +-
.../spark/sql/RelationalGroupedDataset.scala | 8 +-
.../scala/org/apache/spark/sql/SparkSession.scala | 6 +
.../org/apache/spark/sql/UDFRegistration.scala | 1272 +--
.../org/apache/spark/sql/avro/functions.scala | 4 +-
.../apache/spark/sql/expressions/Aggregator.scala | 166 -
.../sql/expressions/UserDefinedFunction.scala | 220 -
.../org/apache/spark/sql/expressions/Window.scala | 241 -
.../apache/spark/sql/expressions/WindowSpec.scala | 240 -
.../scala/org/apache/spark/sql/functions.scala | 8754 --------------------
.../spark/sql/internal/UdfToProtoUtils.scala | 107 +
.../{connect => internal}/columnNodeSupport.scala | 88 +-
.../main/scala/org/apache/spark/sql/package.scala | 27 +
.../org/apache/spark/sql/protobuf/functions.scala | 10 +-
.../org/apache/spark/sql/ClientDatasetSuite.scala | 2 +-
.../org/apache/spark/sql/ColumnTestSuite.scala | 10 +-
.../org/apache/spark/sql/FunctionTestSuite.scala | 45 +-
.../apache/spark/sql/PlanGenerationTestSuite.scala | 19 +-
.../apache/spark/sql/UDFClassLoadingE2ESuite.scala | 24 +-
.../sql/UserDefinedFunctionE2ETestSuite.scala | 43 +-
.../spark/sql/UserDefinedFunctionSuite.scala | 10 +-
.../CheckConnectJvmClientCompatibility.scala | 84 +-
.../ColumnNodeToProtoConverterSuite.scala | 69 +-
.../apache/spark/sql/test/ConnectFunSuite.scala | 6 +
docs/_plugins/build_api_docs.rb | 4 +-
project/MimaExcludes.scala | 16 +-
project/SparkBuild.scala | 6 +-
.../main/scala/org/apache/spark/sql/Column.scala | 32 +-
.../scala/org/apache/spark/sql/api/Dataset.scala | 14 +-
.../apache/spark/sql/api}/UDFRegistration.scala | 179 +-
.../apache/spark/sql/catalyst/trees/origin.scala | 46 +
.../spark/sql/errors/CompilationErrors.scala | 28 +
.../apache/spark/sql/expressions/Aggregator.scala | 0
.../sql/expressions/UserDefinedFunction.scala | 13 +
.../org/apache/spark/sql/expressions/Window.scala | 0
.../apache/spark/sql/expressions/WindowSpec.scala | 6 +-
.../org/apache/spark/sql/expressions/udaf.scala | 0
.../scala/org/apache/spark/sql/functions.scala | 68 +-
.../org/apache/spark/sql/internal/SqlApiConf.scala | 4 +
.../org/apache/spark/sql/internal/ToScalaUDF.scala | 3 +-
.../apache/spark/sql/internal/columnNodes.scala | 11 +
.../spark/sql/errors/QueryCompilationErrors.scala | 28 -
.../org/apache/spark/sql/internal/SQLConf.scala | 6 +-
.../apache/spark/sql/connect/common/UdfUtils.scala | 5 +
.../explain-results/function_aggregate.explain | 2 +-
.../function_aggregate_with_finish_lambda.explain | 2 +
.../function_array_sort_with_comparator.explain | 2 +-
.../function_call_function_orphaned.explain | 2 +
.../explain-results/function_exists.explain | 2 +-
.../explain-results/function_filter.explain | 2 +-
.../function_filter_with_pair_input.explain | 2 +-
.../explain-results/function_forall.explain | 2 +-
.../function_from_json_orphaned.explain | 2 +
.../function_from_json_with_json_schema.explain | 2 +
.../explain-results/function_from_xml.explain | 2 +
.../function_from_xml_with_json_schema.explain | 2 +
.../explain-results/function_map_filter.explain | 2 +-
.../function_map_from_entries.explain | 2 +-
.../explain-results/function_map_zip_with.explain | 2 +-
.../explain-results/function_mode_orphaned.explain | 2 +
.../explain-results/function_reduce.explain | 2 +-
.../function_session_window.explain | 2 +-
...in => function_session_window_orphaned.explain} | 0
.../explain-results/function_transform.explain | 2 +-
.../function_transform_keys.explain | 2 +-
.../function_transform_values.explain | 2 +-
.../function_transform_with_index.explain | 2 +-
.../explain-results/function_zip_with.explain | 2 +-
.../explain-results/groupby_agg_orphaned.explain | 2 +
.../query-tests/queries/function_aggregate.json | 12 +-
.../queries/function_aggregate.proto.bin | Bin 235 -> 247 bytes
... => function_aggregate_with_finish_lambda.json} | 23 +-
...unction_aggregate_with_finish_lambda.proto.bin} | Bin 235 -> 260 bytes
.../function_array_sort_with_comparator.json | 8 +-
.../function_array_sort_with_comparator.proto.bin | Bin 214 -> 222 bytes
.../queries/function_call_function.json | 5 +-
.../queries/function_call_function.proto.bin | Bin 175 -> 176 bytes
...n.json => function_call_function_orphaned.json} | 0
...n => function_call_function_orphaned.proto.bin} | Bin
.../query-tests/queries/function_exists.json | 4 +-
.../query-tests/queries/function_exists.proto.bin | Bin 204 -> 208 bytes
.../query-tests/queries/function_filter.json | 4 +-
.../query-tests/queries/function_filter.proto.bin | Bin 204 -> 208 bytes
.../queries/function_filter_with_pair_input.json | 8 +-
.../function_filter_with_pair_input.proto.bin | Bin 238 -> 246 bytes
.../query-tests/queries/function_forall.json | 4 +-
.../query-tests/queries/function_forall.proto.bin | Bin 204 -> 208 bytes
.../query-tests/queries/function_from_json.json | 2 +-
.../queries/function_from_json.proto.bin | Bin 394 -> 221 bytes
..._json.json => function_from_json_orphaned.json} | 0
...o.bin => function_from_json_orphaned.proto.bin} | Bin
...on => function_from_json_with_json_schema.json} | 0
... function_from_json_with_json_schema.proto.bin} | Bin
.../{function_mode.json => function_from_xml.json} | 6 +-
...ilter.proto.bin => function_from_xml.proto.bin} | Bin 204 -> 220 bytes
...son => function_from_xml_with_json_schema.json} | 2 +-
...> function_from_xml_with_json_schema.proto.bin} | Bin 394 -> 393 bytes
.../query-tests/queries/function_map_filter.json | 6 +-
.../queries/function_map_filter.proto.bin | Bin 223 -> 229 bytes
.../queries/function_map_from_entries.json | 8 +-
.../queries/function_map_from_entries.proto.bin | Bin 240 -> 248 bytes
.../query-tests/queries/function_map_zip_with.json | 10 +-
.../queries/function_map_zip_with.proto.bin | Bin 252 -> 262 bytes
.../query-tests/queries/function_mode.json | 4 -
.../query-tests/queries/function_mode.proto.bin | Bin 179 -> 173 bytes
...ction_mode.json => function_mode_orphaned.json} | 0
....proto.bin => function_mode_orphaned.proto.bin} | Bin
.../query-tests/queries/function_reduce.json | 12 +-
.../query-tests/queries/function_reduce.proto.bin | Bin 232 -> 244 bytes
.../queries/function_session_window.json | 25 +-
.../queries/function_session_window.proto.bin | Bin 165 -> 145 bytes
....json => function_session_window_orphaned.json} | 0
... => function_session_window_orphaned.proto.bin} | Bin
.../query-tests/queries/function_transform.json | 4 +-
.../queries/function_transform.proto.bin | Bin 207 -> 211 bytes
.../queries/function_transform_keys.json | 8 +-
.../queries/function_transform_keys.proto.bin | Bin 235 -> 243 bytes
.../queries/function_transform_values.json | 8 +-
.../queries/function_transform_values.proto.bin | Bin 222 -> 230 bytes
.../queries/function_transform_with_index.json | 8 +-
.../function_transform_with_index.proto.bin | Bin 213 -> 221 bytes
.../query-tests/queries/function_zip_with.json | 8 +-
.../queries/function_zip_with.proto.bin | Bin 219 -> 227 bytes
.../resources/query-tests/queries/groupby_agg.json | 4 +-
.../query-tests/queries/groupby_agg.proto.bin | Bin 210 -> 210 bytes
...{groupby_agg.json => groupby_agg_orphaned.json} | 0
...gg.proto.bin => groupby_agg_orphaned.proto.bin} | Bin
.../sql/connect/planner/SparkConnectPlanner.scala | 3 +-
.../apache/spark/sql/DataFrameStatFunctions.scala | 1 +
.../main/scala/org/apache/spark/sql/Dataset.scala | 3 +-
.../org/apache/spark/sql/UDFRegistration.scala | 535 +-
.../main/scala/org/apache/spark/sql/package.scala | 47 -
138 files changed, 855 insertions(+), 13575 deletions(-)
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala
deleted file mode 100644
index 356267589822..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala
+++ /dev/null
@@ -1,1514 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.spark.sql
-
-import scala.jdk.CollectionConverters._
-
-import org.apache.spark.annotation.{DeveloperApi, Since}
-import org.apache.spark.connect.proto
-import org.apache.spark.connect.proto.Expression.SortOrder.NullOrdering
-import org.apache.spark.connect.proto.Expression.SortOrder.SortDirection
-import org.apache.spark.internal.Logging
-import org.apache.spark.sql.catalyst.encoders.AgnosticEncoder
-import org.apache.spark.sql.catalyst.parser.DataTypeParser
-import org.apache.spark.sql.connect.common.DataTypeProtoConverter
-import org.apache.spark.sql.expressions.Window
-import org.apache.spark.sql.functions.lit
-import org.apache.spark.sql.types._
-import org.apache.spark.util.ArrayImplicits._
-
-/**
- * A column that will be computed based on the data in a `DataFrame`.
- *
- * A new column can be constructed based on the input columns present in a DataFrame:
- *
- * {{{
- * df("columnName") // On a specific `df` DataFrame.
- * col("columnName") // A generic column not yet associated with a DataFrame.
- * col("columnName.field") // Extracting a struct field
- * col("`a.column.with.dots`") // Escape `.` in column names.
- * $"columnName" // Scala short hand for a named column.
- * }}}
- *
- * [[Column]] objects can be composed to form complex expressions:
- *
- * {{{
- * $"a" + 1
- * }}}
- *
- * @since 3.4.0
- */
-class Column(@DeveloperApi val expr: proto.Expression) extends Logging {
-
- private[sql] def this(name: String, planId: Option[Long]) =
- this(Column.nameToExpression(name, planId))
-
- private[sql] def this(name: String) =
- this(name, None)
-
- private def fn(name: String): Column = Column.fn(name, this)
- private def fn(name: String, other: Column): Column = Column.fn(name, this, other)
- private def fn(name: String, other: Any): Column = Column.fn(name, this, lit(other))
-
- override def toString: String = expr.toString
-
- override def equals(that: Any): Boolean = that match {
- case that: Column => expr == that.expr
- case _ => false
- }
-
- override def hashCode: Int = expr.hashCode()
-
- /**
- * Provides a type hint about the expected return value of this column. This information can be
- * used by operations such as `select` on a [[Dataset]] to automatically convert the results
- * into the correct JVM types.
- * @since 3.4.0
- */
- def as[U: Encoder]: TypedColumn[Any, U] = {
- val encoder = implicitly[Encoder[U]].asInstanceOf[AgnosticEncoder[U]]
- new TypedColumn[Any, U](expr, encoder)
- }
-
- /**
- * Extracts a value or values from a complex type. The following types of extraction are
- * supported:
- * - Given an Array, an integer ordinal can be used to retrieve a single value.
- * - Given a Map, a key of the correct type can be used to retrieve an individual value.
- * - Given a Struct, a string fieldName can be used to extract that field.
- * - Given an Array of Structs, a string fieldName can be used to extract filed of every
- * struct in that array, and return an Array of fields.
- * @group expr_ops
- * @since 3.4.0
- */
- def apply(extraction: Any): Column = Column { builder =>
- builder.getUnresolvedExtractValueBuilder
- .setChild(expr)
- .setExtraction(lit(extraction).expr)
- }
-
- /**
- * Unary minus, i.e. negate the expression.
- * {{{
- * // Scala: select the amount column and negates all values.
- * df.select( -df("amount") )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * df.select( negate(col("amount") );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def unary_- : Column = fn("negative")
-
- /**
- * Inversion of boolean expression, i.e. NOT.
- * {{{
- * // Scala: select rows that are not active (isActive === false)
- * df.filter( !df("isActive") )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * df.filter( not(df.col("isActive")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def unary_! : Column = fn("!")
-
- /**
- * Equality test.
- * {{{
- * // Scala:
- * df.filter( df("colA") === df("colB") )
- *
- * // Java
- * import static org.apache.spark.sql.functions.*;
- * df.filter( col("colA").equalTo(col("colB")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def ===(other: Any): Column = fn("=", other)
-
- /**
- * Equality test.
- * {{{
- * // Scala:
- * df.filter( df("colA") === df("colB") )
- *
- * // Java
- * import static org.apache.spark.sql.functions.*;
- * df.filter( col("colA").equalTo(col("colB")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def equalTo(other: Any): Column = this === other
-
- /**
- * Inequality test.
- * {{{
- * // Scala:
- * df.select( df("colA") =!= df("colB") )
- * df.select( !(df("colA") === df("colB")) )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * df.filter( col("colA").notEqual(col("colB")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def =!=(other: Any): Column = !(this === other)
-
- /**
- * Inequality test.
- * {{{
- * // Scala:
- * df.select( df("colA") !== df("colB") )
- * df.select( !(df("colA") === df("colB")) )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * df.filter( col("colA").notEqual(col("colB")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- @deprecated("!== does not have the same precedence as ===, use =!= instead", "2.0.0")
- def !==(other: Any): Column = this =!= other
-
- /**
- * Inequality test.
- * {{{
- * // Scala:
- * df.select( df("colA") !== df("colB") )
- * df.select( !(df("colA") === df("colB")) )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * df.filter( col("colA").notEqual(col("colB")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def notEqual(other: Any): Column = this =!= other
-
- /**
- * Greater than.
- * {{{
- * // Scala: The following selects people older than 21.
- * people.select( people("age") > 21 )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * people.select( people.col("age").gt(21) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def >(other: Any): Column = fn(">", other)
-
- /**
- * Greater than.
- * {{{
- * // Scala: The following selects people older than 21.
- * people.select( people("age") > lit(21) )
- *
- * // Java:
- * import static org.apache.spark.sql.functions.*;
- * people.select( people.col("age").gt(21) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def gt(other: Any): Column = this > other
-
- /**
- * Less than.
- * {{{
- * // Scala: The following selects people younger than 21.
- * people.select( people("age") < 21 )
- *
- * // Java:
- * people.select( people.col("age").lt(21) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def <(other: Any): Column = fn("<", other)
-
- /**
- * Less than.
- * {{{
- * // Scala: The following selects people younger than 21.
- * people.select( people("age") < 21 )
- *
- * // Java:
- * people.select( people.col("age").lt(21) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def lt(other: Any): Column = this < other
-
- /**
- * Less than or equal to.
- * {{{
- * // Scala: The following selects people age 21 or younger than 21.
- * people.select( people("age") <= 21 )
- *
- * // Java:
- * people.select( people.col("age").leq(21) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def <=(other: Any): Column = fn("<=", other)
-
- /**
- * Less than or equal to.
- * {{{
- * // Scala: The following selects people age 21 or younger than 21.
- * people.select( people("age") <= 21 )
- *
- * // Java:
- * people.select( people.col("age").leq(21) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def leq(other: Any): Column = this <= other
-
- /**
- * Greater than or equal to an expression.
- * {{{
- * // Scala: The following selects people age 21 or older than 21.
- * people.select( people("age") >= 21 )
- *
- * // Java:
- * people.select( people.col("age").geq(21) )
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def >=(other: Any): Column = fn(">=", other)
-
- /**
- * Greater than or equal to an expression.
- * {{{
- * // Scala: The following selects people age 21 or older than 21.
- * people.select( people("age") >= 21 )
- *
- * // Java:
- * people.select( people.col("age").geq(21) )
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def geq(other: Any): Column = this >= other
-
- /**
- * Equality test that is safe for null values.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def <=>(other: Any): Column = fn("<=>", other)
-
- /**
- * Equality test that is safe for null values.
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def eqNullSafe(other: Any): Column = this <=> other
-
- private def extractWhen(name: String): java.util.List[proto.Expression] = {
- def fail(): Nothing = {
- throw new IllegalArgumentException(
- s"$name() can only be applied on a Column previously generated by when() function")
- }
- if (!expr.hasUnresolvedFunction) {
- fail()
- }
- val parentFn = expr.getUnresolvedFunction
- if (parentFn.getFunctionName != "when") {
- fail()
- }
- parentFn.getArgumentsList
- }
-
- /**
- * Evaluates a list of conditions and returns one of multiple possible result expressions. If
- * otherwise is not defined at the end, null is returned for unmatched conditions.
- *
- * {{{
- * // Example: encoding gender string column into integer.
- *
- * // Scala:
- * people.select(when(people("gender") === "male", 0)
- * .when(people("gender") === "female", 1)
- * .otherwise(2))
- *
- * // Java:
- * people.select(when(col("gender").equalTo("male"), 0)
- * .when(col("gender").equalTo("female"), 1)
- * .otherwise(2))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def when(condition: Column, value: Any): Column = {
- val expressions = extractWhen("when")
- if (expressions.size() % 2 == 1) {
- throw new IllegalArgumentException("when() cannot be applied once otherwise() is applied")
- }
- Column { builder =>
- builder.getUnresolvedFunctionBuilder
- .setFunctionName("when")
- .addAllArguments(expressions)
- .addArguments(condition.expr)
- .addArguments(lit(value).expr)
- }
- }
-
- /**
- * Evaluates a list of conditions and returns one of multiple possible result expressions. If
- * otherwise is not defined at the end, null is returned for unmatched conditions.
- *
- * {{{
- * // Example: encoding gender string column into integer.
- *
- * // Scala:
- * people.select(when(people("gender") === "male", 0)
- * .when(people("gender") === "female", 1)
- * .otherwise(2))
- *
- * // Java:
- * people.select(when(col("gender").equalTo("male"), 0)
- * .when(col("gender").equalTo("female"), 1)
- * .otherwise(2))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def otherwise(value: Any): Column = {
- val expressions = extractWhen("otherwise")
- if (expressions.size() % 2 == 1) {
- throw new IllegalArgumentException(
- "otherwise() can only be applied once on a Column previously generated by when()")
- }
- Column { builder =>
- builder.getUnresolvedFunctionBuilder
- .setFunctionName("when")
- .addAllArguments(expressions)
- .addArguments(lit(value).expr)
- }
- }
-
- /**
- * True if the current column is between the lower bound and upper bound, inclusive.
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def between(lowerBound: Any, upperBound: Any): Column = {
- (this >= lowerBound) && (this <= upperBound)
- }
-
- /**
- * True if the current expression is NaN.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def isNaN: Column = fn("isNaN")
-
- /**
- * True if the current expression is null.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def isNull: Column = fn("isNull")
-
- /**
- * True if the current expression is NOT null.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def isNotNull: Column = fn("isNotNull")
-
- /**
- * Boolean OR.
- * {{{
- * // Scala: The following selects people that are in school or employed.
- * people.filter( people("inSchool") || people("isEmployed") )
- *
- * // Java:
- * people.filter( people.col("inSchool").or(people.col("isEmployed")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def ||(other: Any): Column = fn("or", other)
-
- /**
- * Boolean OR.
- * {{{
- * // Scala: The following selects people that are in school or employed.
- * people.filter( people("inSchool") || people("isEmployed") )
- *
- * // Java:
- * people.filter( people.col("inSchool").or(people.col("isEmployed")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def or(other: Column): Column = this || other
-
- /**
- * Boolean AND.
- * {{{
- * // Scala: The following selects people that are in school and employed at the same time.
- * people.select( people("inSchool") && people("isEmployed") )
- *
- * // Java:
- * people.select( people.col("inSchool").and(people.col("isEmployed")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def &&(other: Any): Column = fn("and", other)
-
- /**
- * Boolean AND.
- * {{{
- * // Scala: The following selects people that are in school and employed at the same time.
- * people.select( people("inSchool") && people("isEmployed") )
- *
- * // Java:
- * people.select( people.col("inSchool").and(people.col("isEmployed")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def and(other: Column): Column = this && other
-
- /**
- * Sum of this expression and another expression.
- * {{{
- * // Scala: The following selects the sum of a person's height and weight.
- * people.select( people("height") + people("weight") )
- *
- * // Java:
- * people.select( people.col("height").plus(people.col("weight")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def +(other: Any): Column = fn("+", other)
-
- /**
- * Sum of this expression and another expression.
- * {{{
- * // Scala: The following selects the sum of a person's height and weight.
- * people.select( people("height") + people("weight") )
- *
- * // Java:
- * people.select( people.col("height").plus(people.col("weight")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def plus(other: Any): Column = this + other
-
- /**
- * Subtraction. Subtract the other expression from this expression.
- * {{{
- * // Scala: The following selects the difference between people's height and their weight.
- * people.select( people("height") - people("weight") )
- *
- * // Java:
- * people.select( people.col("height").minus(people.col("weight")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def -(other: Any): Column = fn("-", other)
-
- /**
- * Subtraction. Subtract the other expression from this expression.
- * {{{
- * // Scala: The following selects the difference between people's height and their weight.
- * people.select( people("height") - people("weight") )
- *
- * // Java:
- * people.select( people.col("height").minus(people.col("weight")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def minus(other: Any): Column = this - other
-
- /**
- * Multiplication of this expression and another expression.
- * {{{
- * // Scala: The following multiplies a person's height by their weight.
- * people.select( people("height") * people("weight") )
- *
- * // Java:
- * people.select( people.col("height").multiply(people.col("weight")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def *(other: Any): Column = fn("*", other)
-
- /**
- * Multiplication of this expression and another expression.
- * {{{
- * // Scala: The following multiplies a person's height by their weight.
- * people.select( people("height") * people("weight") )
- *
- * // Java:
- * people.select( people.col("height").multiply(people.col("weight")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def multiply(other: Any): Column = this * other
-
- /**
- * Division this expression by another expression.
- * {{{
- * // Scala: The following divides a person's height by their weight.
- * people.select( people("height") / people("weight") )
- *
- * // Java:
- * people.select( people.col("height").divide(people.col("weight")) );
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def /(other: Any): Column = fn("/", other)
-
- /**
- * Division this expression by another expression.
- * {{{
- * // Scala: The following divides a person's height by their weight.
- * people.select( people("height") / people("weight") )
- *
- * // Java:
- * people.select( people.col("height").divide(people.col("weight")) );
- * }}}
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def divide(other: Any): Column = this / other
-
- /**
- * Modulo (a.k.a. remainder) expression.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def %(other: Any): Column = fn("%", other)
-
- /**
- * Modulo (a.k.a. remainder) expression.
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def mod(other: Any): Column = this % other
-
- /**
- * A boolean expression that is evaluated to true if the value of this expression is contained
- * by the evaluated values of the arguments.
- *
- * Note: Since the type of the elements in the list are inferred only during the run time, the
- * elements will be "up-casted" to the most common type for comparison. For eg: 1) In the case
- * of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look like
- * "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted to
- * "Double" and the comparison will look like "Double vs Double"
- *
- * @group expr_ops
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def isin(list: Any*): Column = Column.fn("in", this +: list.map(lit): _*)
-
- /**
- * A boolean expression that is evaluated to true if the value of this expression is contained
- * by the provided collection.
- *
- * Note: Since the type of the elements in the collection are inferred only during the run time,
- * the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the
- * case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look
- * like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted
- * to "Double" and the comparison will look like "Double vs Double"
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def isInCollection(values: scala.collection.Iterable[_]): Column = isin(values.toSeq: _*)
-
- /**
- * A boolean expression that is evaluated to true if the value of this expression is contained
- * by the provided collection.
- *
- * Note: Since the type of the elements in the collection are inferred only during the run time,
- * the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the
- * case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look
- * like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted
- * to "Double" and the comparison will look like "Double vs Double"
- *
- * @group java_expr_ops
- * @since 3.4.0
- */
- def isInCollection(values: java.lang.Iterable[_]): Column = isInCollection(values.asScala)
-
- /**
- * SQL like expression. Returns a boolean column based on a SQL LIKE match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def like(literal: String): Column = fn("like", literal)
-
- /**
- * SQL RLIKE expression (LIKE with Regex). Returns a boolean column based on a regex match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def rlike(literal: String): Column = fn("rlike", literal)
-
- /**
- * SQL ILIKE expression (case insensitive LIKE).
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def ilike(literal: String): Column = fn("ilike", literal)
-
- /**
- * An expression that gets an item at position `ordinal` out of an array, or gets a value by key
- * `key` in a `MapType`.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def getItem(key: Any): Column = apply(key)
-
- // scalastyle:off line.size.limit
- /**
- * An expression that adds/replaces field in `StructType` by name.
- *
- * {{{
- * val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
- * df.select($"struct_col".withField("c", lit(3)))
- * // result: {"a":1,"b":2,"c":3}
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
- * df.select($"struct_col".withField("b", lit(3)))
- * // result: {"a":1,"b":3}
- *
- * val df = sql("SELECT CAST(NULL AS struct<a:int,b:int>) struct_col")
- * df.select($"struct_col".withField("c", lit(3)))
- * // result: null of type struct<a:int,b:int,c:int>
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2, 'b', 3) struct_col")
- * df.select($"struct_col".withField("b", lit(100)))
- * // result: {"a":1,"b":100,"b":100}
- *
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".withField("a.c", lit(3)))
- * // result: {"a":{"a":1,"b":2,"c":3}}
- *
- * val df = sql("SELECT named_struct('a', named_struct('b', 1), 'a', named_struct('c', 2)) struct_col")
- * df.select($"struct_col".withField("a.c", lit(3)))
- * // result: org.apache.spark.sql.AnalysisException: Ambiguous reference to fields
- * }}}
- *
- * This method supports adding/replacing nested fields directly e.g.
- *
- * {{{
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".withField("a.c", lit(3)).withField("a.d", lit(4)))
- * // result: {"a":{"a":1,"b":2,"c":3,"d":4}}
- * }}}
- *
- * However, if you are going to add/replace multiple nested fields, it is more optimal to
- * extract out the nested struct before adding/replacing multiple fields e.g.
- *
- * {{{
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".withField("a", $"struct_col.a".withField("c", lit(3)).withField("d", lit(4))))
- * // result: {"a":{"a":1,"b":2,"c":3,"d":4}}
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- // scalastyle:on line.size.limit
- def withField(fieldName: String, col: Column): Column = {
- require(fieldName != null, "fieldName cannot be null")
- require(col != null, "col cannot be null")
- Column { builder =>
- builder.getUpdateFieldsBuilder
- .setStructExpression(expr)
- .setFieldName(fieldName)
- .setValueExpression(col.expr)
- }
- }
-
- // scalastyle:off line.size.limit
- /**
- * An expression that drops fields in `StructType` by name. This is a no-op if schema doesn't
- * contain field name(s).
- *
- * {{{
- * val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
- * df.select($"struct_col".dropFields("b"))
- * // result: {"a":1}
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
- * df.select($"struct_col".dropFields("c"))
- * // result: {"a":1,"b":2}
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2, 'c', 3) struct_col")
- * df.select($"struct_col".dropFields("b", "c"))
- * // result: {"a":1}
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
- * df.select($"struct_col".dropFields("a", "b"))
- * // result: org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.CANNOT_DROP_ALL_FIELDS] Cannot resolve "update_fields(struct_col, dropfield(), dropfield())" due to data type mismatch: Cannot drop all fields in struct.;
- *
- * val df = sql("SELECT CAST(NULL AS struct<a:int,b:int>) struct_col")
- * df.select($"struct_col".dropFields("b"))
- * // result: null of type struct<a:int>
- *
- * val df = sql("SELECT named_struct('a', 1, 'b', 2, 'b', 3) struct_col")
- * df.select($"struct_col".dropFields("b"))
- * // result: {"a":1}
- *
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".dropFields("a.b"))
- * // result: {"a":{"a":1}}
- *
- * val df = sql("SELECT named_struct('a', named_struct('b', 1), 'a', named_struct('c', 2)) struct_col")
- * df.select($"struct_col".dropFields("a.c"))
- * // result: org.apache.spark.sql.AnalysisException: Ambiguous reference to fields
- * }}}
- *
- * This method supports dropping multiple nested fields directly e.g.
- *
- * {{{
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".dropFields("a.b", "a.c"))
- * // result: {"a":{"a":1}}
- * }}}
- *
- * However, if you are going to drop multiple nested fields, it is more optimal to extract out
- * the nested struct before dropping multiple fields from it e.g.
- *
- * {{{
- * val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
- * df.select($"struct_col".withField("a", $"struct_col.a".dropFields("b", "c")))
- * // result: {"a":{"a":1}}
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- // scalastyle:on line.size.limit
- def dropFields(fieldNames: String*): Column = {
- fieldNames.foldLeft(this) { case (column, fieldName) =>
- Column { builder =>
- builder.getUpdateFieldsBuilder
- .setStructExpression(column.expr)
- .setFieldName(fieldName)
- }
- }
- }
-
- /**
- * An expression that gets a field by name in a `StructType`.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def getField(fieldName: String): Column = apply(fieldName)
-
- /**
- * An expression that returns a substring.
- * @param startPos
- * expression for the starting position.
- * @param len
- * expression for the length of the substring.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def substr(startPos: Column, len: Column): Column = Column.fn("substr", this, startPos, len)
-
- /**
- * An expression that returns a substring.
- * @param startPos
- * starting position.
- * @param len
- * length of the substring.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def substr(startPos: Int, len: Int): Column = substr(lit(startPos), lit(len))
-
- /**
- * Contains the other element. Returns a boolean column based on a string match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def contains(other: Any): Column = fn("contains", other)
-
- /**
- * String starts with. Returns a boolean column based on a string match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def startsWith(other: Column): Column = fn("startswith", other)
-
- /**
- * String starts with another string literal. Returns a boolean column based on a string match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def startsWith(literal: String): Column = startsWith(lit(literal))
-
- /**
- * String ends with. Returns a boolean column based on a string match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def endsWith(other: Column): Column = fn("endswith", other)
-
- /**
- * String ends with another string literal. Returns a boolean column based on a string match.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def endsWith(literal: String): Column = endsWith(lit(literal))
-
- /**
- * Gives the column an alias. Same as `as`.
- * {{{
- * // Renames colA to colB in select output.
- * df.select($"colA".alias("colB"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def alias(alias: String): Column = name(alias)
-
- /**
- * Gives the column an alias.
- * {{{
- * // Renames colA to colB in select output.
- * df.select($"colA".as("colB"))
- * }}}
- *
- * If the current column has metadata associated with it, this metadata will be propagated to
- * the new column. If this not desired, use the API `as(alias: String, metadata: Metadata)` with
- * explicit metadata.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def as(alias: String): Column = name(alias)
-
- /**
- * (Scala-specific) Assigns the given aliases to the results of a table generating function.
- * {{{
- * // Renames colA to colB in select output.
- * df.select(explode($"myMap").as("key" :: "value" :: Nil))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def as(aliases: Seq[String]): Column = Column { builder =>
- builder.getAliasBuilder.setExpr(expr).addAllName(aliases.asJava)
- }
-
- /**
- * Assigns the given aliases to the results of a table generating function.
- * {{{
- * // Renames colA to colB in select output.
- * df.select(explode($"myMap").as("key" :: "value" :: Nil))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def as(aliases: Array[String]): Column = as(aliases.toImmutableArraySeq)
-
- /**
- * Gives the column an alias.
- * {{{
- * // Renames colA to colB in select output.
- * df.select($"colA".as("colB"))
- * }}}
- *
- * If the current column has metadata associated with it, this metadata will be propagated to
- * the new column. If this not desired, use the API `as(alias: String, metadata: Metadata)` with
- * explicit metadata.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def as(alias: Symbol): Column = name(alias.name)
-
- /**
- * Gives the column an alias with metadata.
- * {{{
- * val metadata: Metadata = ...
- * df.select($"colA".as("colB", metadata))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def as(alias: String, metadata: Metadata): Column = Column { builder =>
- builder.getAliasBuilder
- .setExpr(expr)
- .addName(alias)
- .setMetadata(metadata.json)
- }
-
- /**
- * Gives the column a name (alias).
- * {{{
- * // Renames colA to colB in select output.
- * df.select($"colA".name("colB"))
- * }}}
- *
- * If the current column has metadata associated with it, this metadata will be propagated to
- * the new column. If this not desired, use the API `as(alias: String, metadata: Metadata)` with
- * explicit metadata.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def name(alias: String): Column = as(alias :: Nil)
-
- /**
- * Casts the column to a different data type.
- * {{{
- * // Casts colA to IntegerType.
- * import org.apache.spark.sql.types.IntegerType
- * df.select(df("colA").cast(IntegerType))
- *
- * // equivalent to
- * df.select(df("colA").cast("int"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def cast(to: DataType): Column = Column { builder =>
- builder.getCastBuilder
- .setExpr(expr)
- .setType(DataTypeProtoConverter.toConnectProtoType(to))
- }
-
- /**
- * Casts the column to a different data type, using the canonical string representation of the
- * type. The supported types are: `string`, `boolean`, `byte`, `short`, `int`, `long`, `float`,
- * `double`, `decimal`, `date`, `timestamp`.
- * {{{
- * // Casts colA to integer.
- * df.select(df("colA").cast("int"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def cast(to: String): Column = cast(DataTypeParser.parseDataType(to))
-
- /**
- * Casts the column to a different data type and the result is null on failure.
- * {{{
- * // Casts colA to IntegerType.
- * import org.apache.spark.sql.types.IntegerType
- * df.select(df("colA").try_cast(IntegerType))
- *
- * // equivalent to
- * df.select(df("colA").try_cast("int"))
- * }}}
- *
- * @group expr_ops
- * @since 4.0.0
- */
- def try_cast(to: DataType): Column = Column { builder =>
- builder.getCastBuilder
- .setExpr(expr)
- .setType(DataTypeProtoConverter.toConnectProtoType(to))
- .setEvalMode(proto.Expression.Cast.EvalMode.EVAL_MODE_TRY)
- }
-
- /**
- * Casts the column to a different data type and the result is null on failure.
- * {{{
- * // Casts colA to integer.
- * df.select(df("colA").try_cast("int"))
- * }}}
- *
- * @group expr_ops
- * @since 4.0.0
- */
- def try_cast(to: String): Column = {
- try_cast(DataTypeParser.parseDataType(to))
- }
-
- /**
- * Returns a sort expression based on the descending order of the column.
- * {{{
- * // Scala
- * df.sort(df("age").desc)
- *
- * // Java
- * df.sort(df.col("age").desc());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def desc: Column = desc_nulls_last
-
- /**
- * Returns a sort expression based on the descending order of the column, and null values appear
- * before non-null values.
- * {{{
- * // Scala: sort a DataFrame by age column in descending order and null values appearing first.
- * df.sort(df("age").desc_nulls_first)
- *
- * // Java
- * df.sort(df.col("age").desc_nulls_first());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def desc_nulls_first: Column =
- buildSortOrder(SortDirection.SORT_DIRECTION_DESCENDING, NullOrdering.SORT_NULLS_FIRST)
-
- /**
- * Returns a sort expression based on the descending order of the column, and null values appear
- * after non-null values.
- * {{{
- * // Scala: sort a DataFrame by age column in descending order and null values appearing last.
- * df.sort(df("age").desc_nulls_last)
- *
- * // Java
- * df.sort(df.col("age").desc_nulls_last());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def desc_nulls_last: Column =
- buildSortOrder(SortDirection.SORT_DIRECTION_DESCENDING, NullOrdering.SORT_NULLS_LAST)
-
- /**
- * Returns a sort expression based on ascending order of the column.
- * {{{
- * // Scala: sort a DataFrame by age column in ascending order.
- * df.sort(df("age").asc)
- *
- * // Java
- * df.sort(df.col("age").asc());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def asc: Column = asc_nulls_first
-
- /**
- * Returns a sort expression based on ascending order of the column, and null values return
- * before non-null values.
- * {{{
- * // Scala: sort a DataFrame by age column in ascending order and null values appearing first.
- * df.sort(df("age").asc_nulls_first)
- *
- * // Java
- * df.sort(df.col("age").asc_nulls_first());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def asc_nulls_first: Column =
- buildSortOrder(SortDirection.SORT_DIRECTION_ASCENDING, NullOrdering.SORT_NULLS_FIRST)
-
- /**
- * Returns a sort expression based on ascending order of the column, and null values appear
- * after non-null values.
- * {{{
- * // Scala: sort a DataFrame by age column in ascending order and null values appearing last.
- * df.sort(df("age").asc_nulls_last)
- *
- * // Java
- * df.sort(df.col("age").asc_nulls_last());
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def asc_nulls_last: Column =
- buildSortOrder(SortDirection.SORT_DIRECTION_ASCENDING, NullOrdering.SORT_NULLS_LAST)
-
- private def buildSortOrder(sortDirection: SortDirection, nullOrdering: NullOrdering): Column = {
- Column { builder =>
- builder.getSortOrderBuilder
- .setChild(expr)
- .setDirection(sortDirection)
- .setNullOrdering(nullOrdering)
- }
- }
-
- private[sql] def sortOrder: proto.Expression.SortOrder = {
- val base = if (expr.hasSortOrder) {
- expr
- } else {
- asc.expr
- }
- base.getSortOrder
- }
-
- /**
- * Prints the expression to the console for debugging purposes.
- *
- * @group df_ops
- * @since 3.4.0
- */
- def explain(extended: Boolean): Unit = {
- // scalastyle:off println
- if (extended) {
- println(expr)
- } else {
- println(toString)
- }
- // scalastyle:on println
- }
-
- /**
- * Compute bitwise OR of this expression with another expression.
- * {{{
- * df.select($"colA".bitwiseOR($"colB"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def bitwiseOR(other: Any): Column = fn("|", other)
-
- /**
- * Compute bitwise AND of this expression with another expression.
- * {{{
- * df.select($"colA".bitwiseAND($"colB"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def bitwiseAND(other: Any): Column = fn("&", other)
-
- /**
- * Compute bitwise XOR of this expression with another expression.
- * {{{
- * df.select($"colA".bitwiseXOR($"colB"))
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def bitwiseXOR(other: Any): Column = fn("^", other)
-
- /**
- * Defines a windowing column.
- *
- * {{{
- * val w = Window.partitionBy("name").orderBy("id")
- * df.select(
- * sum("price").over(w.rangeBetween(Window.unboundedPreceding, 2)),
- * avg("price").over(w.rowsBetween(Window.currentRow, 4))
- * )
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def over(window: expressions.WindowSpec): Column = window.withAggregate(this)
-
- /**
- * Defines an empty analytic clause. In this case the analytic function is applied and presented
- * for all rows in the result set.
- *
- * {{{
- * df.select(
- * sum("price").over(),
- * avg("price").over()
- * )
- * }}}
- *
- * @group expr_ops
- * @since 3.4.0
- */
- def over(): Column = over(Window.spec)
-}
-
-object Column {
-
- private[sql] def apply(name: String): Column = new Column(name)
-
- private[sql] def apply(name: String, planId: Option[Long]): Column = new Column(name, planId)
-
- private[sql] def nameToExpression(
- name: String,
- planId: Option[Long] = None): proto.Expression = {
- val builder = proto.Expression.newBuilder()
- name match {
- case "*" =>
- val starBuilder = builder.getUnresolvedStarBuilder
- planId.foreach(starBuilder.setPlanId)
- case _ if name.endsWith(".*") =>
- builder.getUnresolvedStarBuilder.setUnparsedTarget(name)
- case _ =>
- val attributeBuilder = builder.getUnresolvedAttributeBuilder.setUnparsedIdentifier(name)
- planId.foreach(attributeBuilder.setPlanId)
- }
- builder.build()
- }
-
- @Since("4.0.0")
- @DeveloperApi
- def apply(f: proto.Expression.Builder => Unit): Column = {
- val builder = proto.Expression.newBuilder()
- f(builder)
- new Column(builder.build())
- }
-
- private[sql] def fn(name: String, inputs: Column*): Column = {
- fn(name, isDistinct = false, inputs: _*)
- }
-
- private[sql] def fn(name: String, isDistinct: Boolean, inputs: Column*): Column = Column {
- builder =>
- builder.getUnresolvedFunctionBuilder
- .setFunctionName(name)
- .setIsDistinct(isDistinct)
- .addAllArguments(inputs.map(_.expr).asJava)
- }
-}
-
-/**
- * A convenient class used for constructing schema.
- *
- * @since 3.4.0
- */
-class ColumnName(name: String) extends Column(name) {
-
- /**
- * Creates a new `StructField` of type boolean.
- * @since 3.4.0
- */
- def boolean: StructField = StructField(name, BooleanType)
-
- /**
- * Creates a new `StructField` of type byte.
- * @since 3.4.0
- */
- def byte: StructField = StructField(name, ByteType)
-
- /**
- * Creates a new `StructField` of type short.
- * @since 3.4.0
- */
- def short: StructField = StructField(name, ShortType)
-
- /**
- * Creates a new `StructField` of type int.
- * @since 3.4.0
- */
- def int: StructField = StructField(name, IntegerType)
-
- /**
- * Creates a new `StructField` of type long.
- * @since 3.4.0
- */
- def long: StructField = StructField(name, LongType)
-
- /**
- * Creates a new `StructField` of type float.
- * @since 3.4.0
- */
- def float: StructField = StructField(name, FloatType)
-
- /**
- * Creates a new `StructField` of type double.
- * @since 3.4.0
- */
- def double: StructField = StructField(name, DoubleType)
-
- /**
- * Creates a new `StructField` of type string.
- * @since 3.4.0
- */
- def string: StructField = StructField(name, StringType)
-
- /**
- * Creates a new `StructField` of type date.
- * @since 3.4.0
- */
- def date: StructField = StructField(name, DateType)
-
- /**
- * Creates a new `StructField` of type decimal.
- * @since 3.4.0
- */
- def decimal: StructField = StructField(name, DecimalType.USER_DEFAULT)
-
- /**
- * Creates a new `StructField` of type decimal.
- * @since 3.4.0
- */
- def decimal(precision: Int, scale: Int): StructField =
- StructField(name, DecimalType(precision, scale))
-
- /**
- * Creates a new `StructField` of type timestamp.
- * @since 3.4.0
- */
- def timestamp: StructField = StructField(name, TimestampType)
-
- /**
- * Creates a new `StructField` of type binary.
- * @since 3.4.0
- */
- def binary: StructField = StructField(name, BinaryType)
-
- /**
- * Creates a new `StructField` of type array.
- * @since 3.4.0
- */
- def array(dataType: DataType): StructField = StructField(name, ArrayType(dataType))
-
- /**
- * Creates a new `StructField` of type map.
- * @since 3.4.0
- */
- def map(keyType: DataType, valueType: DataType): StructField =
- map(MapType(keyType, valueType))
-
- /**
- * Creates a new `StructField` of type map.
- * @since 3.4.0
- */
- def map(mapType: MapType): StructField = StructField(name, mapType)
-
- /**
- * Creates a new `StructField` of type struct.
- * @since 3.4.0
- */
- def struct(fields: StructField*): StructField = struct(StructType(fields))
-
- /**
- * Creates a new `StructField` of type struct.
- * @since 3.4.0
- */
- def struct(structType: StructType): StructField = StructField(name, structType)
-}
-
-/**
- * A [[Column]] where an [[Encoder]] has been given for the expected input and return type. To
- * create a [[TypedColumn]], use the `as` function on a [[Column]].
- *
- * @tparam T
- * The input type expected for this expression. Can be `Any` if the expression is type checked
- * by the analyzer instead of the compiler (i.e. `expr("sum(...)")`).
- * @tparam U
- * The output type of this column.
- *
- * @since 3.4.0
- */
-class TypedColumn[-T, U] private[sql] (
- expr: proto.Expression,
- private[sql] val encoder: AgnosticEncoder[U])
- extends Column(expr) {
-
- /**
- * Gives the [[TypedColumn]] a name (alias). If the current `TypedColumn` has metadata
- * associated with it, this metadata will be propagated to the new column.
- *
- * @group expr_ops
- * @since 3.4.0
- */
- override def name(alias: String): TypedColumn[T, U] =
- new TypedColumn[T, U](super.name(alias).expr, encoder)
-}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
index 04970f67e99f..7d484d82ec25 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
@@ -32,6 +32,7 @@ import org.apache.spark.util.ArrayImplicits._
* @since 3.4.0
*/
final class DataFrameNaFunctions private[sql] (sparkSession: SparkSession, root: Relation) {
+ import sparkSession.RichColumn
/**
* Returns a new `DataFrame` that drops rows containing any null or NaN values.
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala
index 4eef26da706f..6365f387afce 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala
@@ -34,6 +34,7 @@ import org.apache.spark.util.sketch.{BloomFilter, CountMinSketch}
* @since 3.4.0
*/
final class DataFrameStatFunctions private[sql] (sparkSession: SparkSession, root: Relation) {
+ import sparkSession.RichColumn
/**
* Calculates the approximate quantiles of a numerical column of a DataFrame.
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala
index dd0fc2fb35d2..3f9b22400391 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala
@@ -32,6 +32,7 @@ import org.apache.spark.connect.proto
@Experimental
final class DataFrameWriterV2[T] private[sql] (table: String, ds: Dataset[T])
extends CreateTableWriter[T] {
+ import ds.sparkSession.RichColumn
private var provider: Option[String] = None
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala
index d6cea3723f95..e216fd6dd3e6 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -35,8 +35,9 @@ import org.apache.spark.sql.catalyst.expressions.OrderUtils
import org.apache.spark.sql.connect.client.SparkResult
import org.apache.spark.sql.connect.common.{DataTypeProtoConverter, StorageLevelProtoConverter, UdfUtils}
import org.apache.spark.sql.errors.DataTypeErrors.toSQLId
-import org.apache.spark.sql.expressions.ScalaUserDefinedFunction
+import org.apache.spark.sql.expressions.SparkUserDefinedFunction
import org.apache.spark.sql.functions.{struct, to_json}
+import org.apache.spark.sql.internal.{ColumnNodeToProtoConverter, UnresolvedAttribute, UnresolvedRegex}
import org.apache.spark.sql.streaming.DataStreamWriter
import org.apache.spark.sql.types.{Metadata, StructType}
import org.apache.spark.storage.StorageLevel
@@ -133,7 +134,12 @@ class Dataset[T] private[sql] (
val sparkSession: SparkSession,
@DeveloperApi val plan: proto.Plan,
val encoder: Encoder[T])
- extends Serializable {
+ extends api.Dataset[T]
+ with Serializable {
+ override type DS[_] = Dataset[_]
+
+ import sparkSession.RichColumn
+
// Make sure we don't forget to set plan id.
assert(plan.getRoot.getCommon.hasPlanId)
@@ -844,12 +850,15 @@ class Dataset[T] private[sql] (
builder.setJoinType(proto.Join.JoinType.JOIN_TYPE_CROSS)
}
- private def buildSort(global: Boolean, sortExprs: Seq[Column]): Dataset[T] = {
+ private def buildSort(global: Boolean, sortColumns: Seq[Column]): Dataset[T] = {
+ val sortExprs = sortColumns.map { c =>
+ ColumnNodeToProtoConverter(c.sortOrder).getSortOrder
+ }
sparkSession.newDataset(agnosticEncoder) { builder =>
builder.getSortBuilder
.setInput(plan.getRoot)
.setIsGlobal(global)
- .addAllOrder(sortExprs.map(_.sortOrder).asJava)
+ .addAllOrder(sortExprs.asJava)
}
}
@@ -1052,9 +1061,7 @@ class Dataset[T] private[sql] (
* @group untypedrel
* @since 3.4.0
*/
- def col(colName: String): Column = {
- Column.apply(colName, getPlanId)
- }
+ def col(colName: String): Column = new Column(colName, getPlanId)
/**
* Selects a metadata column based on its logical column name, and returns it as a [[Column]].
@@ -1065,11 +1072,8 @@ class Dataset[T] private[sql] (
* @group untypedrel
* @since 3.5.0
*/
- def metadataColumn(colName: String): Column = Column { builder =>
- val attributeBuilder = builder.getUnresolvedAttributeBuilder
- .setUnparsedIdentifier(colName)
- .setIsMetadataColumn(true)
- getPlanId.foreach(attributeBuilder.setPlanId)
+ def metadataColumn(colName: String): Column = {
+ Column(UnresolvedAttribute(colName, getPlanId, isMetadataColumn = true))
}
/**
@@ -1078,10 +1082,7 @@ class Dataset[T] private[sql] (
* @since 3.4.0
*/
def colRegex(colName: String): Column = {
- Column { builder =>
- val unresolvedRegexBuilder = builder.getUnresolvedRegexBuilder.setColName(colName)
- getPlanId.foreach(unresolvedRegexBuilder.setPlanId)
- }
+ Column(UnresolvedRegex(colName, getPlanId))
}
/**
@@ -1178,16 +1179,16 @@ class Dataset[T] private[sql] (
* @since 3.4.0
*/
def select[U1](c1: TypedColumn[T, U1]): Dataset[U1] = {
- val encoder = c1.encoder
- val expr = if (encoder.schema == encoder.dataType) {
- functions.inline(functions.array(c1)).expr
+ val encoder = encoderFor(c1.encoder)
+ val col = if (encoder.schema == encoder.dataType) {
+ functions.inline(functions.array(c1))
} else {
- c1.expr
+ c1
}
sparkSession.newDataset(encoder) { builder =>
builder.getProjectBuilder
.setInput(plan.getRoot)
- .addExpressions(expr)
+ .addExpressions(col.typedExpr(this.encoder))
}
}
@@ -1197,7 +1198,7 @@ class Dataset[T] private[sql] (
* cast appropriately for the user facing interface.
*/
private def selectUntyped(columns: TypedColumn[_, _]*): Dataset[_] = {
- val encoder = ProductEncoder.tuple(columns.map(_.encoder))
+ val encoder = ProductEncoder.tuple(columns.map(c => encoderFor(c.encoder)))
selectUntyped(encoder, columns)
}
@@ -1209,7 +1210,7 @@ class Dataset[T] private[sql] (
sparkSession.newDataset(encoder) { builder =>
builder.getProjectBuilder
.setInput(plan.getRoot)
- .addAllExpressions(cols.map(_.expr).asJava)
+ .addAllExpressions(cols.map(_.typedExpr(this.encoder)).asJava)
}
}
@@ -1388,7 +1389,7 @@ class Dataset[T] private[sql] (
* @since 3.5.0
*/
def reduce(func: (T, T) => T): T = {
- val udf = ScalaUserDefinedFunction(
+ val udf = SparkUserDefinedFunction(
function = func,
inputEncoders = agnosticEncoder :: agnosticEncoder :: Nil,
outputEncoder = agnosticEncoder)
@@ -2706,7 +2707,7 @@ class Dataset[T] private[sql] (
* @since 3.5.0
*/
def filter(func: T => Boolean): Dataset[T] = {
- val udf = ScalaUserDefinedFunction(
+ val udf = SparkUserDefinedFunction(
function = func,
inputEncoders = agnosticEncoder :: Nil,
outputEncoder = PrimitiveBooleanEncoder)
@@ -2759,7 +2760,7 @@ class Dataset[T] private[sql] (
*/
def mapPartitions[U: Encoder](func: Iterator[T] => Iterator[U]): Dataset[U] = {
val outputEncoder = encoderFor[U]
- val udf = ScalaUserDefinedFunction(
+ val udf = SparkUserDefinedFunction(
function = func,
inputEncoders = agnosticEncoder :: Nil,
outputEncoder = outputEncoder)
@@ -2831,7 +2832,7 @@ class Dataset[T] private[sql] (
*/
@deprecated("use flatMap() or select() with functions.explode() instead", "3.5.0")
def explode[A <: Product: TypeTag](input: Column*)(f: Row => IterableOnce[A]): DataFrame = {
- val generator = ScalaUserDefinedFunction(
+ val generator = SparkUserDefinedFunction(
UdfUtils.iterableOnceToSeq(f),
UnboundRowEncoder :: Nil,
ScalaReflection.encoderFor[Seq[A]])
@@ -2863,7 +2864,7 @@ class Dataset[T] private[sql] (
@deprecated("use flatMap() or select() with functions.explode() instead", "3.5.0")
def explode[A, B: TypeTag](inputColumn: String, outputColumn: String)(
f: A => IterableOnce[B]): DataFrame = {
- val generator = ScalaUserDefinedFunction(
+ val generator = SparkUserDefinedFunction(
UdfUtils.iterableOnceToSeq(f),
Nil,
ScalaReflection.encoderFor[Seq[B]])
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
index 38a4157622ea..04b620bdf8f9 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
@@ -27,8 +27,9 @@ import org.apache.spark.connect.proto
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoder
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.ProductEncoder
import org.apache.spark.sql.connect.common.UdfUtils
-import org.apache.spark.sql.expressions.ScalaUserDefinedFunction
+import org.apache.spark.sql.expressions.SparkUserDefinedFunction
import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.ColumnNodeToProtoConverter.toExpr
import org.apache.spark.sql.streaming.{GroupState, GroupStateTimeout, OutputMode, StatefulProcessor, StatefulProcessorWithInitialState, TimeMode}
/**
@@ -941,6 +942,7 @@ private class KeyValueGroupedDatasetImpl[K, V, IK, IV](
private val valueMapFunc: IV => V,
private val keysFunc: () => Dataset[IK])
extends KeyValueGroupedDataset[K, V] {
+ import sparkSession.RichColumn
override def keyAs[L: Encoder]: KeyValueGroupedDataset[L, V] = {
new KeyValueGroupedDatasetImpl[L, V, IK, IV](
@@ -1011,25 +1013,26 @@ private class KeyValueGroupedDatasetImpl[K, V, IK, IV](
override protected def aggUntyped(columns: TypedColumn[_, _]*): Dataset[_] = {
// TODO(SPARK-43415): For each column, apply the valueMap func first
- val rEnc = ProductEncoder.tuple(kEncoder +: columns.map(_.encoder)) // apply keyAs change
+ // apply keyAs change
+ val rEnc = ProductEncoder.tuple(kEncoder +: columns.map(c => encoderFor(c.encoder)))
sparkSession.newDataset(rEnc) { builder =>
builder.getAggregateBuilder
.setInput(plan.getRoot)
.setGroupType(proto.Aggregate.GroupType.GROUP_TYPE_GROUPBY)
.addAllGroupingExpressions(groupingExprs)
- .addAllAggregateExpressions(columns.map(_.expr).asJava)
+ .addAllAggregateExpressions(columns.map(_.typedExpr(vEncoder)).asJava)
}
}
override def reduceGroups(f: (V, V) => V): Dataset[(K, V)] = {
val inputEncoders = Seq(vEncoder, vEncoder)
- val udf = ScalaUserDefinedFunction(
+ val udf = SparkUserDefinedFunction(
function = f,
inputEncoders = inputEncoders,
outputEncoder = vEncoder)
val input = udf.apply(inputEncoders.map(_ => col("*")): _*)
- val expr = Column.fn("reduce", input).expr
- val aggregator: TypedColumn[V, V] = new TypedColumn[V, V](expr, vEncoder)
+ val expr = Column.fn("reduce", input)
+ val aggregator: TypedColumn[V, V] = new TypedColumn[V, V](expr.node, vEncoder)
agg(aggregator)
}
@@ -1083,7 +1086,7 @@ private class KeyValueGroupedDatasetImpl[K, V, IK, IV](
private def getUdf[U: Encoder](nf: AnyRef, outputEncoder: AgnosticEncoder[U])(
inEncoders: AgnosticEncoder[_]*): proto.CommonInlineUserDefinedFunction = {
val inputEncoders = kEncoder +: inEncoders // Apply keyAs changes by setting kEncoder
- val udf = ScalaUserDefinedFunction(
+ val udf = SparkUserDefinedFunction(
function = nf,
inputEncoders = inputEncoders,
outputEncoder = outputEncoder)
@@ -1102,18 +1105,19 @@ private object KeyValueGroupedDatasetImpl {
ds: Dataset[V],
kEncoder: AgnosticEncoder[K],
groupingFunc: V => K): KeyValueGroupedDatasetImpl[K, V, K, V] = {
- val gf = ScalaUserDefinedFunction(
+ val gf = SparkUserDefinedFunction(
function = groupingFunc,
inputEncoders = ds.agnosticEncoder :: Nil, // Using the original value and key encoders
outputEncoder = kEncoder)
+ val session = ds.sparkSession
new KeyValueGroupedDatasetImpl(
- ds.sparkSession,
+ session,
ds.plan,
kEncoder,
kEncoder,
ds.agnosticEncoder,
ds.agnosticEncoder,
- Arrays.asList(gf.apply(col("*")).expr),
+ Arrays.asList(toExpr(gf.apply(col("*")))),
UdfUtils.identical(),
() => ds.map(groupingFunc)(kEncoder))
}
@@ -1124,19 +1128,19 @@ private object KeyValueGroupedDatasetImpl {
vEncoder: AgnosticEncoder[V],
groupingExprs: Seq[Column]): KeyValueGroupedDatasetImpl[K, V, K, V] = {
// Use a dummy udf to pass the K V encoders
- val dummyGroupingFunc = ScalaUserDefinedFunction(
+ val dummyGroupingFunc = SparkUserDefinedFunction(
function = UdfUtils.noOp[V, K](),
inputEncoders = vEncoder :: Nil,
outputEncoder = kEncoder).apply(col("*"))
-
+ val session = df.sparkSession
new KeyValueGroupedDatasetImpl(
- df.sparkSession,
+ session,
df.plan,
kEncoder,
kEncoder,
vEncoder,
vEncoder,
- (Seq(dummyGroupingFunc) ++ groupingExprs).map(_.expr).asJava,
+ (Seq(dummyGroupingFunc) ++ groupingExprs).map(toExpr).asJava,
UdfUtils.identical(),
() => df.select(groupingExprs: _*).as(kEncoder))
}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/MergeIntoWriter.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/MergeIntoWriter.scala
index 197cfac32a4b..71813af1e354 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/MergeIntoWriter.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/MergeIntoWriter.scala
@@ -24,7 +24,6 @@ import org.apache.spark.annotation.Experimental
import org.apache.spark.connect.proto
import org.apache.spark.connect.proto.{Expression, MergeIntoTableCommand}
import org.apache.spark.connect.proto.MergeAction
-import org.apache.spark.sql.MergeIntoWriter.buildMergeAction
import org.apache.spark.sql.functions.expr
/**
@@ -50,6 +49,7 @@ class MergeIntoWriter[T] private[sql] (
ds: Dataset[T],
on: Column,
schemaEvolutionEnabled: Boolean = false) {
+ import ds.sparkSession.RichColumn
private[sql] var matchedActions: Seq[MergeAction] = Seq.empty[MergeAction]
private[sql] var notMatchedActions: Seq[MergeAction] = Seq.empty[MergeAction]
@@ -92,7 +92,7 @@ class MergeIntoWriter[T] private[sql] (
* a new `WhenMatched` object configured with the specified condition.
*/
def whenMatched(condition: Column): WhenMatched[T] = {
- new WhenMatched[T](this, Some(condition.expr))
+ new WhenMatched[T](this, Some(condition))
}
/**
@@ -130,7 +130,7 @@ class MergeIntoWriter[T] private[sql] (
* a new `WhenNotMatched` object configured with the specified condition.
*/
def whenNotMatched(condition: Column): WhenNotMatched[T] = {
- new WhenNotMatched[T](this, Some(condition.expr))
+ new WhenNotMatched[T](this, Some(condition))
}
/**
@@ -171,7 +171,7 @@ class MergeIntoWriter[T] private[sql] (
* a new `WhenNotMatchedBySource` object configured with the specified condition.
*/
def whenNotMatchedBySource(condition: Column): WhenNotMatchedBySource[T] = {
- new WhenNotMatchedBySource[T](this, Some(condition.expr))
+ new WhenNotMatchedBySource[T](this, Some(condition))
}
/**
@@ -234,6 +234,26 @@ class MergeIntoWriter[T] private[sql] (
this.notMatchedBySourceActions = this.notMatchedBySourceActions :++ action
this
}
+
+ private[sql] def buildMergeAction(
+ actionType: MergeAction.ActionType,
+ conditionOpt: Option[Column],
+ assignmentsOpt: Option[Map[String, Column]] = None): MergeAction = {
+ val assignmentsProtoOpt = assignmentsOpt.map {
+ _.map { case (k, v) =>
+ MergeAction.Assignment
+ .newBuilder()
+ .setKey(expr(k).expr)
+ .setValue(v.expr)
+ .build()
+ }.toSeq.asJava
+ }
+
+ val builder = MergeAction.newBuilder().setActionType(actionType)
+ conditionOpt.map(c => builder.setCondition(c.expr))
+ assignmentsProtoOpt.map(builder.addAllAssignments)
+ builder.build()
+ }
}
/**
@@ -251,7 +271,7 @@ class MergeIntoWriter[T] private[sql] (
*/
case class WhenMatched[T] private[sql] (
mergeIntoWriter: MergeIntoWriter[T],
- condition: Option[Expression]) {
+ condition: Option[Column]) {
/**
* Specifies an action to update all matched rows in the DataFrame.
@@ -261,7 +281,8 @@ case class WhenMatched[T] private[sql] (
*/
def updateAll(): MergeIntoWriter[T] = {
mergeIntoWriter.withNewMatchedActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE_STAR, condition))
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE_STAR, condition))
}
/**
@@ -275,7 +296,8 @@ case class WhenMatched[T] private[sql] (
*/
def update(map: Map[String, Column]): MergeIntoWriter[T] = {
mergeIntoWriter.withNewMatchedActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE, condition, Some(map)))
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE, condition, Some(map)))
}
/**
@@ -286,7 +308,7 @@ case class WhenMatched[T] private[sql] (
*/
def delete(): MergeIntoWriter[T] = {
mergeIntoWriter.withNewMatchedActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_DELETE, condition))
+ mergeIntoWriter.buildMergeAction(MergeAction.ActionType.ACTION_TYPE_DELETE, condition))
}
}
@@ -305,7 +327,7 @@ case class WhenMatched[T] private[sql] (
*/
case class WhenNotMatched[T] private[sql] (
mergeIntoWriter: MergeIntoWriter[T],
- condition: Option[Expression]) {
+ condition: Option[Column]) {
/**
* Specifies an action to insert all non-matched rows into the DataFrame.
@@ -315,7 +337,8 @@ case class WhenNotMatched[T] private[sql] (
*/
def insertAll(): MergeIntoWriter[T] = {
mergeIntoWriter.withNewNotMatchedActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_INSERT_STAR, condition))
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_INSERT_STAR, condition))
}
/**
@@ -329,8 +352,8 @@ case class WhenNotMatched[T] private[sql] (
*/
def insert(map: Map[String, Column]): MergeIntoWriter[T] = {
mergeIntoWriter.withNewNotMatchedActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_INSERT, condition, Some(map)))
-
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_INSERT, condition, Some(map)))
}
}
@@ -347,7 +370,7 @@ case class WhenNotMatched[T] private[sql] (
*/
case class WhenNotMatchedBySource[T] private[sql] (
mergeIntoWriter: MergeIntoWriter[T],
- condition: Option[Expression]) {
+ condition: Option[Column]) {
/**
* Specifies an action to update all non-matched rows in the target DataFrame when not matched
@@ -358,7 +381,8 @@ case class WhenNotMatchedBySource[T] private[sql] (
*/
def updateAll(): MergeIntoWriter[T] = {
mergeIntoWriter.withNewNotMatchedBySourceActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE_STAR, condition))
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE_STAR, condition))
}
/**
@@ -372,7 +396,8 @@ case class WhenNotMatchedBySource[T] private[sql] (
*/
def update(map: Map[String, Column]): MergeIntoWriter[T] = {
mergeIntoWriter.withNewNotMatchedBySourceActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE, condition, Some(map)))
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_UPDATE, condition, Some(map)))
}
/**
@@ -384,28 +409,7 @@ case class WhenNotMatchedBySource[T] private[sql] (
*/
def delete(): MergeIntoWriter[T] = {
mergeIntoWriter.withNewNotMatchedBySourceActions(
- buildMergeAction(MergeAction.ActionType.ACTION_TYPE_DELETE, condition))
- }
-}
-
-private object MergeIntoWriter {
- private[sql] def buildMergeAction(
- actionType: MergeAction.ActionType,
- conditionOpt: Option[Expression],
- assignmentsOpt: Option[Map[String, Column]] = None): MergeAction = {
- val assignmentsProtoOpt = assignmentsOpt.map {
- _.map { case (k, v) =>
- MergeAction.Assignment
- .newBuilder()
- .setKey(expr(k).expr)
- .setValue(v.expr)
- .build()
- }.toSeq.asJava
- }
-
- var builder = MergeAction.newBuilder().setActionType(actionType)
- builder = conditionOpt.map(builder.setCondition).getOrElse(builder)
- builder = assignmentsProtoOpt.map(builder.addAllAssignments).getOrElse(builder)
- builder.build()
+ mergeIntoWriter
+ .buildMergeAction(MergeAction.ActionType.ACTION_TYPE_DELETE, condition))
}
}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala
index 754906f736cf..0c8657e12d8d 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala
@@ -41,13 +41,14 @@ class RelationalGroupedDataset private[sql] (
groupType: proto.Aggregate.GroupType,
pivot: Option[proto.Aggregate.Pivot] = None,
groupingSets: Option[Seq[proto.Aggregate.GroupingSets]] = None) {
+ import df.sparkSession.RichColumn
private[this] def toDF(aggExprs: Seq[Column]): DataFrame = {
df.sparkSession.newDataFrame { builder =>
builder.getAggregateBuilder
.setInput(df.plan.getRoot)
.addAllGroupingExpressions(groupingExprs.map(_.expr).asJava)
- .addAllAggregateExpressions(aggExprs.map(e => e.expr).asJava)
+ .addAllAggregateExpressions(aggExprs.map(e => e.typedExpr(df.encoder)).asJava)
groupType match {
case proto.Aggregate.GroupType.GROUP_TYPE_ROLLUP =>
@@ -182,10 +183,7 @@ class RelationalGroupedDataset private[sql] (
*/
@scala.annotation.varargs
def agg(expr: Column, exprs: Column*): DataFrame = {
- toDF((expr +: exprs).map { case c =>
- c
- // TODO: deal with typed columns.
- })
+ toDF(expr +: exprs)
}
/**
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala
index 2e54617928aa..50985d4373d6 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -42,6 +42,7 @@ import org.apache.spark.sql.connect.client.SparkConnectClient.Configuration
import org.apache.spark.sql.connect.client.arrow.ArrowSerializer
import org.apache.spark.sql.functions.lit
import org.apache.spark.sql.internal.{CatalogImpl, SessionCleaner, SqlApiConf}
+import org.apache.spark.sql.internal.ColumnNodeToProtoConverter.{toExpr, toTypedExpr}
import org.apache.spark.sql.streaming.DataStreamReader
import org.apache.spark.sql.streaming.StreamingQueryManager
import org.apache.spark.sql.types.StructType
@@ -820,6 +821,11 @@ class SparkSession private[sql] (
observationOrNull.setMetricsAndNotify(Some(metrics))
}
}
+
+ implicit class RichColumn(c: Column) {
+ def expr: proto.Expression = toExpr(c)
+ def typedExpr[T](e: Encoder[T]): proto.Expression = toTypedExpr(c, e)
+ }
}
// The minimal builder needed to create a spark session.
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/UDFRegistration.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/UDFRegistration.scala
index 5965a2b7a61d..3a84d43ceae3 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/UDFRegistration.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/UDFRegistration.scala
@@ -17,13 +17,8 @@
package org.apache.spark.sql
-import scala.reflect.runtime.universe.{typeTag, TypeTag}
-
-import org.apache.spark.internal.Logging
-import org.apache.spark.sql.api.java._
-import org.apache.spark.sql.connect.common.UdfUtils
-import org.apache.spark.sql.expressions.{ScalaUserDefinedFunction, UserDefinedFunction}
-import org.apache.spark.sql.types.DataType
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.internal.UdfToProtoUtils
/**
* Functions for registering user-defined functions. Use `SparkSession.udf` to access this:
@@ -34,1261 +29,14 @@ import org.apache.spark.sql.types.DataType
*
* @since 3.5.0
*/
-class UDFRegistration(session: SparkSession) extends Logging {
-
- /**
- * Registers a user-defined function (UDF), for a UDF that's already defined using the Dataset
- * API (i.e. of type UserDefinedFunction). To change a UDF to nondeterministic, call the API
- * `UserDefinedFunction.asNondeterministic()`. To change a UDF to nonNullable, call the API
- * `UserDefinedFunction.asNonNullable()`.
- *
- * Example:
- * {{{
- * val foo = udf(() => Math.random())
- * spark.udf.register("random", foo.asNondeterministic())
- *
- * val bar = udf(() => "bar")
- * spark.udf.register("stringLit", bar.asNonNullable())
- * }}}
- *
- * @param name
- * the name of the UDF.
- * @param udf
- * the UDF needs to be registered.
- * @return
- * the registered UDF.
- *
- * @since 3.5.0
- */
- def register(name: String, udf: UserDefinedFunction): UserDefinedFunction = {
- udf.withName(name) match {
- case scalarUdf: ScalaUserDefinedFunction =>
- session.registerUdf(scalarUdf.toProto)
- scalarUdf
- case other =>
- throw new UnsupportedOperationException(
- s"Registering a UDF of type " +
- s"${other.getClass.getSimpleName} is currently unsupported.")
- }
- }
-
- // scalastyle:off line.size.limit
-
- /* register 0-22 were generated by this script:
- (0 to 22).foreach { x =>
- val params = (1 to x).map(num => s"A$num").mkString(", ")
- val typeTags = (1 to x).map(i => s"A$i: TypeTag").foldLeft("RT: TypeTag")(_ + ", " + _)
- println(s"""
- |/**
- | * Registers a deterministic Scala closure of $x arguments as user-defined function (UDF).
- | * @tparam RT return type of UDF.
- | * @since 3.5.0
- | */
- |def register[$typeTags](name: String, func: ($params) => RT): UserDefinedFunction = {
- | register(name, functions.udf(func))
- |}""".stripMargin)
- }
- */
-
- /**
- * Registers a deterministic Scala closure of 0 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag](name: String, func: () => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(func, typeTag[RT])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 1 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag, A1: TypeTag](name: String, func: (A1) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(func, typeTag[RT], typeTag[A1])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 2 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag, A1: TypeTag, A2: TypeTag](
- name: String,
- func: (A1, A2) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(func, typeTag[RT], typeTag[A1], typeTag[A2])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 3 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag, A1: TypeTag, A2: TypeTag, A3: TypeTag](
- name: String,
- func: (A1, A2, A3) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(func, typeTag[RT], typeTag[A1], typeTag[A2], typeTag[A3])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 4 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag, A1: TypeTag, A2: TypeTag, A3: TypeTag, A4: TypeTag](
- name: String,
- func: (A1, A2, A3, A4) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 5 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[RT: TypeTag, A1: TypeTag, A2: TypeTag, A3: TypeTag, A4: TypeTag, A5: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 6 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag](name: String, func: (A1, A2, A3, A4, A5, A6) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 7 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 8 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 9 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 10 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 11 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 12 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 13 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13) => RT)
- : UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 14 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14) => RT)
- : UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 15 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15) => RT)
- : UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 16 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16) => RT)
- : UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 17 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag](
- name: String,
- func: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17) => RT)
- : UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 18 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag,
- A18: TypeTag](
- name: String,
- func: (
- A1,
- A2,
- A3,
- A4,
- A5,
- A6,
- A7,
- A8,
- A9,
- A10,
- A11,
- A12,
- A13,
- A14,
- A15,
- A16,
- A17,
- A18) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17],
- typeTag[A18])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 19 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag,
- A18: TypeTag,
- A19: TypeTag](
- name: String,
- func: (
- A1,
- A2,
- A3,
- A4,
- A5,
- A6,
- A7,
- A8,
- A9,
- A10,
- A11,
- A12,
- A13,
- A14,
- A15,
- A16,
- A17,
- A18,
- A19) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17],
- typeTag[A18],
- typeTag[A19])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 20 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag,
- A18: TypeTag,
- A19: TypeTag,
- A20: TypeTag](
- name: String,
- func: (
- A1,
- A2,
- A3,
- A4,
- A5,
- A6,
- A7,
- A8,
- A9,
- A10,
- A11,
- A12,
- A13,
- A14,
- A15,
- A16,
- A17,
- A18,
- A19,
- A20) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17],
- typeTag[A18],
- typeTag[A19],
- typeTag[A20])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 21 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag,
- A18: TypeTag,
- A19: TypeTag,
- A20: TypeTag,
- A21: TypeTag](
- name: String,
- func: (
- A1,
- A2,
- A3,
- A4,
- A5,
- A6,
- A7,
- A8,
- A9,
- A10,
- A11,
- A12,
- A13,
- A14,
- A15,
- A16,
- A17,
- A18,
- A19,
- A20,
- A21) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17],
- typeTag[A18],
- typeTag[A19],
- typeTag[A20],
- typeTag[A21])
- register(name, udf)
- }
-
- /**
- * Registers a deterministic Scala closure of 22 arguments as user-defined function (UDF).
- * @tparam RT
- * return type of UDF.
- * @since 3.5.0
- */
- def register[
- RT: TypeTag,
- A1: TypeTag,
- A2: TypeTag,
- A3: TypeTag,
- A4: TypeTag,
- A5: TypeTag,
- A6: TypeTag,
- A7: TypeTag,
- A8: TypeTag,
- A9: TypeTag,
- A10: TypeTag,
- A11: TypeTag,
- A12: TypeTag,
- A13: TypeTag,
- A14: TypeTag,
- A15: TypeTag,
- A16: TypeTag,
- A17: TypeTag,
- A18: TypeTag,
- A19: TypeTag,
- A20: TypeTag,
- A21: TypeTag,
- A22: TypeTag](
- name: String,
- func: (
- A1,
- A2,
- A3,
- A4,
- A5,
- A6,
- A7,
- A8,
- A9,
- A10,
- A11,
- A12,
- A13,
- A14,
- A15,
- A16,
- A17,
- A18,
- A19,
- A20,
- A21,
- A22) => RT): UserDefinedFunction = {
- val udf = ScalaUserDefinedFunction(
- func,
- typeTag[RT],
- typeTag[A1],
- typeTag[A2],
- typeTag[A3],
- typeTag[A4],
- typeTag[A5],
- typeTag[A6],
- typeTag[A7],
- typeTag[A8],
- typeTag[A9],
- typeTag[A10],
- typeTag[A11],
- typeTag[A12],
- typeTag[A13],
- typeTag[A14],
- typeTag[A15],
- typeTag[A16],
- typeTag[A17],
- typeTag[A18],
- typeTag[A19],
- typeTag[A20],
- typeTag[A21],
- typeTag[A22])
- register(name, udf)
- }
-
- // (0 to 22).foreach { i =>
- // val extTypeArgs = (0 to i).map(_ => "_").mkString(", ")
- // val version = "3.5.0"
- // println(s"""
- // |/**
- // | * Register a deterministic Java UDF$i instance as user-defined function (UDF).
- // | * @since $version
- // | */
- // |def register(name: String, f: UDF$i[$extTypeArgs], returnType: DataType): Unit = {
- // | val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- // | register(name, udf)
- // |}""".stripMargin)
- // }
-
- /**
- * Register a deterministic Java UDF0 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF0[_], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF1 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF1[_, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF2 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF2[_, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF3 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF3[_, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF4 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF4[_, _, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF5 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF5[_, _, _, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF6 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF6[_, _, _, _, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF7 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF7[_, _, _, _, _, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF8 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(name: String, f: UDF8[_, _, _, _, _, _, _, _, _], returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF9 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF9[_, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF10 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF10[_, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF11 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF11[_, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF12 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF12[_, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF13 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF13[_, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF14 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF14[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF15 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF16 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF17 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF18 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF19 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF20 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF21 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
- name: String,
- f: UDF21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
- }
-
- /**
- * Register a deterministic Java UDF22 instance as user-defined function (UDF).
- * @since 3.5.0
- */
- def register(
+class UDFRegistration(session: SparkSession) extends api.UDFRegistration {
+ override protected def register(
name: String,
- f: UDF22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _],
- returnType: DataType): Unit = {
- val udf = ScalaUserDefinedFunction(UdfUtils.wrap(f), returnType)
- register(name, udf)
+ udf: UserDefinedFunction,
+ source: String,
+ validateParameterCount: Boolean): UserDefinedFunction = {
+ val named = udf.withName(name)
+ session.registerUdf(UdfToProtoUtils.toProto(named))
+ named
}
- // scalastyle:on line.size.limit
}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/avro/functions.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/avro/functions.scala
index 61a9c908c7a4..e80bccfee4c9 100644
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/avro/functions.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/avro/functions.scala
@@ -21,7 +21,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.Column
-import org.apache.spark.sql.functions.{fnWithOptions, lit}
+import org.apache.spark.sql.functions.lit
// scalastyle:off: object.name
object functions {
@@ -64,7 +64,7 @@ object functions {
data: Column,
jsonFormatSchema: String,
options: java.util.Map[String, String]): Column = {
- fnWithOptions("from_avro", options.asScala.iterator, data, lit(jsonFormatSchema))
+ Column.fnWithOptions("from_avro", options.asScala.iterator, data, lit(jsonFormatSchema))
}
/**
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Aggregator.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Aggregator.scala
deleted file mode 100644
index b177c110285d..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Aggregator.scala
+++ /dev/null
@@ -1,166 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.expressions
-
-import scala.reflect.runtime.universe._
-
-import org.apache.spark.connect.proto
-import org.apache.spark.sql.{encoderFor, Encoder, TypedColumn}
-import org.apache.spark.sql.catalyst.ScalaReflection
-
-/**
- * A base class for user-defined aggregations, which can be used in `Dataset` operations to take
- * all of the elements of a group and reduce them to a single value.
- *
- * For example, the following aggregator extracts an `int` from a specific class and adds them up:
- * {{{
- * case class Data(i: Int)
- *
- * val customSummer = new Aggregator[Data, Int, Int] {
- * def zero: Int = 0
- * def reduce(b: Int, a: Data): Int = b + a.i
- * def merge(b1: Int, b2: Int): Int = b1 + b2
- * def finish(r: Int): Int = r
- * def bufferEncoder: Encoder[Int] = Encoders.scalaInt
- * def outputEncoder: Encoder[Int] = Encoders.scalaInt
- * }
- *
- * spark.udf.register("customSummer", udaf(customSummer))
- * val ds: Dataset[Data] = ...
- * val aggregated = ds.selectExpr("customSummer(i)")
- * }}}
- *
- * Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird
- *
- * @tparam IN
- * The input type for the aggregation.
- * @tparam BUF
- * The type of the intermediate value of the reduction.
- * @tparam OUT
- * The type of the final output result.
- * @since 4.0.0
- */
-@SerialVersionUID(2093413866369130093L)
-abstract class Aggregator[-IN, BUF, OUT] extends Serializable {
-
- /**
- * A zero value for this aggregation. Should satisfy the property that any b + zero = b.
- * @since 4.0.0
- */
- def zero: BUF
-
- /**
- * Combine two values to produce a new value. For performance, the function may modify `b` and
- * return it instead of constructing new object for b.
- * @since 4.0.0
- */
- def reduce(b: BUF, a: IN): BUF
-
- /**
- * Merge two intermediate values.
- * @since 4.0.0
- */
- def merge(b1: BUF, b2: BUF): BUF
-
- /**
- * Transform the output of the reduction.
- * @since 4.0.0
- */
- def finish(reduction: BUF): OUT
-
- /**
- * Specifies the `Encoder` for the intermediate value type.
- * @since 4.0.0
- */
- def bufferEncoder: Encoder[BUF]
-
- /**
- * Specifies the `Encoder` for the final output value type.
- * @since 4.0.0
- */
- def outputEncoder: Encoder[OUT]
-
- /**
- * Returns this `Aggregator` as a `TypedColumn` that can be used in `Dataset` operations.
- * @since 4.0.0
- */
- def toColumn: TypedColumn[IN, OUT] = {
- val ttpe = getInputTypeTag[IN]
- val inputEncoder = ScalaReflection.encoderFor(ttpe)
- val udaf =
- ScalaUserDefinedFunction(
- this,
- Seq(inputEncoder),
- encoderFor(outputEncoder),
- aggregate = true)
-
- val builder = proto.TypedAggregateExpression.newBuilder()
- builder.setScalarScalaUdf(udaf.udf)
- val expr = proto.Expression.newBuilder().setTypedAggregateExpression(builder).build()
-
- new TypedColumn(expr, encoderFor(outputEncoder))
- }
-
- private final def getInputTypeTag[T]: TypeTag[T] = {
- val mirror = runtimeMirror(this.getClass.getClassLoader)
- val tpe = mirror.classSymbol(this.getClass).toType
- // Find the most generic (last in the tree) Aggregator class
- val baseAgg =
- tpe.baseClasses
- .findLast(_.asClass.toType <:< typeOf[Aggregator[_, _, _]])
- .getOrElse(throw new IllegalStateException("Could not find the Aggregator base class."))
- val typeArgs = tpe.baseType(baseAgg).typeArgs
- assert(
- typeArgs.length == 3,
- s"Aggregator should have 3 type arguments, " +
- s"but found ${typeArgs.length}: ${typeArgs.mkString}.")
- val inType = typeArgs.head
-
- import scala.reflect.api._
- def areCompatibleMirrors(one: Mirror[_], another: Mirror[_]): Boolean = {
- def checkAllParents(target: JavaMirror, candidate: JavaMirror): Boolean = {
- var current = candidate.classLoader
- while (current != null) {
- if (current == target.classLoader) {
- return true
- }
- current = current.getParent
- }
- false
- }
-
- (one, another) match {
- case (a: JavaMirror, b: JavaMirror) =>
- a == b || checkAllParents(a, b) || checkAllParents(b, a)
- case _ => one == another
- }
- }
-
- TypeTag(
- mirror,
- new TypeCreator {
- def apply[U <: Universe with Singleton](m: Mirror[U]): U#Type =
- if (areCompatibleMirrors(m, mirror)) {
- inType.asInstanceOf[U#Type]
- } else {
- throw new IllegalArgumentException(
- s"Type tag defined in [$mirror] cannot be migrated to another mirror [$m].")
- }
- })
- }
-}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
deleted file mode 100644
index 781e5c17f5f8..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
+++ /dev/null
@@ -1,220 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.spark.sql.expressions
-
-import scala.collection.mutable
-import scala.jdk.CollectionConverters._
-import scala.reflect.runtime.universe.TypeTag
-import scala.util.control.NonFatal
-
-import com.google.protobuf.ByteString
-
-import org.apache.spark.SparkException
-import org.apache.spark.connect.proto
-import org.apache.spark.sql.Column
-import org.apache.spark.sql.catalyst.ScalaReflection
-import org.apache.spark.sql.catalyst.encoders.{AgnosticEncoder, RowEncoder}
-import org.apache.spark.sql.connect.common.{DataTypeProtoConverter, UdfPacket}
-import org.apache.spark.sql.internal.UserDefinedFunctionLike
-import org.apache.spark.sql.types.DataType
-import org.apache.spark.util.{ClosureCleaner, SparkClassUtils, SparkSerDeUtils}
-
-/**
- * A user-defined function. To create one, use the `udf` functions in `functions`.
- *
- * As an example:
- * {{{
- * // Define a UDF that returns true or false based on some numeric score.
- * val predict = udf((score: Double) => score > 0.5)
- *
- * // Projects a column that adds a prediction column based on the score column.
- * df.select( predict(df("score")) )
- * }}}
- *
- * @since 3.4.0
- */
-sealed abstract class UserDefinedFunction {
-
- /**
- * Returns true when the UDF can return a nullable value.
- *
- * @since 3.4.0
- */
- def nullable: Boolean
-
- /**
- * Returns true iff the UDF is deterministic, i.e. the UDF produces the same output given the
- * same input.
- *
- * @since 3.4.0
- */
- def deterministic: Boolean
-
- /**
- * Returns an expression that invokes the UDF, using the given arguments.
- *
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def apply(exprs: Column*): Column
-
- /**
- * Updates UserDefinedFunction with a given name.
- *
- * @since 3.4.0
- */
- def withName(name: String): UserDefinedFunction
-
- /**
- * Updates UserDefinedFunction to non-nullable.
- *
- * @since 3.4.0
- */
- def asNonNullable(): UserDefinedFunction
-
- /**
- * Updates UserDefinedFunction to nondeterministic.
- *
- * @since 3.4.0
- */
- def asNondeterministic(): UserDefinedFunction
-}
-
-/**
- * Holder class for a scala user-defined function and it's input/output encoder(s).
- */
-case class ScalaUserDefinedFunction private[sql] (
- // SPARK-43198: Eagerly serialize to prevent the UDF from containing a reference to this class.
- serializedUdfPacket: Array[Byte],
- inputTypes: Seq[proto.DataType],
- outputType: proto.DataType,
- givenName: Option[String],
- override val nullable: Boolean,
- override val deterministic: Boolean,
- aggregate: Boolean)
- extends UserDefinedFunction
- with UserDefinedFunctionLike {
-
- private[sql] lazy val udf = {
- val scalaUdfBuilder = proto.ScalarScalaUDF
- .newBuilder()
- .setPayload(ByteString.copyFrom(serializedUdfPacket))
- // Send the real inputs and return types to obtain the types without deser the udf bytes.
- .addAllInputTypes(inputTypes.asJava)
- .setOutputType(outputType)
- .setNullable(nullable)
- .setAggregate(aggregate)
-
- scalaUdfBuilder.build()
- }
-
- @scala.annotation.varargs
- override def apply(exprs: Column*): Column = Column { builder =>
- val udfBuilder = builder.getCommonInlineUserDefinedFunctionBuilder
- udfBuilder
- .setDeterministic(deterministic)
- .setScalarScalaUdf(udf)
- .addAllArguments(exprs.map(_.expr).asJava)
-
- givenName.foreach(udfBuilder.setFunctionName)
- }
-
- override def withName(name: String): ScalaUserDefinedFunction = copy(givenName = Option(name))
-
- override def asNonNullable(): ScalaUserDefinedFunction = copy(nullable = false)
-
- override def asNondeterministic(): ScalaUserDefinedFunction = copy(deterministic = false)
-
- def toProto: proto.CommonInlineUserDefinedFunction = {
- val builder = proto.CommonInlineUserDefinedFunction.newBuilder()
- builder
- .setDeterministic(deterministic)
- .setScalarScalaUdf(udf)
-
- givenName.foreach(builder.setFunctionName)
- builder.build()
- }
-
- override def name: String = givenName.getOrElse("UDF")
-}
-
-object ScalaUserDefinedFunction {
- private val LAMBDA_DESERIALIZATION_ERR_MSG: String =
- "cannot assign instance of java.lang.invoke.SerializedLambda to field"
-
- private def checkDeserializable(bytes: Array[Byte]): Unit = {
- try {
- SparkSerDeUtils.deserialize(bytes, SparkClassUtils.getContextOrSparkClassLoader)
- } catch {
- case e: ClassCastException if e.getMessage.contains(LAMBDA_DESERIALIZATION_ERR_MSG) =>
- throw new SparkException(
- "UDF cannot be executed on a Spark cluster: it cannot be deserialized. " +
- "This is very likely to be caused by the lambda function (the UDF) having a " +
- "self-reference. This is not supported by java serialization.")
- case NonFatal(e) =>
- throw new SparkException(
- "UDF cannot be executed on a Spark cluster: it cannot be deserialized.",
- e)
- }
- }
-
- private[sql] def apply(
- function: AnyRef,
- returnType: TypeTag[_],
- parameterTypes: TypeTag[_]*): ScalaUserDefinedFunction = {
-
- ScalaUserDefinedFunction(
- function = function,
- // Input can be a row because the input data schema can be found from the plan.
- inputEncoders =
- parameterTypes.map(tag => ScalaReflection.encoderForWithRowEncoderSupport(tag)),
- // Output cannot be a row as there is no good way to get the return data type.
- outputEncoder = ScalaReflection.encoderFor(returnType))
- }
-
- private[sql] def apply(
- function: AnyRef,
- inputEncoders: Seq[AgnosticEncoder[_]],
- outputEncoder: AgnosticEncoder[_],
- aggregate: Boolean = false): ScalaUserDefinedFunction = {
- SparkConnectClosureCleaner.clean(function)
- val udfPacketBytes =
- SparkSerDeUtils.serialize(UdfPacket(function, inputEncoders, outputEncoder))
- checkDeserializable(udfPacketBytes)
- ScalaUserDefinedFunction(
- serializedUdfPacket = udfPacketBytes,
- inputTypes = inputEncoders.map(_.dataType).map(DataTypeProtoConverter.toConnectProtoType),
- outputType = DataTypeProtoConverter.toConnectProtoType(outputEncoder.dataType),
- givenName = None,
- nullable = true,
- deterministic = true,
- aggregate = aggregate)
- }
-
- private[sql] def apply(function: AnyRef, returnType: DataType): ScalaUserDefinedFunction = {
- ScalaUserDefinedFunction(
- function = function,
- inputEncoders = Seq.empty[AgnosticEncoder[_]],
- outputEncoder = RowEncoder.encoderForDataType(returnType, lenient = false))
- }
-}
-
-private object SparkConnectClosureCleaner {
- def clean(closure: AnyRef): Unit = {
- ClosureCleaner.clean(closure, cleanTransitively = true, mutable.Map.empty)
- }
-}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Window.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Window.scala
deleted file mode 100644
index c85e7bc9c5c0..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/Window.scala
+++ /dev/null
@@ -1,241 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.expressions
-
-import org.apache.spark.annotation.Stable
-import org.apache.spark.sql.Column
-
-/**
- * Utility functions for defining window in DataFrames.
- *
- * {{{
- * // PARTITION BY country ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
- * Window.partitionBy("country").orderBy("date")
- * .rowsBetween(Window.unboundedPreceding, Window.currentRow)
- *
- * // PARTITION BY country ORDER BY date ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING
- * Window.partitionBy("country").orderBy("date").rowsBetween(-3, 3)
- * }}}
- *
- * @note
- * When ordering is not defined, an unbounded window frame (rowFrame, unboundedPreceding,
- * unboundedFollowing) is used by default. When ordering is defined, a growing window frame
- * (rangeFrame, unboundedPreceding, currentRow) is used by default.
- *
- * @since 3.4.0
- */
-@Stable
-object Window {
-
- /**
- * Creates a [[WindowSpec]] with the partitioning defined.
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def partitionBy(colName: String, colNames: String*): WindowSpec = {
- spec.partitionBy(colName, colNames: _*)
- }
-
- /**
- * Creates a [[WindowSpec]] with the partitioning defined.
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def partitionBy(cols: Column*): WindowSpec = {
- spec.partitionBy(cols: _*)
- }
-
- /**
- * Creates a [[WindowSpec]] with the ordering defined.
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def orderBy(colName: String, colNames: String*): WindowSpec = {
- spec.orderBy(colName, colNames: _*)
- }
-
- /**
- * Creates a [[WindowSpec]] with the ordering defined.
- * @since 1.4.0
- */
- @scala.annotation.varargs
- def orderBy(cols: Column*): WindowSpec = {
- spec.orderBy(cols: _*)
- }
-
- /**
- * Value representing the first row in the partition, equivalent to "UNBOUNDED PRECEDING" in
- * SQL. This can be used to specify the frame boundaries:
- *
- * {{{
- * Window.rowsBetween(Window.unboundedPreceding, Window.currentRow)
- * }}}
- *
- * @since 3.4.0
- */
- def unboundedPreceding: Long = Long.MinValue
-
- /**
- * Value representing the last row in the partition, equivalent to "UNBOUNDED FOLLOWING" in SQL.
- * This can be used to specify the frame boundaries:
- *
- * {{{
- * Window.rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)
- * }}}
- *
- * @since 3.4.0
- */
- def unboundedFollowing: Long = Long.MaxValue
-
- /**
- * Value representing the current row. This can be used to specify the frame boundaries:
- *
- * {{{
- * Window.rowsBetween(Window.unboundedPreceding, Window.currentRow)
- * }}}
- *
- * @since 3.4.0
- */
- def currentRow: Long = 0
-
- /**
- * Creates a [[WindowSpec]] with the frame boundaries defined, from `start` (inclusive) to `end`
- * (inclusive).
- *
- * Both `start` and `end` are positions relative to the current row. For example, "0" means
- * "current row", while "-1" means the row before the current row, and "5" means the fifth row
- * after the current row.
- *
- * We recommend users use `Window.unboundedPreceding`, `Window.unboundedFollowing`, and
- * `Window.currentRow` to specify special boundary values, rather than using integral values
- * directly.
- *
- * A row based boundary is based on the position of the row within the partition. An offset
- * indicates the number of rows above or below the current row, the frame for the current row
- * starts or ends. For instance, given a row based sliding frame with a lower bound offset of -1
- * and a upper bound offset of +2. The frame for row with index 5 would range from index 4 to
- * index 7.
- *
- * {{{
- * import org.apache.spark.sql.expressions.Window
- * val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b"))
- * .toDF("id", "category")
- * val byCategoryOrderedById =
- * Window.partitionBy($"category").orderBy($"id").rowsBetween(Window.currentRow, 1)
- * df.withColumn("sum", sum($"id") over byCategoryOrderedById).show()
- *
- * +---+--------+---+
- * | id|category|sum|
- * +---+--------+---+
- * | 1| b| 3|
- * | 2| b| 5|
- * | 3| b| 3|
- * | 1| a| 2|
- * | 1| a| 3|
- * | 2| a| 2|
- * +---+--------+---+
- * }}}
- *
- * @param start
- * boundary start, inclusive. The frame is unbounded if this is the minimum long value
- * (`Window.unboundedPreceding`).
- * @param end
- * boundary end, inclusive. The frame is unbounded if this is the maximum long value
- * (`Window.unboundedFollowing`).
- * @since 3.4.0
- */
- // Note: when updating the doc for this method, also update WindowSpec.rowsBetween.
- def rowsBetween(start: Long, end: Long): WindowSpec = {
- spec.rowsBetween(start, end)
- }
-
- /**
- * Creates a [[WindowSpec]] with the frame boundaries defined, from `start` (inclusive) to `end`
- * (inclusive).
- *
- * Both `start` and `end` are relative to the current row. For example, "0" means "current row",
- * while "-1" means one off before the current row, and "5" means the five off after the current
- * row.
- *
- * We recommend users use `Window.unboundedPreceding`, `Window.unboundedFollowing`, and
- * `Window.currentRow` to specify special boundary values, rather than using long values
- * directly.
- *
- * A range-based boundary is based on the actual value of the ORDER BY expression(s). An offset
- * is used to alter the value of the ORDER BY expression, for instance if the current ORDER BY
- * expression has a value of 10 and the lower bound offset is -3, the resulting lower bound for
- * the current row will be 10 - 3 = 7. This however puts a number of constraints on the ORDER BY
- * expressions: there can be only one expression and this expression must have a numerical data
- * type. An exception can be made when the offset is unbounded, because no value modification is
- * needed, in this case multiple and non-numeric ORDER BY expression are allowed.
- *
- * {{{
- * import org.apache.spark.sql.expressions.Window
- * val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b"))
- * .toDF("id", "category")
- * val byCategoryOrderedById =
- * Window.partitionBy($"category").orderBy($"id").rangeBetween(Window.currentRow, 1)
- * df.withColumn("sum", sum($"id") over byCategoryOrderedById).show()
- *
- * +---+--------+---+
- * | id|category|sum|
- * +---+--------+---+
- * | 1| b| 3|
- * | 2| b| 5|
- * | 3| b| 3|
- * | 1| a| 4|
- * | 1| a| 4|
- * | 2| a| 2|
- * +---+--------+---+
- * }}}
- *
- * @param start
- * boundary start, inclusive. The frame is unbounded if this is the minimum long value
- * (`Window.unboundedPreceding`).
- * @param end
- * boundary end, inclusive. The frame is unbounded if this is the maximum long value
- * (`Window.unboundedFollowing`).
- * @since 3.4.0
- */
- // Note: when updating the doc for this method, also update WindowSpec.rangeBetween.
- def rangeBetween(start: Long, end: Long): WindowSpec = {
- spec.rangeBetween(start, end)
- }
-
- private[sql] def spec: WindowSpec = {
- new WindowSpec(Seq.empty, Seq.empty, None)
- }
-
-}
-
-/**
- * Utility functions for defining window in DataFrames.
- *
- * {{{
- * // PARTITION BY country ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
- * Window.partitionBy("country").orderBy("date")
- * .rowsBetween(Window.unboundedPreceding, Window.currentRow)
- *
- * // PARTITION BY country ORDER BY date ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING
- * Window.partitionBy("country").orderBy("date").rowsBetween(-3, 3)
- * }}}
- *
- * @since 3.4.0
- */
-@Stable
-class Window private () // So we can see Window in JavaDoc.
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala
deleted file mode 100644
index 681eec17cf22..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala
+++ /dev/null
@@ -1,240 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.expressions
-
-import scala.jdk.CollectionConverters._
-
-import org.apache.spark.annotation.Stable
-import org.apache.spark.connect.proto
-import org.apache.spark.sql.Column
-
-/**
- * A window specification that defines the partitioning, ordering, and frame boundaries.
- *
- * Use the static methods in [[Window]] to create a [[WindowSpec]].
- *
- * @since 3.4.0
- */
-@Stable
-class WindowSpec private[sql] (
- partitionSpec: Seq[proto.Expression],
- orderSpec: Seq[proto.Expression.SortOrder],
- frame: Option[proto.Expression.Window.WindowFrame]) {
-
- /**
- * Defines the partitioning columns in a [[WindowSpec]].
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def partitionBy(colName: String, colNames: String*): WindowSpec = {
- partitionBy((colName +: colNames).map(Column(_)): _*)
- }
-
- /**
- * Defines the partitioning columns in a [[WindowSpec]].
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def partitionBy(cols: Column*): WindowSpec = {
- new WindowSpec(cols.map(_.expr), orderSpec, frame)
- }
-
- /**
- * Defines the ordering columns in a [[WindowSpec]].
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def orderBy(colName: String, colNames: String*): WindowSpec = {
- orderBy((colName +: colNames).map(Column(_)): _*)
- }
-
- /**
- * Defines the ordering columns in a [[WindowSpec]].
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def orderBy(cols: Column*): WindowSpec = {
- val sortOrder: Seq[proto.Expression.SortOrder] = cols.map(_.sortOrder)
- new WindowSpec(partitionSpec, sortOrder, frame)
- }
-
- /**
- * Defines the frame boundaries, from `start` (inclusive) to `end` (inclusive).
- *
- * Both `start` and `end` are relative positions from the current row. For example, "0" means
- * "current row", while "-1" means the row before the current row, and "5" means the fifth row
- * after the current row.
- *
- * We recommend users use `Window.unboundedPreceding`, `Window.unboundedFollowing`, and
- * `Window.currentRow` to specify special boundary values, rather than using integral values
- * directly.
- *
- * A row based boundary is based on the position of the row within the partition. An offset
- * indicates the number of rows above or below the current row, the frame for the current row
- * starts or ends. For instance, given a row based sliding frame with a lower bound offset of -1
- * and a upper bound offset of +2. The frame for row with index 5 would range from index 4 to
- * index 7.
- *
- * {{{
- * import org.apache.spark.sql.expressions.Window
- * val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b"))
- * .toDF("id", "category")
- * val byCategoryOrderedById =
- * Window.partitionBy($"category").orderBy($"id").rowsBetween(Window.currentRow, 1)
- * df.withColumn("sum", sum($"id") over byCategoryOrderedById).show()
- *
- * +---+--------+---+
- * | id|category|sum|
- * +---+--------+---+
- * | 1| b| 3|
- * | 2| b| 5|
- * | 3| b| 3|
- * | 1| a| 2|
- * | 1| a| 3|
- * | 2| a| 2|
- * +---+--------+---+
- * }}}
- *
- * @param start
- * boundary start, inclusive. The frame is unbounded if this is the minimum long value
- * (`Window.unboundedPreceding`).
- * @param end
- * boundary end, inclusive. The frame is unbounded if this is the maximum long value
- * (`Window.unboundedFollowing`).
- * @since 3.4.0
- */
- // Note: when updating the doc for this method, also update Window.rowsBetween.
- def rowsBetween(start: Long, end: Long): WindowSpec = {
- new WindowSpec(
- partitionSpec,
- orderSpec,
- Some(
- toWindowFrame(
- proto.Expression.Window.WindowFrame.FrameType.FRAME_TYPE_ROW,
- start,
- end,
- true)))
- }
-
- /**
- * Defines the frame boundaries, from `start` (inclusive) to `end` (inclusive).
- *
- * Both `start` and `end` are relative from the current row. For example, "0" means "current
- * row", while "-1" means one off before the current row, and "5" means the five off after the
- * current row.
- *
- * We recommend users use `Window.unboundedPreceding`, `Window.unboundedFollowing`, and
- * `Window.currentRow` to specify special boundary values, rather than using long values
- * directly.
- *
- * A range-based boundary is based on the actual value of the ORDER BY expression(s). An offset
- * is used to alter the value of the ORDER BY expression, for instance if the current order by
- * expression has a value of 10 and the lower bound offset is -3, the resulting lower bound for
- * the current row will be 10 - 3 = 7. This however puts a number of constraints on the ORDER BY
- * expressions: there can be only one expression and this expression must have a numerical data
- * type. An exception can be made when the offset is unbounded, because no value modification is
- * needed, in this case multiple and non-numeric ORDER BY expression are allowed.
- *
- * {{{
- * import org.apache.spark.sql.expressions.Window
- * val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b"))
- * .toDF("id", "category")
- * val byCategoryOrderedById =
- * Window.partitionBy($"category").orderBy($"id").rangeBetween(Window.currentRow, 1)
- * df.withColumn("sum", sum($"id") over byCategoryOrderedById).show()
- *
- * +---+--------+---+
- * | id|category|sum|
- * +---+--------+---+
- * | 1| b| 3|
- * | 2| b| 5|
- * | 3| b| 3|
- * | 1| a| 4|
- * | 1| a| 4|
- * | 2| a| 2|
- * +---+--------+---+
- * }}}
- *
- * @param start
- * boundary start, inclusive. The frame is unbounded if this is the minimum long value
- * (`Window.unboundedPreceding`).
- * @param end
- * boundary end, inclusive. The frame is unbounded if this is the maximum long value
- * (`Window.unboundedFollowing`).
- * @since 3.4.0
- */
- // Note: when updating the doc for this method, also update Window.rangeBetween.
- def rangeBetween(start: Long, end: Long): WindowSpec = {
- new WindowSpec(
- partitionSpec,
- orderSpec,
- Some(
- toWindowFrame(
- proto.Expression.Window.WindowFrame.FrameType.FRAME_TYPE_RANGE,
- start,
- end,
- false)))
- }
-
- /**
- * Converts this [[WindowSpec]] into a [[Column]] with an aggregate expression.
- */
- private[sql] def withAggregate(aggregate: Column): Column = {
- Column { builder =>
- val windowBuilder = builder.getWindowBuilder
- windowBuilder.setWindowFunction(aggregate.expr)
- if (frame.isDefined) {
- windowBuilder.setFrameSpec(frame.get)
- }
- windowBuilder.addAllPartitionSpec(partitionSpec.asJava)
- windowBuilder.addAllOrderSpec(orderSpec.asJava)
- }
- }
-
- private[sql] def toWindowFrame(
- frameType: proto.Expression.Window.WindowFrame.FrameType,
- start: Long,
- end: Long,
- isRowBetween: Boolean): proto.Expression.Window.WindowFrame = {
- val windowFrameBuilder = proto.Expression.Window.WindowFrame.newBuilder()
- windowFrameBuilder.setFrameType(frameType)
- start match {
- case 0 => windowFrameBuilder.getLowerBuilder.setCurrentRow(true)
- case Long.MinValue => windowFrameBuilder.getLowerBuilder.setUnbounded(true)
- case x if isRowBetween && Int.MinValue <= x && x <= Int.MaxValue =>
- windowFrameBuilder.getLowerBuilder.getValueBuilder.getLiteralBuilder
- .setInteger(start.toInt)
- case _ if !isRowBetween =>
- windowFrameBuilder.getLowerBuilder.getValueBuilder.getLiteralBuilder.setLong(start)
- case _ => throw new UnsupportedOperationException()
- }
-
- end match {
- case 0 => windowFrameBuilder.getUpperBuilder.setCurrentRow(true)
- case Long.MaxValue => windowFrameBuilder.getUpperBuilder.setUnbounded(true)
- case x if isRowBetween && Int.MinValue <= x && x <= Int.MaxValue =>
- windowFrameBuilder.getUpperBuilder.getValueBuilder.getLiteralBuilder
- .setInteger(end.toInt)
- case _ if !isRowBetween =>
- windowFrameBuilder.getUpperBuilder.getValueBuilder.getLiteralBuilder.setLong(end)
- case _ => throw new UnsupportedOperationException()
- }
-
- windowFrameBuilder.build()
- }
-}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala
deleted file mode 100644
index 3b6675362d55..000000000000
--- a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala
+++ /dev/null
@@ -1,8754 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.spark.sql
-
-import java.util.Collections
-
-import scala.jdk.CollectionConverters._
-import scala.reflect.runtime.universe.{typeTag, TypeTag}
-
-import org.apache.spark.connect.proto
-import org.apache.spark.sql.api.java._
-import org.apache.spark.sql.catalyst.ScalaReflection
-import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.PrimitiveLongEncoder
-import org.apache.spark.sql.connect.common.LiteralValueProtoConverter._
-import org.apache.spark.sql.connect.common.UdfUtils
-import org.apache.spark.sql.errors.DataTypeErrors
-import org.apache.spark.sql.expressions.{Aggregator, ScalaUserDefinedFunction, UserDefinedFunction}
-import org.apache.spark.sql.types.{DataType, StructType}
-import org.apache.spark.sql.types.DataType.parseTypeWithFallback
-import org.apache.spark.util.SparkClassUtils
-
-/**
- * Commonly used functions available for DataFrame operations. Using functions defined here
- * provides a little bit more compile-time safety to make sure the function exists.
- *
- * Spark also includes more built-in functions that are less common and are not defined here. You
- * can still access them (and all the functions defined here) using the `functions.expr()` API and
- * calling them through a SQL expression string. You can find the entire list of functions at SQL
- * API documentation of your Spark version, see also <a
- * href="https://spark.apache.org/docs/latest/api/sql/index.html">the latest list</a>
- *
- * As an example, `isnan` is a function that is defined here. You can use `isnan(col("myCol"))` to
- * invoke the `isnan` function. This way the programming language's compiler ensures `isnan`
- * exists and is of the proper form. You can also use `expr("isnan(myCol)")` function to invoke
- * the same function. In this case, Spark itself will ensure `isnan` exists when it analyzes the
- * query.
- *
- * `regr_count` is an example of a function that is built-in but not defined here, because it is
- * less commonly used. To invoke it, use `expr("regr_count(yCol, xCol)")`.
- *
- * This function APIs usually have methods with `Column` signature only because it can support not
- * only `Column` but also other types such as a native string. The other variants currently exist
- * for historical reasons.
- *
- * @groupname udf_funcs UDF, UDAF and UDT
- * @groupname agg_funcs Aggregate functions
- * @groupname datetime_funcs Date and Timestamp functions
- * @groupname sort_funcs Sort functions
- * @groupname normal_funcs Normal functions
- * @groupname math_funcs Mathematical functions
- * @groupname bitwise_funcs Bitwise functions
- * @groupname predicate_funcs Predicate functions
- * @groupname conditional_funcs Conditional functions
- * @groupname hash_funcs Hash functions
- * @groupname misc_funcs Misc functions
- * @groupname window_funcs Window functions
- * @groupname generator_funcs Generator functions
- * @groupname string_funcs String functions
- * @groupname collection_funcs Collection functions
- * @groupname array_funcs Array functions
- * @groupname map_funcs Map functions
- * @groupname struct_funcs Struct functions
- * @groupname csv_funcs CSV functions
- * @groupname json_funcs JSON functions
- * @groupname variant_funcs VARIANT functions
- * @groupname xml_funcs XML functions
- * @groupname url_funcs URL functions
- * @groupname partition_transforms Partition transform functions
- * @groupname Ungrouped Support functions for DataFrames
- *
- * @since 3.4.0
- */
-// scalastyle:off
-object functions {
-// scalastyle:on
-
- /**
- * Returns a [[Column]] based on the given column name.
- *
- * @group normal_funcs
- * @since 3.4.0
- */
- def col(colName: String): Column = Column(colName)
-
- /**
- * Returns a [[Column]] based on the given column name. Alias of [[col]].
- *
- * @group normal_funcs
- * @since 3.4.0
- */
- def column(colName: String): Column = col(colName)
-
- private def createLiteral(literalBuilder: proto.Expression.Literal.Builder): Column = Column {
- builder => builder.setLiteral(literalBuilder)
- }
-
- /**
- * Creates a [[Column]] of literal value.
- *
- * The passed in object is returned directly if it is already a [[Column]]. If the object is a
- * Scala Symbol, it is converted into a [[Column]] also. Otherwise, a new [[Column]] is created
- * to represent the literal value.
- *
- * @group normal_funcs
- * @since 3.4.0
- */
- def lit(literal: Any): Column = {
- literal match {
- case c: Column => c
- case s: Symbol => Column(s.name)
- case _ => createLiteral(toLiteralProtoBuilder(literal))
- }
- }
-
- /**
- * Creates a [[Column]] of literal value.
- *
- * An alias of `typedlit`, and it is encouraged to use `typedlit` directly.
- *
- * @group normal_funcs
- * @since 3.5.0
- */
- def typedLit[T: TypeTag](literal: T): Column = typedlit(literal)
-
- /**
- * Creates a [[Column]] of literal value.
- *
- * The passed in object is returned directly if it is already a [[Column]]. If the object is a
- * Scala Symbol, it is converted into a [[Column]] also. Otherwise, a new [[Column]] is created
- * to represent the literal value. The difference between this function and [[lit]] is that this
- * function can handle parameterized scala types e.g.: List, Seq and Map.
- *
- * @note
- * `typedlit` will call expensive Scala reflection APIs. `lit` is preferred if parameterized
- * Scala types are not used.
- *
- * @group normal_funcs
- * @since 3.5.0
- */
- def typedlit[T: TypeTag](literal: T): Column = literal match {
- case c: Column => c
- case s: Symbol => new Column(s.name)
- case _ => createLiteral(create(literal))
- }
-
- /**
- * Creates a struct with the given field names and values.
- *
- * @group struct_funcs
- * @since 3.5.0
- */
- def named_struct(cols: Column*): Column = Column.fn("named_struct", cols: _*)
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Sort functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Returns a sort expression based on ascending order of the column.
- * {{{
- * df.sort(asc("dept"), desc("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def asc(columnName: String): Column = Column(columnName).asc
-
- /**
- * Returns a sort expression based on ascending order of the column, and null values return
- * before non-null values.
- * {{{
- * df.sort(asc_nulls_first("dept"), desc("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def asc_nulls_first(columnName: String): Column = Column(columnName).asc_nulls_first
-
- /**
- * Returns a sort expression based on ascending order of the column, and null values appear
- * after non-null values.
- * {{{
- * df.sort(asc_nulls_last("dept"), desc("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def asc_nulls_last(columnName: String): Column = Column(columnName).asc_nulls_last
-
- /**
- * Returns a sort expression based on the descending order of the column.
- * {{{
- * df.sort(asc("dept"), desc("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def desc(columnName: String): Column = Column(columnName).desc
-
- /**
- * Returns a sort expression based on the descending order of the column, and null values appear
- * before non-null values.
- * {{{
- * df.sort(asc("dept"), desc_nulls_first("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def desc_nulls_first(columnName: String): Column = Column(columnName).desc_nulls_first
-
- /**
- * Returns a sort expression based on the descending order of the column, and null values appear
- * after non-null values.
- * {{{
- * df.sort(asc("dept"), desc_nulls_last("age"))
- * }}}
- *
- * @group sort_funcs
- * @since 3.4.0
- */
- def desc_nulls_last(columnName: String): Column = Column(columnName).desc_nulls_last
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Aggregate functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use approx_count_distinct", "2.1.0")
- def approxCountDistinct(e: Column): Column = approx_count_distinct(e)
-
- /**
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use approx_count_distinct", "2.1.0")
- def approxCountDistinct(columnName: String): Column = approx_count_distinct(columnName)
-
- /**
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use approx_count_distinct", "2.1.0")
- def approxCountDistinct(e: Column, rsd: Double): Column = approx_count_distinct(e, rsd)
-
- /**
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use approx_count_distinct", "2.1.0")
- def approxCountDistinct(columnName: String, rsd: Double): Column = {
- approx_count_distinct(Column(columnName), rsd)
- }
-
- /**
- * Aggregate function: returns the approximate number of distinct items in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def approx_count_distinct(e: Column): Column = Column.fn("approx_count_distinct", e)
-
- /**
- * Aggregate function: returns the approximate number of distinct items in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def approx_count_distinct(columnName: String): Column = approx_count_distinct(
- column(columnName))
-
- /**
- * Aggregate function: returns the approximate number of distinct items in a group.
- *
- * @param rsd
- * maximum relative standard deviation allowed (default = 0.05)
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def approx_count_distinct(e: Column, rsd: Double): Column = {
- Column.fn("approx_count_distinct", e, lit(rsd))
- }
-
- /**
- * Aggregate function: returns the approximate number of distinct items in a group.
- *
- * @param rsd
- * maximum relative standard deviation allowed (default = 0.05)
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def approx_count_distinct(columnName: String, rsd: Double): Column = {
- approx_count_distinct(Column(columnName), rsd)
- }
-
- /**
- * Aggregate function: returns the average of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def avg(e: Column): Column = Column.fn("avg", e)
-
- /**
- * Aggregate function: returns the average of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def avg(columnName: String): Column = avg(Column(columnName))
-
- /**
- * Aggregate function: returns a list of objects with duplicates.
- *
- * @note
- * The function is non-deterministic because the order of collected results depends on the
- * order of the rows which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def collect_list(e: Column): Column = Column.fn("collect_list", e)
-
- /**
- * Aggregate function: returns a list of objects with duplicates.
- *
- * @note
- * The function is non-deterministic because the order of collected results depends on the
- * order of the rows which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def collect_list(columnName: String): Column = collect_list(Column(columnName))
-
- /**
- * Aggregate function: returns a set of objects with duplicate elements eliminated.
- *
- * @note
- * The function is non-deterministic because the order of collected results depends on the
- * order of the rows which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def collect_set(e: Column): Column = Column.fn("collect_set", e)
-
- /**
- * Aggregate function: returns a set of objects with duplicate elements eliminated.
- *
- * @note
- * The function is non-deterministic because the order of collected results depends on the
- * order of the rows which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def collect_set(columnName: String): Column = collect_set(Column(columnName))
-
- /**
- * Aggregate function: returns the Pearson Correlation Coefficient for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def corr(column1: Column, column2: Column): Column = Column.fn("corr", column1, column2)
-
- /**
- * Aggregate function: returns the Pearson Correlation Coefficient for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def corr(columnName1: String, columnName2: String): Column = {
- corr(Column(columnName1), Column(columnName2))
- }
-
- /**
- * Aggregate function: returns the number of items in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def count(e: Column): Column = {
- val withoutStar = e.expr.getExprTypeCase match {
- // Turn count(*) into count(1)
- case proto.Expression.ExprTypeCase.UNRESOLVED_STAR => lit(1)
- case _ => e
- }
- Column.fn("count", withoutStar)
- }
-
- /**
- * Aggregate function: returns the number of items in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def count(columnName: String): TypedColumn[Any, Long] =
- count(Column(columnName)).as(PrimitiveLongEncoder)
-
- /**
- * Aggregate function: returns the number of distinct items in a group.
- *
- * An alias of `count_distinct`, and it is encouraged to use `count_distinct` directly.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def countDistinct(expr: Column, exprs: Column*): Column = count_distinct(expr, exprs: _*)
-
- /**
- * Aggregate function: returns the number of distinct items in a group.
- *
- * An alias of `count_distinct`, and it is encouraged to use `count_distinct` directly.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def countDistinct(columnName: String, columnNames: String*): Column =
- count_distinct(Column(columnName), columnNames.map(Column.apply): _*)
-
- /**
- * Aggregate function: returns the number of distinct items in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def count_distinct(expr: Column, exprs: Column*): Column =
- Column.fn("count", isDistinct = true, expr +: exprs: _*)
-
- /**
- * Aggregate function: returns the population covariance for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def covar_pop(column1: Column, column2: Column): Column =
- Column.fn("covar_pop", column1, column2)
-
- /**
- * Aggregate function: returns the population covariance for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def covar_pop(columnName1: String, columnName2: String): Column = {
- covar_pop(Column(columnName1), Column(columnName2))
- }
-
- /**
- * Aggregate function: returns the sample covariance for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def covar_samp(column1: Column, column2: Column): Column =
- Column.fn("covar_samp", column1, column2)
-
- /**
- * Aggregate function: returns the sample covariance for two columns.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def covar_samp(columnName1: String, columnName2: String): Column =
- covar_samp(Column(columnName1), Column(columnName2))
-
- /**
- * Aggregate function: returns the first value in a group.
- *
- * The function by default returns the first values it sees. It will return the first non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def first(e: Column, ignoreNulls: Boolean): Column =
- Column.fn("first", e, lit(ignoreNulls))
-
- /**
- * Aggregate function: returns the first value of a column in a group.
- *
- * The function by default returns the first values it sees. It will return the first non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def first(columnName: String, ignoreNulls: Boolean): Column = {
- first(Column(columnName), ignoreNulls)
- }
-
- /**
- * Aggregate function: returns the first value in a group.
- *
- * The function by default returns the first values it sees. It will return the first non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def first(e: Column): Column = first(e, ignoreNulls = false)
-
- /**
- * Aggregate function: returns the first value of a column in a group.
- *
- * The function by default returns the first values it sees. It will return the first non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def first(columnName: String): Column = first(Column(columnName))
-
- /**
- * Aggregate function: returns the first value in a group.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def first_value(e: Column): Column = Column.fn("first_value", e)
-
- /**
- * Aggregate function: returns the first value in a group.
- *
- * The function by default returns the first values it sees. It will return the first non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def first_value(e: Column, ignoreNulls: Column): Column =
- Column.fn("first_value", e, ignoreNulls)
-
- /**
- * Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or
- * not, returns 1 for aggregated or 0 for not aggregated in the result set.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def grouping(e: Column): Column = Column.fn("grouping", e)
-
- /**
- * Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or
- * not, returns 1 for aggregated or 0 for not aggregated in the result set.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def grouping(columnName: String): Column = grouping(Column(columnName))
-
- /**
- * Aggregate function: returns the level of grouping, equals to
- *
- * {{{
- * (grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
- * }}}
- *
- * @note
- * The list of columns should match with grouping columns exactly, or empty (means all the
- * grouping columns).
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def grouping_id(cols: Column*): Column = Column.fn("grouping_id", cols: _*)
-
- /**
- * Aggregate function: returns the level of grouping, equals to
- *
- * {{{
- * (grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
- * }}}
- *
- * @note
- * The list of columns should match with grouping columns exactly.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def grouping_id(colName: String, colNames: String*): Column =
- grouping_id((Seq(colName) ++ colNames).map(n => Column(n)): _*)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches HllSketch
- * configured with lgConfigK arg.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_sketch_agg(e: Column, lgConfigK: Column): Column =
- Column.fn("hll_sketch_agg", e, lgConfigK)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches HllSketch
- * configured with lgConfigK arg.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_sketch_agg(e: Column, lgConfigK: Int): Column =
- Column.fn("hll_sketch_agg", e, lit(lgConfigK))
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches HllSketch
- * configured with lgConfigK arg.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_sketch_agg(columnName: String, lgConfigK: Int): Column =
- hll_sketch_agg(Column(columnName), lgConfigK)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches HllSketch
- * configured with default lgConfigK value.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_sketch_agg(e: Column): Column =
- Column.fn("hll_sketch_agg", e)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches HllSketch
- * configured with default lgConfigK value.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_sketch_agg(columnName: String): Column = hll_sketch_agg(Column(columnName))
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches
- * HllSketch, generated by merging previously created Datasketches HllSketch instances via a
- * Datasketches Union instance. Throws an exception if sketches have different lgConfigK values
- * and allowDifferentLgConfigK is set to false.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_union_agg(e: Column, allowDifferentLgConfigK: Column): Column =
- Column.fn("hll_union_agg", e, allowDifferentLgConfigK)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches
- * HllSketch, generated by merging previously created Datasketches HllSketch instances via a
- * Datasketches Union instance. Throws an exception if sketches have different lgConfigK values
- * and allowDifferentLgConfigK is set to false.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_union_agg(e: Column, allowDifferentLgConfigK: Boolean): Column =
- Column.fn("hll_union_agg", e, lit(allowDifferentLgConfigK))
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches
- * HllSketch, generated by merging previously created Datasketches HllSketch instances via a
- * Datasketches Union instance. Throws an exception if sketches have different lgConfigK values
- * and allowDifferentLgConfigK is set to false.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_union_agg(columnName: String, allowDifferentLgConfigK: Boolean): Column =
- hll_union_agg(Column(columnName), allowDifferentLgConfigK)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches
- * HllSketch, generated by merging previously created Datasketches HllSketch instances via a
- * Datasketches Union instance. Throws an exception if sketches have different lgConfigK values.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_union_agg(e: Column): Column =
- Column.fn("hll_union_agg", e)
-
- /**
- * Aggregate function: returns the updatable binary representation of the Datasketches
- * HllSketch, generated by merging previously created Datasketches HllSketch instances via a
- * Datasketches Union instance. Throws an exception if sketches have different lgConfigK values.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def hll_union_agg(columnName: String): Column = hll_union_agg(Column(columnName))
-
- /**
- * Aggregate function: returns the kurtosis of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def kurtosis(e: Column): Column = Column.fn("kurtosis", e)
-
- /**
- * Aggregate function: returns the kurtosis of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def kurtosis(columnName: String): Column = kurtosis(Column(columnName))
-
- /**
- * Aggregate function: returns the last value in a group.
- *
- * The function by default returns the last values it sees. It will return the last non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def last(e: Column, ignoreNulls: Boolean): Column =
- Column.fn("last", e, lit(ignoreNulls))
-
- /**
- * Aggregate function: returns the last value of the column in a group.
- *
- * The function by default returns the last values it sees. It will return the last non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def last(columnName: String, ignoreNulls: Boolean): Column =
- last(Column(columnName), ignoreNulls)
-
- /**
- * Aggregate function: returns the last value in a group.
- *
- * The function by default returns the last values it sees. It will return the last non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def last(e: Column): Column = last(e, ignoreNulls = false)
-
- /**
- * Aggregate function: returns the last value of the column in a group.
- *
- * The function by default returns the last values it sees. It will return the last non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def last(columnName: String): Column = last(Column(columnName), ignoreNulls = false)
-
- /**
- * Aggregate function: returns the last value in a group.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def last_value(e: Column): Column = Column.fn("last_value", e)
-
- /**
- * Aggregate function: returns the last value in a group.
- *
- * The function by default returns the last values it sees. It will return the last non-null
- * value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- *
- * @note
- * The function is non-deterministic because its results depends on the order of the rows
- * which may be non-deterministic after a shuffle.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def last_value(e: Column, ignoreNulls: Column): Column =
- Column.fn("last_value", e, ignoreNulls)
-
- /**
- * Aggregate function: returns the most frequent value in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def mode(e: Column): Column = mode(e, deterministic = false)
-
- /**
- * Aggregate function: returns the most frequent value in a group.
- *
- * When multiple values have the same greatest frequency then either any of values is returned
- * if deterministic is false or is not defined, or the lowest value is returned if deterministic
- * is true.
- *
- * @group agg_funcs
- * @since 4.0.0
- */
- def mode(e: Column, deterministic: Boolean): Column = Column.fn("mode", e, lit(deterministic))
-
- /**
- * Aggregate function: returns the maximum value of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def max(e: Column): Column = Column.fn("max", e)
-
- /**
- * Aggregate function: returns the maximum value of the column in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def max(columnName: String): Column = max(Column(columnName))
-
- /**
- * Aggregate function: returns the value associated with the maximum value of ord.
- *
- * @note
- * The function is non-deterministic so the output order can be different for those associated
- * the same values of `e`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def max_by(e: Column, ord: Column): Column = Column.fn("max_by", e, ord)
-
- /**
- * Aggregate function: returns the average of the values in a group. Alias for avg.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def mean(e: Column): Column = avg(e)
-
- /**
- * Aggregate function: returns the average of the values in a group. Alias for avg.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def mean(columnName: String): Column = avg(columnName)
-
- /**
- * Aggregate function: returns the median of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def median(e: Column): Column = Column.fn("median", e)
-
- /**
- * Aggregate function: returns the minimum value of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def min(e: Column): Column = Column.fn("min", e)
-
- /**
- * Aggregate function: returns the minimum value of the column in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def min(columnName: String): Column = min(Column(columnName))
-
- /**
- * Aggregate function: returns the value associated with the minimum value of ord.
- *
- * @note
- * The function is non-deterministic so the output order can be different for those associated
- * the same values of `e`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def min_by(e: Column, ord: Column): Column = Column.fn("min_by", e, ord)
-
- /**
- * Aggregate function: returns the exact percentile(s) of numeric column `expr` at the given
- * percentage(s) with value range in [0.0, 1.0].
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def percentile(e: Column, percentage: Column): Column = Column.fn("percentile", e, percentage)
-
- /**
- * Aggregate function: returns the exact percentile(s) of numeric column `expr` at the given
- * percentage(s) with value range in [0.0, 1.0].
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def percentile(e: Column, percentage: Column, frequency: Column): Column =
- Column.fn("percentile", e, percentage, frequency)
-
- /**
- * Aggregate function: returns the approximate `percentile` of the numeric column `col` which is
- * the smallest value in the ordered `col` values (sorted from least to greatest) such that no
- * more than `percentage` of `col` values is less than the value or equal to that value.
- *
- * If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating
- * point value, it must be between 0.0 and 1.0.
- *
- * The accuracy parameter is a positive numeric literal which controls approximation accuracy at
- * the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the
- * relative error of the approximation.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def percentile_approx(e: Column, percentage: Column, accuracy: Column): Column =
- Column.fn("percentile_approx", e, percentage, accuracy)
-
- /**
- * Aggregate function: returns the approximate `percentile` of the numeric column `col` which is
- * the smallest value in the ordered `col` values (sorted from least to greatest) such that no
- * more than `percentage` of `col` values is less than the value or equal to that value.
- *
- * If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating
- * point value, it must be between 0.0 and 1.0.
- *
- * The accuracy parameter is a positive numeric literal which controls approximation accuracy at
- * the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the
- * relative error of the approximation.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def approx_percentile(e: Column, percentage: Column, accuracy: Column): Column = {
- Column.fn("approx_percentile", e, percentage, accuracy)
- }
-
- /**
- * Aggregate function: returns the product of all numerical elements in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def product(e: Column): Column = Column.fn("product", e)
-
- /**
- * Aggregate function: returns the skewness of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def skewness(e: Column): Column = Column.fn("skewness", e)
-
- /**
- * Aggregate function: returns the skewness of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def skewness(columnName: String): Column = skewness(Column(columnName))
-
- /**
- * Aggregate function: alias for `stddev_samp`.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def std(e: Column): Column = Column.fn("std", e)
-
- /**
- * Aggregate function: alias for `stddev_samp`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev(e: Column): Column = Column.fn("stddev", e)
-
- /**
- * Aggregate function: alias for `stddev_samp`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev(columnName: String): Column = stddev(Column(columnName))
-
- /**
- * Aggregate function: returns the sample standard deviation of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev_samp(e: Column): Column = Column.fn("stddev_samp", e)
-
- /**
- * Aggregate function: returns the sample standard deviation of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev_samp(columnName: String): Column = stddev_samp(Column(columnName))
-
- /**
- * Aggregate function: returns the population standard deviation of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev_pop(e: Column): Column = Column.fn("stddev_pop", e)
-
- /**
- * Aggregate function: returns the population standard deviation of the expression in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def stddev_pop(columnName: String): Column = stddev_pop(Column(columnName))
-
- /**
- * Aggregate function: returns the sum of all values in the expression.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def sum(e: Column): Column = Column.fn("sum", e)
-
- /**
- * Aggregate function: returns the sum of all values in the given column.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def sum(columnName: String): Column = sum(Column(columnName))
-
- /**
- * Aggregate function: returns the sum of distinct values in the expression.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use sum_distinct", "3.2.0")
- def sumDistinct(e: Column): Column = sum_distinct(e)
-
- /**
- * Aggregate function: returns the sum of distinct values in the expression.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- @deprecated("Use sum_distinct", "3.2.0")
- def sumDistinct(columnName: String): Column = sum_distinct(Column(columnName))
-
- /**
- * Aggregate function: returns the sum of distinct values in the expression.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def sum_distinct(e: Column): Column = Column.fn("sum", isDistinct = true, e)
-
- /**
- * Aggregate function: alias for `var_samp`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def variance(e: Column): Column = Column.fn("variance", e)
-
- /**
- * Aggregate function: alias for `var_samp`.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def variance(columnName: String): Column = variance(Column(columnName))
-
- /**
- * Aggregate function: returns the unbiased variance of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def var_samp(e: Column): Column = Column.fn("var_samp", e)
-
- /**
- * Aggregate function: returns the unbiased variance of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def var_samp(columnName: String): Column = var_samp(Column(columnName))
-
- /**
- * Aggregate function: returns the population variance of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def var_pop(e: Column): Column = Column.fn("var_pop", e)
-
- /**
- * Aggregate function: returns the population variance of the values in a group.
- *
- * @group agg_funcs
- * @since 3.4.0
- */
- def var_pop(columnName: String): Column = var_pop(Column(columnName))
-
- /**
- * Aggregate function: returns the average of the independent variable for non-null pairs in a
- * group, where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_avgx(y: Column, x: Column): Column = Column.fn("regr_avgx", y, x)
-
- /**
- * Aggregate function: returns the average of the independent variable for non-null pairs in a
- * group, where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_avgy(y: Column, x: Column): Column = Column.fn("regr_avgy", y, x)
-
- /**
- * Aggregate function: returns the number of non-null number pairs in a group, where `y` is the
- * dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_count(y: Column, x: Column): Column = Column.fn("regr_count", y, x)
-
- /**
- * Aggregate function: returns the intercept of the univariate linear regression line for
- * non-null pairs in a group, where `y` is the dependent variable and `x` is the independent
- * variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_intercept(y: Column, x: Column): Column = Column.fn("regr_intercept", y, x)
-
- /**
- * Aggregate function: returns the coefficient of determination for non-null pairs in a group,
- * where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_r2(y: Column, x: Column): Column = Column.fn("regr_r2", y, x)
-
- /**
- * Aggregate function: returns the slope of the linear regression line for non-null pairs in a
- * group, where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_slope(y: Column, x: Column): Column = Column.fn("regr_slope", y, x)
-
- /**
- * Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group,
- * where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_sxx(y: Column, x: Column): Column = Column.fn("regr_sxx", y, x)
-
- /**
- * Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group,
- * where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_sxy(y: Column, x: Column): Column = Column.fn("regr_sxy", y, x)
-
- /**
- * Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group,
- * where `y` is the dependent variable and `x` is the independent variable.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def regr_syy(y: Column, x: Column): Column = Column.fn("regr_syy", y, x)
-
- /**
- * Aggregate function: returns some value of `e` for a group of rows.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def any_value(e: Column): Column = Column.fn("any_value", e)
-
- /**
- * Aggregate function: returns some value of `e` for a group of rows. If `isIgnoreNull` is true,
- * returns only non-null values.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def any_value(e: Column, ignoreNulls: Column): Column =
- Column.fn("any_value", e, ignoreNulls)
-
- /**
- * Aggregate function: returns the number of `TRUE` values for the expression.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def count_if(e: Column): Column = Column.fn("count_if", e)
-
- /**
- * Aggregate function: computes a histogram on numeric 'expr' using nb bins. The return value is
- * an array of (x,y) pairs representing the centers of the histogram's bins. As the value of
- * 'nb' is increased, the histogram approximation gets finer-grained, but may yield artifacts
- * around outliers. In practice, 20-40 histogram bins appear to work well, with more bins being
- * required for skewed or smaller datasets. Note that this function creates a histogram with
- * non-uniform bin widths. It offers no guarantees in terms of the mean-squared-error of the
- * histogram, but in practice is comparable to the histograms produced by the R/S-Plus
- * statistical computing packages. Note: the output type of the 'x' field in the return value is
- * propagated from the input value consumed in the aggregate function.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def histogram_numeric(e: Column, nBins: Column): Column =
- Column.fn("histogram_numeric", e, nBins)
-
- /**
- * Aggregate function: returns true if all values of `e` are true.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def every(e: Column): Column = Column.fn("every", e)
-
- /**
- * Aggregate function: returns true if all values of `e` are true.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bool_and(e: Column): Column = Column.fn("bool_and", e)
-
- /**
- * Aggregate function: returns true if at least one value of `e` is true.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def some(e: Column): Column = Column.fn("some", e)
-
- /**
- * Aggregate function: returns true if at least one value of `e` is true.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def any(e: Column): Column = Column.fn("any", e)
-
- /**
- * Aggregate function: returns true if at least one value of `e` is true.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bool_or(e: Column): Column = Column.fn("bool_or", e)
-
- /**
- * Aggregate function: returns the bitwise AND of all non-null input values, or null if none.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bit_and(e: Column): Column = Column.fn("bit_and", e)
-
- /**
- * Aggregate function: returns the bitwise OR of all non-null input values, or null if none.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bit_or(e: Column): Column = Column.fn("bit_or", e)
-
- /**
- * Aggregate function: returns the bitwise XOR of all non-null input values, or null if none.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bit_xor(e: Column): Column = Column.fn("bit_xor", e)
-
- /**
- * Aggregate function: returns a list of objects with duplicates.
- *
- * @note
- * The function is non-deterministic because the order of collected results depends on the
- * order of the rows which may be non-deterministic after a shuffle.
- * @group agg_funcs
- * @since 3.5.0
- */
- def array_agg(e: Column): Column = Column.fn("array_agg", e)
-
- /**
- * Returns a count-min sketch of a column with the given esp, confidence and seed. The result is
- * an array of bytes, which can be deserialized to a `CountMinSketch` before usage. Count-min
- * sketch is a probabilistic data structure used for cardinality estimation using sub-linear
- * space.
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def count_min_sketch(e: Column, eps: Column, confidence: Column, seed: Column): Column =
- Column.fn("count_min_sketch", e, eps, confidence, seed)
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Window functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Window function: returns the cumulative distribution of values within a window partition,
- * i.e. the fraction of rows that are below the current row.
- *
- * {{{
- * N = total number of rows in the partition
- * cumeDist(x) = number of values before (and including) x / N
- * }}}
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def cume_dist(): Column = Column.fn("cume_dist")
-
- /**
- * Window function: returns the rank of rows within a window partition, without any gaps.
- *
- * The difference between rank and dense_rank is that denseRank leaves no gaps in ranking
- * sequence when there are ties. That is, if you were ranking a competition using dense_rank and
- * had three people tie for second place, you would say that all three were in second place and
- * that the next person came in third. Rank would give me sequential numbers, making the person
- * that came in third place (after the ties) would register as coming in fifth.
- *
- * This is equivalent to the DENSE_RANK function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def dense_rank(): Column = Column.fn("dense_rank")
-
- /**
- * Window function: returns the value that is `offset` rows before the current row, and `null`
- * if there is less than `offset` rows before the current row. For example, an `offset` of one
- * will return the previous row at any given point in the window partition.
- *
- * This is equivalent to the LAG function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lag(e: Column, offset: Int): Column = lag(e, offset, null)
-
- /**
- * Window function: returns the value that is `offset` rows before the current row, and `null`
- * if there is less than `offset` rows before the current row. For example, an `offset` of one
- * will return the previous row at any given point in the window partition.
- *
- * This is equivalent to the LAG function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lag(columnName: String, offset: Int): Column = lag(columnName, offset, null)
-
- /**
- * Window function: returns the value that is `offset` rows before the current row, and
- * `defaultValue` if there is less than `offset` rows before the current row. For example, an
- * `offset` of one will return the previous row at any given point in the window partition.
- *
- * This is equivalent to the LAG function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lag(columnName: String, offset: Int, defaultValue: Any): Column = {
- lag(Column(columnName), offset, defaultValue)
- }
-
- /**
- * Window function: returns the value that is `offset` rows before the current row, and
- * `defaultValue` if there is less than `offset` rows before the current row. For example, an
- * `offset` of one will return the previous row at any given point in the window partition.
- *
- * This is equivalent to the LAG function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lag(e: Column, offset: Int, defaultValue: Any): Column = {
- lag(e, offset, defaultValue, ignoreNulls = false)
- }
-
- /**
- * Window function: returns the value that is `offset` rows before the current row, and
- * `defaultValue` if there is less than `offset` rows before the current row. `ignoreNulls`
- * determines whether null values of row are included in or eliminated from the calculation. For
- * example, an `offset` of one will return the previous row at any given point in the window
- * partition.
- *
- * This is equivalent to the LAG function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lag(e: Column, offset: Int, defaultValue: Any, ignoreNulls: Boolean): Column =
- Column.fn("lag", e, lit(offset), lit(defaultValue), lit(ignoreNulls))
-
- /**
- * Window function: returns the value that is `offset` rows after the current row, and `null` if
- * there is less than `offset` rows after the current row. For example, an `offset` of one will
- * return the next row at any given point in the window partition.
- *
- * This is equivalent to the LEAD function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lead(columnName: String, offset: Int): Column = {
- lead(columnName, offset, null)
- }
-
- /**
- * Window function: returns the value that is `offset` rows after the current row, and `null` if
- * there is less than `offset` rows after the current row. For example, an `offset` of one will
- * return the next row at any given point in the window partition.
- *
- * This is equivalent to the LEAD function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lead(e: Column, offset: Int): Column = {
- lead(e, offset, null)
- }
-
- /**
- * Window function: returns the value that is `offset` rows after the current row, and
- * `defaultValue` if there is less than `offset` rows after the current row. For example, an
- * `offset` of one will return the next row at any given point in the window partition.
- *
- * This is equivalent to the LEAD function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lead(columnName: String, offset: Int, defaultValue: Any): Column = {
- lead(Column(columnName), offset, defaultValue)
- }
-
- /**
- * Window function: returns the value that is `offset` rows after the current row, and
- * `defaultValue` if there is less than `offset` rows after the current row. For example, an
- * `offset` of one will return the next row at any given point in the window partition.
- *
- * This is equivalent to the LEAD function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lead(e: Column, offset: Int, defaultValue: Any): Column = {
- lead(e, offset, defaultValue, ignoreNulls = false)
- }
-
- /**
- * Window function: returns the value that is `offset` rows after the current row, and
- * `defaultValue` if there is less than `offset` rows after the current row. `ignoreNulls`
- * determines whether null values of row are included in or eliminated from the calculation. The
- * default value of `ignoreNulls` is false. For example, an `offset` of one will return the next
- * row at any given point in the window partition.
- *
- * This is equivalent to the LEAD function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def lead(e: Column, offset: Int, defaultValue: Any, ignoreNulls: Boolean): Column =
- Column.fn("lead", e, lit(offset), lit(defaultValue), lit(ignoreNulls))
-
- /**
- * Window function: returns the value that is the `offset`th row of the window frame (counting
- * from 1), and `null` if the size of window frame is less than `offset` rows.
- *
- * It will return the `offset`th non-null value it sees when ignoreNulls is set to true. If all
- * values are null, then null is returned.
- *
- * This is equivalent to the nth_value function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def nth_value(e: Column, offset: Int, ignoreNulls: Boolean): Column =
- Column.fn("nth_value", e, lit(offset), lit(ignoreNulls))
-
- /**
- * Window function: returns the value that is the `offset`th row of the window frame (counting
- * from 1), and `null` if the size of window frame is less than `offset` rows.
- *
- * This is equivalent to the nth_value function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def nth_value(e: Column, offset: Int): Column = nth_value(e, offset, false)
-
- /**
- * Window function: returns the ntile group id (from 1 to `n` inclusive) in an ordered window
- * partition. For example, if `n` is 4, the first quarter of the rows will get value 1, the
- * second quarter will get 2, the third quarter will get 3, and the last quarter will get 4.
- *
- * This is equivalent to the NTILE function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def ntile(n: Int): Column = Column.fn("ntile", lit(n))
-
- /**
- * Window function: returns the relative rank (i.e. percentile) of rows within a window
- * partition.
- *
- * This is computed by:
- * {{{
- * (rank of row in its partition - 1) / (number of rows in the partition - 1)
- * }}}
- *
- * This is equivalent to the PERCENT_RANK function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def percent_rank(): Column = Column.fn("percent_rank")
-
- /**
- * Window function: returns the rank of rows within a window partition.
- *
- * The difference between rank and dense_rank is that dense_rank leaves no gaps in ranking
- * sequence when there are ties. That is, if you were ranking a competition using dense_rank and
- * had three people tie for second place, you would say that all three were in second place and
- * that the next person came in third. Rank would give me sequential numbers, making the person
- * that came in third place (after the ties) would register as coming in fifth.
- *
- * This is equivalent to the RANK function in SQL.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def rank(): Column = Column.fn("rank")
-
- /**
- * Window function: returns a sequential number starting at 1 within a window partition.
- *
- * @group window_funcs
- * @since 3.4.0
- */
- def row_number(): Column = Column.fn("row_number")
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Non-aggregate functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Creates a new array column. The input columns must all have the same data type.
- *
- * @group array_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def array(cols: Column*): Column = Column.fn("array", cols: _*)
-
- /**
- * Creates a new array column. The input columns must all have the same data type.
- *
- * @group array_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def array(colName: String, colNames: String*): Column = {
- array((colName +: colNames).map(col): _*)
- }
-
- /**
- * Creates a new map column. The input columns must be grouped as key-value pairs, e.g. (key1,
- * value1, key2, value2, ...). The key columns must all have the same data type, and can't be
- * null. The value columns must all have the same data type.
- *
- * @group map_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def map(cols: Column*): Column = Column.fn("map", cols: _*)
-
- /**
- * Creates a new map column. The array in the first column is used for keys. The array in the
- * second column is used for values. All elements in the array for key should not be null.
- *
- * @group map_funcs
- * @since 3.4.0
- */
- def map_from_arrays(keys: Column, values: Column): Column =
- Column.fn("map_from_arrays", keys, values)
-
- /**
- * Creates a map after splitting the text into key/value pairs using delimiters. Both
- * `pairDelim` and `keyValueDelim` are treated as regular expressions.
- *
- * @group map_funcs
- * @since 3.5.0
- */
- def str_to_map(text: Column, pairDelim: Column, keyValueDelim: Column): Column =
- Column.fn("str_to_map", text, pairDelim, keyValueDelim)
-
- /**
- * Creates a map after splitting the text into key/value pairs using delimiters. The `pairDelim`
- * is treated as regular expressions.
- *
- * @group map_funcs
- * @since 3.5.0
- */
- def str_to_map(text: Column, pairDelim: Column): Column =
- Column.fn("str_to_map", text, pairDelim)
-
- /**
- * Creates a map after splitting the text into key/value pairs using delimiters.
- *
- * @group map_funcs
- * @since 3.5.0
- */
- def str_to_map(text: Column): Column = Column.fn("str_to_map", text)
-
- /**
- * Marks a DataFrame as small enough for use in broadcast joins.
- *
- * The following example marks the right DataFrame for broadcast hash join using `joinKey`.
- * {{{
- * // left and right are DataFrames
- * left.join(broadcast(right), "joinKey")
- * }}}
- *
- * @group normal_funcs
- * @since 3.4.0
- */
- def broadcast[T](df: Dataset[T]): Dataset[T] = {
- df.hint("broadcast")
- }
-
- /**
- * Returns the first column that is not null, or null if all inputs are null.
- *
- * For example, `coalesce(a, b, c)` will return a if a is not null, or b if a is null and b is
- * not null, or c if both a and b are null but c is not null.
- *
- * @group conditional_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def coalesce(e: Column*): Column = Column.fn("coalesce", e: _*)
-
- /**
- * Creates a string column for the file name of the current Spark task.
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def input_file_name(): Column = Column.fn("input_file_name")
-
- /**
- * Return true iff the column is NaN.
- *
- * @group predicate_funcs
- * @since 3.4.0
- */
- def isnan(e: Column): Column = e.isNaN
-
- /**
- * Return true iff the column is null.
- *
- * @group predicate_funcs
- * @since 3.4.0
- */
- def isnull(e: Column): Column = e.isNull
-
- /**
- * A column expression that generates monotonically increasing 64-bit integers.
- *
- * The generated ID is guaranteed to be monotonically increasing and unique, but not
- * consecutive. The current implementation puts the partition ID in the upper 31 bits, and the
- * record number within each partition in the lower 33 bits. The assumption is that the data
- * frame has less than 1 billion partitions, and each partition has less than 8 billion records.
- *
- * As an example, consider a `DataFrame` with two partitions, each with 3 records. This
- * expression would return the following IDs:
- *
- * {{{
- * 0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.
- * }}}
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- @deprecated("Use monotonically_increasing_id()", "2.0.0")
- def monotonicallyIncreasingId(): Column = monotonically_increasing_id()
-
- /**
- * A column expression that generates monotonically increasing 64-bit integers.
- *
- * The generated ID is guaranteed to be monotonically increasing and unique, but not
- * consecutive. The current implementation puts the partition ID in the upper 31 bits, and the
- * record number within each partition in the lower 33 bits. The assumption is that the data
- * frame has less than 1 billion partitions, and each partition has less than 8 billion records.
- *
- * As an example, consider a `DataFrame` with two partitions, each with 3 records. This
- * expression would return the following IDs:
- *
- * {{{
- * 0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.
- * }}}
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def monotonically_increasing_id(): Column = Column.fn("monotonically_increasing_id")
-
- /**
- * Returns col1 if it is not NaN, or col2 if col1 is NaN.
- *
- * Both inputs should be floating point columns (DoubleType or FloatType).
- *
- * @group conditional_funcs
- * @since 3.4.0
- */
- def nanvl(col1: Column, col2: Column): Column = Column.fn("nanvl", col1, col2)
-
- /**
- * Unary minus, i.e. negate the expression.
- * {{{
- * // Select the amount column and negates all values.
- * // Scala:
- * df.select( -df("amount") )
- *
- * // Java:
- * df.select( negate(df.col("amount")) );
- * }}}
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def negate(e: Column): Column = -e
-
- /**
- * Inversion of boolean expression, i.e. NOT.
- * {{{
- * // Scala: select rows that are not active (isActive === false)
- * df.filter( !df("isActive") )
- *
- * // Java:
- * df.filter( not(df.col("isActive")) );
- * }}}
- *
- * @group predicate_funcs
- * @since 3.4.0
- */
- def not(e: Column): Column = !e
-
- /**
- * Generate a random column with independent and identically distributed (i.i.d.) samples
- * uniformly distributed in [0.0, 1.0).
- *
- * @note
- * The function is non-deterministic in general case.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def rand(seed: Long): Column = Column.fn("rand", lit(seed))
-
- /**
- * Generate a random column with independent and identically distributed (i.i.d.) samples
- * uniformly distributed in [0.0, 1.0).
- *
- * @note
- * The function is non-deterministic in general case.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def rand(): Column = Column.fn("rand", lit(SparkClassUtils.random.nextLong))
-
- /**
- * Generate a column with independent and identically distributed (i.i.d.) samples from the
- * standard normal distribution.
- *
- * @note
- * The function is non-deterministic in general case.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def randn(seed: Long): Column = Column.fn("randn", lit(seed))
-
- /**
- * Generate a column with independent and identically distributed (i.i.d.) samples from the
- * standard normal distribution.
- *
- * @note
- * The function is non-deterministic in general case.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def randn(): Column = Column.fn("randn", lit(SparkClassUtils.random.nextLong))
-
- /**
- * Partition ID.
- *
- * @note
- * This is non-deterministic because it depends on data partitioning and task scheduling.
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def spark_partition_id(): Column = Column.fn("spark_partition_id")
-
- /**
- * Computes the square root of the specified float value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sqrt(e: Column): Column = Column.fn("sqrt", e)
-
- /**
- * Computes the square root of the specified float value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sqrt(colName: String): Column = sqrt(Column(colName))
-
- /**
- * Returns the sum of `left` and `right` and the result is null on overflow. The acceptable
- * input types are the same with the `+` operator.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_add(left: Column, right: Column): Column = Column.fn("try_add", left, right)
-
- /**
- * Returns the mean calculated from values of a group and the result is null on overflow.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_avg(e: Column): Column = Column.fn("try_avg", e)
-
- /**
- * Returns `dividend``/``divisor`. It always performs floating point division. Its result is
- * always null if `divisor` is 0.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_divide(left: Column, right: Column): Column = Column.fn("try_divide", left, right)
-
- /**
- * Returns the remainder of `dividend``/``divisor`. Its result is always null if `divisor` is 0.
- *
- * @group math_funcs
- * @since 4.0.0
- */
- def try_mod(left: Column, right: Column): Column = Column.fn("try_mod", left, right)
-
- /**
- * Returns `left``*``right` and the result is null on overflow. The acceptable input types are
- * the same with the `*` operator.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_multiply(left: Column, right: Column): Column = Column.fn("try_multiply", left, right)
-
- /**
- * Returns `left``-``right` and the result is null on overflow. The acceptable input types are
- * the same with the `-` operator.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_subtract(left: Column, right: Column): Column = Column.fn("try_subtract", left, right)
-
- /**
- * Returns the sum calculated from values of a group and the result is null on overflow.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def try_sum(e: Column): Column = Column.fn("try_sum", e)
-
- /**
- * Creates a new struct column. If the input column is a column in a `DataFrame`, or a derived
- * column expression that is named (i.e. aliased), its name would be retained as the
- * StructField's name, otherwise, the newly generated StructField's name would be auto generated
- * as `col` with a suffix `index + 1`, i.e. col1, col2, col3, ...
- *
- * @group struct_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def struct(cols: Column*): Column = Column.fn("struct", cols: _*)
-
- /**
- * Creates a new struct column that composes multiple input columns.
- *
- * @group struct_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def struct(colName: String, colNames: String*): Column = {
- struct((colName +: colNames).map(col): _*)
- }
-
- /**
- * Evaluates a list of conditions and returns one of multiple possible result expressions. If
- * otherwise is not defined at the end, null is returned for unmatched conditions.
- *
- * {{{
- * // Example: encoding gender string column into integer.
- *
- * // Scala:
- * people.select(when(people("gender") === "male", 0)
- * .when(people("gender") === "female", 1)
- * .otherwise(2))
- *
- * // Java:
- * people.select(when(col("gender").equalTo("male"), 0)
- * .when(col("gender").equalTo("female"), 1)
- * .otherwise(2))
- * }}}
- *
- * @group conditional_funcs
- * @since 3.4.0
- */
- def when(condition: Column, value: Any): Column = Column { builder =>
- builder.getUnresolvedFunctionBuilder
- .setFunctionName("when")
- .addArguments(condition.expr)
- .addArguments(lit(value).expr)
- }
-
- /**
- * Computes bitwise NOT (~) of a number.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- @deprecated("Use bitwise_not", "3.2.0")
- def bitwiseNOT(e: Column): Column = bitwise_not(e)
-
- /**
- * Computes bitwise NOT (~) of a number.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- def bitwise_not(e: Column): Column = Column.fn("~", e)
-
- /**
- * Returns the number of bits that are set in the argument expr as an unsigned 64-bit integer,
- * or NULL if the argument is NULL.
- *
- * @group bitwise_funcs
- * @since 3.5.0
- */
- def bit_count(e: Column): Column = Column.fn("bit_count", e)
-
- /**
- * Returns the value of the bit (0 or 1) at the specified position. The positions are numbered
- * from right to left, starting at zero. The position argument cannot be negative.
- *
- * @group bitwise_funcs
- * @since 3.5.0
- */
- def bit_get(e: Column, pos: Column): Column = Column.fn("bit_get", e, pos)
-
- /**
- * Returns the value of the bit (0 or 1) at the specified position. The positions are numbered
- * from right to left, starting at zero. The position argument cannot be negative.
- *
- * @group bitwise_funcs
- * @since 3.5.0
- */
- def getbit(e: Column, pos: Column): Column = Column.fn("getbit", e, pos)
-
- /**
- * Parses the expression string into the column that it represents, similar to
- * [[Dataset#selectExpr]].
- * {{{
- * // get the number of words of each length
- * df.groupBy(expr("length(word)")).count()
- * }}}
- *
- * @group normal_funcs
- */
- def expr(expr: String): Column = Column { builder =>
- builder.getExpressionStringBuilder.setExpression(expr)
- }
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Math Functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Computes the absolute value of a numeric value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def abs(e: Column): Column = Column.fn("abs", e)
-
- /**
- * @return
- * inverse cosine of `e` in radians, as if computed by `java.lang.Math.acos`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def acos(e: Column): Column = Column.fn("acos", e)
-
- /**
- * @return
- * inverse cosine of `columnName`, as if computed by `java.lang.Math.acos`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def acos(columnName: String): Column = acos(Column(columnName))
-
- /**
- * @return
- * inverse hyperbolic cosine of `e`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def acosh(e: Column): Column = Column.fn("acosh", e)
-
- /**
- * @return
- * inverse hyperbolic cosine of `columnName`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def acosh(columnName: String): Column = acosh(Column(columnName))
-
- /**
- * @return
- * inverse sine of `e` in radians, as if computed by `java.lang.Math.asin`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def asin(e: Column): Column = Column.fn("asin", e)
-
- /**
- * @return
- * inverse sine of `columnName`, as if computed by `java.lang.Math.asin`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def asin(columnName: String): Column = asin(Column(columnName))
-
- /**
- * @return
- * inverse hyperbolic sine of `e`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def asinh(e: Column): Column = Column.fn("asinh", e)
-
- /**
- * @return
- * inverse hyperbolic sine of `columnName`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def asinh(columnName: String): Column = asinh(Column(columnName))
-
- /**
- * @return
- * inverse tangent of `e` as if computed by `java.lang.Math.atan`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan(e: Column): Column = Column.fn("atan", e)
-
- /**
- * @return
- * inverse tangent of `columnName`, as if computed by `java.lang.Math.atan`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan(columnName: String): Column = atan(Column(columnName))
-
- /**
- * @param y
- * coordinate on y-axis
- * @param x
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(y: Column, x: Column): Column = Column.fn("atan2", y, x)
-
- /**
- * @param y
- * coordinate on y-axis
- * @param xName
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(y: Column, xName: String): Column = atan2(y, Column(xName))
-
- /**
- * @param yName
- * coordinate on y-axis
- * @param x
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(yName: String, x: Column): Column = atan2(Column(yName), x)
-
- /**
- * @param yName
- * coordinate on y-axis
- * @param xName
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(yName: String, xName: String): Column =
- atan2(Column(yName), Column(xName))
-
- /**
- * @param y
- * coordinate on y-axis
- * @param xValue
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(y: Column, xValue: Double): Column = atan2(y, lit(xValue))
-
- /**
- * @param yName
- * coordinate on y-axis
- * @param xValue
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(yName: String, xValue: Double): Column = atan2(Column(yName), xValue)
-
- /**
- * @param yValue
- * coordinate on y-axis
- * @param x
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(yValue: Double, x: Column): Column = atan2(lit(yValue), x)
-
- /**
- * @param yValue
- * coordinate on y-axis
- * @param xName
- * coordinate on x-axis
- * @return
- * the <i>theta</i> component of the point (<i>r</i>, <i>theta</i>) in polar coordinates that
- * corresponds to the point (<i>x</i>, <i>y</i>) in Cartesian coordinates, as if computed by
- * `java.lang.Math.atan2`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atan2(yValue: Double, xName: String): Column = atan2(yValue, Column(xName))
-
- /**
- * @return
- * inverse hyperbolic tangent of `e`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atanh(e: Column): Column = Column.fn("atanh", e)
-
- /**
- * @return
- * inverse hyperbolic tangent of `columnName`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def atanh(columnName: String): Column = atanh(Column(columnName))
-
- /**
- * An expression that returns the string representation of the binary value of the given long
- * column. For example, bin("12") returns "1100".
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def bin(e: Column): Column = Column.fn("bin", e)
-
- /**
- * An expression that returns the string representation of the binary value of the given long
- * column. For example, bin("12") returns "1100".
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def bin(columnName: String): Column = bin(Column(columnName))
-
- /**
- * Computes the cube-root of the given value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cbrt(e: Column): Column = Column.fn("cbrt", e)
-
- /**
- * Computes the cube-root of the given column.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cbrt(columnName: String): Column = cbrt(Column(columnName))
-
- /**
- * Computes the ceiling of the given value of `e` to `scale` decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def ceil(e: Column, scale: Column): Column = Column.fn("ceil", e, scale)
-
- /**
- * Computes the ceiling of the given value of `e` to 0 decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def ceil(e: Column): Column = Column.fn("ceil", e)
-
- /**
- * Computes the ceiling of the given value of `e` to 0 decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def ceil(columnName: String): Column = ceil(Column(columnName))
-
- /**
- * Computes the ceiling of the given value of `e` to `scale` decimal places.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def ceiling(e: Column, scale: Column): Column = Column.fn("ceiling", e, scale)
-
- /**
- * Computes the ceiling of the given value of `e` to 0 decimal places.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def ceiling(e: Column): Column = Column.fn("ceiling", e)
-
- /**
- * Convert a number in a string column from one base to another.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def conv(num: Column, fromBase: Int, toBase: Int): Column =
- Column.fn("conv", num, lit(fromBase), lit(toBase))
-
- /**
- * @param e
- * angle in radians
- * @return
- * cosine of the angle, as if computed by `java.lang.Math.cos`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cos(e: Column): Column = Column.fn("cos", e)
-
- /**
- * @param columnName
- * angle in radians
- * @return
- * cosine of the angle, as if computed by `java.lang.Math.cos`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cos(columnName: String): Column = cos(Column(columnName))
-
- /**
- * @param e
- * hyperbolic angle
- * @return
- * hyperbolic cosine of the angle, as if computed by `java.lang.Math.cosh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cosh(e: Column): Column = Column.fn("cosh", e)
-
- /**
- * @param columnName
- * hyperbolic angle
- * @return
- * hyperbolic cosine of the angle, as if computed by `java.lang.Math.cosh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cosh(columnName: String): Column = cosh(Column(columnName))
-
- /**
- * @param e
- * angle in radians
- * @return
- * cotangent of the angle
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def cot(e: Column): Column = Column.fn("cot", e)
-
- /**
- * @param e
- * angle in radians
- * @return
- * cosecant of the angle
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def csc(e: Column): Column = Column.fn("csc", e)
-
- /**
- * Returns Euler's number.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def e(): Column = Column.fn("e")
-
- /**
- * Computes the exponential of the given value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def exp(e: Column): Column = Column.fn("exp", e)
-
- /**
- * Computes the exponential of the given column.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def exp(columnName: String): Column = exp(Column(columnName))
-
- /**
- * Computes the exponential of the given value minus one.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def expm1(e: Column): Column = Column.fn("expm1", e)
-
- /**
- * Computes the exponential of the given column minus one.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def expm1(columnName: String): Column = expm1(Column(columnName))
-
- /**
- * Computes the factorial of the given value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def factorial(e: Column): Column = Column.fn("factorial", e)
-
- /**
- * Computes the floor of the given value of `e` to `scale` decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def floor(e: Column, scale: Column): Column = Column.fn("floor", e, scale)
-
- /**
- * Computes the floor of the given value of `e` to 0 decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def floor(e: Column): Column = Column.fn("floor", e)
-
- /**
- * Computes the floor of the given column value to 0 decimal places.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def floor(columnName: String): Column = floor(Column(columnName))
-
- /**
- * Returns the greatest value of the list of values, skipping null values. This function takes
- * at least 2 parameters. It will return null iff all parameters are null.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def greatest(exprs: Column*): Column = Column.fn("greatest", exprs: _*)
-
- /**
- * Returns the greatest value of the list of column names, skipping null values. This function
- * takes at least 2 parameters. It will return null iff all parameters are null.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def greatest(columnName: String, columnNames: String*): Column =
- greatest((columnName +: columnNames).map(Column.apply): _*)
-
- /**
- * Computes hex value of the given column.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hex(column: Column): Column = Column.fn("hex", column)
-
- /**
- * Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to
- * the byte representation of number.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def unhex(column: Column): Column = Column.fn("unhex", column)
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(l: Column, r: Column): Column = Column.fn("hypot", l, r)
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(l: Column, rightName: String): Column = hypot(l, Column(rightName))
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(leftName: String, r: Column): Column = hypot(Column(leftName), r)
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(leftName: String, rightName: String): Column =
- hypot(Column(leftName), Column(rightName))
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(l: Column, r: Double): Column = hypot(l, lit(r))
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(leftName: String, r: Double): Column = hypot(Column(leftName), r)
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(l: Double, r: Column): Column = hypot(lit(l), r)
-
- /**
- * Computes `sqrt(a^2^ + b^2^)` without intermediate overflow or underflow.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def hypot(l: Double, rightName: String): Column = hypot(l, Column(rightName))
-
- /**
- * Returns the least value of the list of values, skipping null values. This function takes at
- * least 2 parameters. It will return null iff all parameters are null.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def least(exprs: Column*): Column = Column.fn("least", exprs: _*)
-
- /**
- * Returns the least value of the list of column names, skipping null values. This function
- * takes at least 2 parameters. It will return null iff all parameters are null.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def least(columnName: String, columnNames: String*): Column =
- least((columnName +: columnNames).map(Column.apply): _*)
-
- /**
- * Computes the natural logarithm of the given value.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def ln(e: Column): Column = Column.fn("ln", e)
-
- /**
- * Computes the natural logarithm of the given value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log(e: Column): Column = ln(e)
-
- /**
- * Computes the natural logarithm of the given column.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log(columnName: String): Column = log(Column(columnName))
-
- /**
- * Returns the first argument-base logarithm of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log(base: Double, a: Column): Column = Column.fn("log", lit(base), a)
-
- /**
- * Returns the first argument-base logarithm of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log(base: Double, columnName: String): Column = log(base, Column(columnName))
-
- /**
- * Computes the logarithm of the given value in base 10.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log10(e: Column): Column = Column.fn("log10", e)
-
- /**
- * Computes the logarithm of the given value in base 10.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log10(columnName: String): Column = log10(Column(columnName))
-
- /**
- * Computes the natural logarithm of the given value plus one.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log1p(e: Column): Column = Column.fn("log1p", e)
-
- /**
- * Computes the natural logarithm of the given column plus one.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log1p(columnName: String): Column = log1p(Column(columnName))
-
- /**
- * Computes the logarithm of the given column in base 2.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log2(expr: Column): Column = Column.fn("log2", expr)
-
- /**
- * Computes the logarithm of the given value in base 2.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def log2(columnName: String): Column = log2(Column(columnName))
-
- /**
- * Returns the negated value.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def negative(e: Column): Column = Column.fn("negative", e)
-
- /**
- * Returns Pi.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def pi(): Column = Column.fn("pi")
-
- /**
- * Returns the value.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def positive(e: Column): Column = Column.fn("positive", e)
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(l: Column, r: Column): Column = Column.fn("power", l, r)
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(l: Column, rightName: String): Column = pow(l, Column(rightName))
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(leftName: String, r: Column): Column = pow(Column(leftName), r)
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(leftName: String, rightName: String): Column = pow(Column(leftName), Column(rightName))
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(l: Column, r: Double): Column = pow(l, lit(r))
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(leftName: String, r: Double): Column = pow(Column(leftName), r)
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(l: Double, r: Column): Column = pow(lit(l), r)
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pow(l: Double, rightName: String): Column = pow(l, Column(rightName))
-
- /**
- * Returns the value of the first argument raised to the power of the second argument.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def power(l: Column, r: Column): Column = Column.fn("power", l, r)
-
- /**
- * Returns the positive value of dividend mod divisor.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def pmod(dividend: Column, divisor: Column): Column = Column.fn("pmod", dividend, divisor)
-
- /**
- * Returns the double value that is closest in value to the argument and is equal to a
- * mathematical integer.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def rint(e: Column): Column = Column.fn("rint", e)
-
- /**
- * Returns the double value that is closest in value to the argument and is equal to a
- * mathematical integer.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def rint(columnName: String): Column = rint(Column(columnName))
-
- /**
- * Returns the value of the column `e` rounded to 0 decimal places with HALF_UP round mode.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def round(e: Column): Column = round(e, 0)
-
- /**
- * Round the value of `e` to `scale` decimal places with HALF_UP round mode if `scale` is
- * greater than or equal to 0 or at integral part when `scale` is less than 0.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def round(e: Column, scale: Int): Column = Column.fn("round", e, lit(scale))
-
- /**
- * Round the value of `e` to `scale` decimal places with HALF_UP round mode if `scale` is
- * greater than or equal to 0 or at integral part when `scale` is less than 0.
- *
- * @group math_funcs
- * @since 4.0.0
- */
- def round(e: Column, scale: Column): Column = Column.fn("round", e, scale)
-
- /**
- * Returns the value of the column `e` rounded to 0 decimal places with HALF_EVEN round mode.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def bround(e: Column): Column = bround(e, 0)
-
- /**
- * Round the value of `e` to `scale` decimal places with HALF_EVEN round mode if `scale` is
- * greater than or equal to 0 or at integral part when `scale` is less than 0.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def bround(e: Column, scale: Int): Column = Column.fn("bround", e, lit(scale))
-
- /**
- * Round the value of `e` to `scale` decimal places with HALF_EVEN round mode if `scale` is
- * greater than or equal to 0 or at integral part when `scale` is less than 0.
- *
- * @group math_funcs
- * @since 4.0.0
- */
- def bround(e: Column, scale: Column): Column = Column.fn("bround", e, scale)
-
- /**
- * @param e
- * angle in radians
- * @return
- * secant of the angle
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sec(e: Column): Column = Column.fn("sec", e)
-
- /**
- * Shift the given value numBits left. If the given value is a long value, this function will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- @deprecated("Use shiftleft", "3.2.0")
- def shiftLeft(e: Column, numBits: Int): Column = shiftleft(e, numBits)
-
- /**
- * Shift the given value numBits left. If the given value is a long value, this function will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- def shiftleft(e: Column, numBits: Int): Column = Column.fn("shiftleft", e, lit(numBits))
-
- /**
- * (Signed) shift the given value numBits right. If the given value is a long value, it will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- @deprecated("Use shiftright", "3.2.0")
- def shiftRight(e: Column, numBits: Int): Column = shiftright(e, numBits)
-
- /**
- * (Signed) shift the given value numBits right. If the given value is a long value, it will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- def shiftright(e: Column, numBits: Int): Column = Column.fn("shiftright", e, lit(numBits))
-
- /**
- * Unsigned shift the given value numBits right. If the given value is a long value, it will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- @deprecated("Use shiftrightunsigned", "3.2.0")
- def shiftRightUnsigned(e: Column, numBits: Int): Column = shiftrightunsigned(e, numBits)
-
- /**
- * Unsigned shift the given value numBits right. If the given value is a long value, it will
- * return a long value else it will return an integer value.
- *
- * @group bitwise_funcs
- * @since 3.4.0
- */
- def shiftrightunsigned(e: Column, numBits: Int): Column =
- Column.fn("shiftrightunsigned", e, lit(numBits))
-
- /**
- * Computes the signum of the given value.
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def sign(e: Column): Column = Column.fn("sign", e)
-
- /**
- * Computes the signum of the given value.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def signum(e: Column): Column = Column.fn("signum", e)
-
- /**
- * Computes the signum of the given column.
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def signum(columnName: String): Column = signum(Column(columnName))
-
- /**
- * @param e
- * angle in radians
- * @return
- * sine of the angle, as if computed by `java.lang.Math.sin`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sin(e: Column): Column = Column.fn("sin", e)
-
- /**
- * @param columnName
- * angle in radians
- * @return
- * sine of the angle, as if computed by `java.lang.Math.sin`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sin(columnName: String): Column = sin(Column(columnName))
-
- /**
- * @param e
- * hyperbolic angle
- * @return
- * hyperbolic sine of the given value, as if computed by `java.lang.Math.sinh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sinh(e: Column): Column = Column.fn("sinh", e)
-
- /**
- * @param columnName
- * hyperbolic angle
- * @return
- * hyperbolic sine of the given value, as if computed by `java.lang.Math.sinh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def sinh(columnName: String): Column = sinh(Column(columnName))
-
- /**
- * @param e
- * angle in radians
- * @return
- * tangent of the given value, as if computed by `java.lang.Math.tan`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def tan(e: Column): Column = Column.fn("tan", e)
-
- /**
- * @param columnName
- * angle in radians
- * @return
- * tangent of the given value, as if computed by `java.lang.Math.tan`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def tan(columnName: String): Column = tan(Column(columnName))
-
- /**
- * @param e
- * hyperbolic angle
- * @return
- * hyperbolic tangent of the given value, as if computed by `java.lang.Math.tanh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def tanh(e: Column): Column = Column.fn("tanh", e)
-
- /**
- * @param columnName
- * hyperbolic angle
- * @return
- * hyperbolic tangent of the given value, as if computed by `java.lang.Math.tanh`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def tanh(columnName: String): Column = tanh(Column(columnName))
-
- /**
- * @group math_funcs
- * @since 3.4.0
- */
- @deprecated("Use degrees", "2.1.0")
- def toDegrees(e: Column): Column = degrees(e)
-
- /**
- * @group math_funcs
- * @since 3.4.0
- */
- @deprecated("Use degrees", "2.1.0")
- def toDegrees(columnName: String): Column = degrees(Column(columnName))
-
- /**
- * Converts an angle measured in radians to an approximately equivalent angle measured in
- * degrees.
- *
- * @param e
- * angle in radians
- * @return
- * angle in degrees, as if computed by `java.lang.Math.toDegrees`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def degrees(e: Column): Column = Column.fn("degrees", e)
-
- /**
- * Converts an angle measured in radians to an approximately equivalent angle measured in
- * degrees.
- *
- * @param columnName
- * angle in radians
- * @return
- * angle in degrees, as if computed by `java.lang.Math.toDegrees`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def degrees(columnName: String): Column = degrees(Column(columnName))
-
- /**
- * @group math_funcs
- * @since 3.4.0
- */
- @deprecated("Use radians", "2.1.0")
- def toRadians(e: Column): Column = radians(e)
-
- /**
- * @group math_funcs
- * @since 3.4.0
- */
- @deprecated("Use radians", "2.1.0")
- def toRadians(columnName: String): Column = radians(Column(columnName))
-
- /**
- * Converts an angle measured in degrees to an approximately equivalent angle measured in
- * radians.
- *
- * @param e
- * angle in degrees
- * @return
- * angle in radians, as if computed by `java.lang.Math.toRadians`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def radians(e: Column): Column = Column.fn("radians", e)
-
- /**
- * Converts an angle measured in degrees to an approximately equivalent angle measured in
- * radians.
- *
- * @param columnName
- * angle in degrees
- * @return
- * angle in radians, as if computed by `java.lang.Math.toRadians`
- *
- * @group math_funcs
- * @since 3.4.0
- */
- def radians(columnName: String): Column = radians(Column(columnName))
-
- /**
- * Returns the bucket number into which the value of this expression would fall after being
- * evaluated. Note that input arguments must follow conditions listed below; otherwise, the
- * method will return null.
- *
- * @param v
- * value to compute a bucket number in the histogram
- * @param min
- * minimum value of the histogram
- * @param max
- * maximum value of the histogram
- * @param numBucket
- * the number of buckets
- * @return
- * the bucket number into which the value would fall after being evaluated
- * @group math_funcs
- * @since 3.5.0
- */
- def width_bucket(v: Column, min: Column, max: Column, numBucket: Column): Column =
- Column.fn("width_bucket", v, min, max, numBucket)
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // Misc functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Returns the current catalog.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def current_catalog(): Column = Column.fn("current_catalog")
-
- /**
- * Returns the current database.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def current_database(): Column = Column.fn("current_database")
-
- /**
- * Returns the current schema.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def current_schema(): Column = Column.fn("current_schema")
-
- /**
- * Returns the user name of current execution context.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def current_user(): Column = Column.fn("current_user")
-
- /**
- * Calculates the MD5 digest of a binary column and returns the value as a 32 character hex
- * string.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- def md5(e: Column): Column = Column.fn("md5", e)
-
- /**
- * Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex
- * string.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- def sha1(e: Column): Column = Column.fn("sha1", e)
-
- /**
- * Calculates the SHA-2 family of hash functions of a binary column and returns the value as a
- * hex string.
- *
- * @param e
- * column to compute SHA-2 on.
- * @param numBits
- * one of 224, 256, 384, or 512.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- def sha2(e: Column, numBits: Int): Column = {
- require(
- Seq(0, 224, 256, 384, 512).contains(numBits),
- s"numBits $numBits is not in the permitted values (0, 224, 256, 384, 512)")
- Column.fn("sha2", e, lit(numBits))
- }
-
- /**
- * Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value
- * as a bigint.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- def crc32(e: Column): Column = Column.fn("crc32", e)
-
- /**
- * Calculates the hash code of given columns, and returns the result as an int column.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def hash(cols: Column*): Column = Column.fn("hash", cols: _*)
-
- /**
- * Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm,
- * and returns the result as a long column. The hash computation uses an initial seed of 42.
- *
- * @group hash_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def xxhash64(cols: Column*): Column = Column.fn("xxhash64", cols: _*)
-
- /**
- * Returns null if the condition is true, and throws an exception otherwise.
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def assert_true(c: Column): Column = Column.fn("assert_true", c)
-
- /**
- * Returns null if the condition is true; throws an exception with the error message otherwise.
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def assert_true(c: Column, e: Column): Column = Column.fn("assert_true", c, e)
-
- /**
- * Throws an exception with the provided error message.
- *
- * @group misc_funcs
- * @since 3.4.0
- */
- def raise_error(c: Column): Column = Column.fn("raise_error", c)
-
- /**
- * Returns the estimated number of unique values given the binary representation of a
- * Datasketches HllSketch.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_sketch_estimate(c: Column): Column =
- Column.fn("hll_sketch_estimate", c)
-
- /**
- * Returns the estimated number of unique values given the binary representation of a
- * Datasketches HllSketch.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_sketch_estimate(columnName: String): Column =
- hll_sketch_estimate(Column(columnName))
-
- /**
- * Merges two binary representations of Datasketches HllSketch objects, using a Datasketches
- * Union object. Throws an exception if sketches have different lgConfigK values.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_union(c1: Column, c2: Column): Column =
- Column.fn("hll_union", c1, c2)
-
- /**
- * Merges two binary representations of Datasketches HllSketch objects, using a Dataskethes
- * Union object. Throws an exception if sketches have different lgConfigK values.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_union(columnName1: String, columnName2: String): Column =
- hll_union(Column(columnName1), Column(columnName2))
-
- /**
- * Merges two binary representations of Datasketches HllSketch objects, using a Datasketches
- * Union object. Throws an exception if sketches have different lgConfigK values and
- * allowDifferentLgConfigK is set to false.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_union(c1: Column, c2: Column, allowDifferentLgConfigK: Boolean): Column =
- Column.fn("hll_union", c1, c2, lit(allowDifferentLgConfigK))
-
- /**
- * Merges two binary representations of Datasketches HllSketch objects, using a Dataskethes
- * Union object. Throws an exception if sketches have different lgConfigK values and
- * allowDifferentLgConfigK is set to false.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def hll_union(
- columnName1: String,
- columnName2: String,
- allowDifferentLgConfigK: Boolean): Column =
- hll_union(Column(columnName1), Column(columnName2), allowDifferentLgConfigK)
-
- /**
- * Returns the user name of current execution context.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def user(): Column = Column.fn("user")
-
- /**
- * Returns the user name of current execution context.
- *
- * @group misc_funcs
- * @since 4.0.0
- */
- def session_user(): Column = Column.fn("session_user")
-
- /**
- * Returns an universally unique identifier (UUID) string. The value is returned as a canonical
- * UUID 36-character string.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def uuid(): Column = Column.fn("uuid", lit(SparkClassUtils.random.nextLong))
-
- /**
- * Returns an encrypted value of `input` using AES in given `mode` with the specified `padding`.
- * Key lengths of 16, 24 and 32 bits are supported. Supported combinations of (`mode`,
- * `padding`) are ('ECB', 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional initialization
- * vectors (IVs) are only supported for CBC and GCM modes. These must be 16 bytes for CBC and 12
- * bytes for GCM. If not provided, a random vector will be generated and prepended to the
- * output. Optional additional authenticated data (AAD) is only supported for GCM. If provided
- * for encryption, the identical AAD value must be provided for decryption. The default mode is
- * GCM.
- *
- * @param input
- * The binary value to encrypt.
- * @param key
- * The passphrase to use to encrypt the data.
- * @param mode
- * Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB,
- * GCM, CBC.
- * @param padding
- * Specifies how to pad messages whose length is not a multiple of the block size. Valid
- * values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS
- * for CBC.
- * @param iv
- * Optional initialization vector. Only supported for CBC and GCM modes. Valid values: None or
- * "". 16-byte array for CBC mode. 12-byte array for GCM mode.
- * @param aad
- * Optional additional authenticated data. Only supported for GCM mode. This can be any
- * free-form input and must be provided for both encryption and decryption.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_encrypt(
- input: Column,
- key: Column,
- mode: Column,
- padding: Column,
- iv: Column,
- aad: Column): Column = Column.fn("aes_encrypt", input, key, mode, padding, iv, aad)
-
- /**
- * Returns an encrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column,
- * Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_encrypt(input: Column, key: Column, mode: Column, padding: Column, iv: Column): Column =
- Column.fn("aes_encrypt", input, key, mode, padding, iv)
-
- /**
- * Returns an encrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column,
- * Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_encrypt(input: Column, key: Column, mode: Column, padding: Column): Column =
- Column.fn("aes_encrypt", input, key, mode, padding)
-
- /**
- * Returns an encrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column,
- * Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_encrypt(input: Column, key: Column, mode: Column): Column =
- Column.fn("aes_encrypt", input, key, mode)
-
- /**
- * Returns an encrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column,
- * Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_encrypt(input: Column, key: Column): Column =
- Column.fn("aes_encrypt", input, key)
-
- /**
- * Returns a decrypted value of `input` using AES in `mode` with `padding`. Key lengths of 16,
- * 24 and 32 bits are supported. Supported combinations of (`mode`, `padding`) are ('ECB',
- * 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional additional authenticated data (AAD) is
- * only supported for GCM. If provided for encryption, the identical AAD value must be provided
- * for decryption. The default mode is GCM.
- *
- * @param input
- * The binary value to decrypt.
- * @param key
- * The passphrase to use to decrypt the data.
- * @param mode
- * Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB,
- * GCM, CBC.
- * @param padding
- * Specifies how to pad messages whose length is not a multiple of the block size. Valid
- * values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS
- * for CBC.
- * @param aad
- * Optional additional authenticated data. Only supported for GCM mode. This can be any
- * free-form input and must be provided for both encryption and decryption.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_decrypt(
- input: Column,
- key: Column,
- mode: Column,
- padding: Column,
- aad: Column): Column =
- Column.fn("aes_decrypt", input, key, mode, padding, aad)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_decrypt(input: Column, key: Column, mode: Column, padding: Column): Column =
- Column.fn("aes_decrypt", input, key, mode, padding)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_decrypt(input: Column, key: Column, mode: Column): Column =
- Column.fn("aes_decrypt", input, key, mode)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def aes_decrypt(input: Column, key: Column): Column =
- Column.fn("aes_decrypt", input, key)
-
- /**
- * This is a special version of `aes_decrypt` that performs the same operation, but returns a
- * NULL value instead of raising an error if the decryption cannot be performed.
- *
- * @param input
- * The binary value to decrypt.
- * @param key
- * The passphrase to use to decrypt the data.
- * @param mode
- * Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB,
- * GCM, CBC.
- * @param padding
- * Specifies how to pad messages whose length is not a multiple of the block size. Valid
- * values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS
- * for CBC.
- * @param aad
- * Optional additional authenticated data. Only supported for GCM mode. This can be any
- * free-form input and must be provided for both encryption and decryption.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def try_aes_decrypt(
- input: Column,
- key: Column,
- mode: Column,
- padding: Column,
- aad: Column): Column =
- Column.fn("try_aes_decrypt", input, key, mode, padding, aad)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def try_aes_decrypt(input: Column, key: Column, mode: Column, padding: Column): Column =
- Column.fn("try_aes_decrypt", input, key, mode, padding)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def try_aes_decrypt(input: Column, key: Column, mode: Column): Column =
- Column.fn("try_aes_decrypt", input, key, mode)
-
- /**
- * Returns a decrypted value of `input`.
- *
- * @see
- * `org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)`
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def try_aes_decrypt(input: Column, key: Column): Column =
- Column.fn("try_aes_decrypt", input, key)
-
- /**
- * Returns a sha1 hash value as a hex string of the `col`.
- *
- * @group hash_funcs
- * @since 3.5.0
- */
- def sha(col: Column): Column = Column.fn("sha", col)
-
- /**
- * Returns the length of the block being read, or -1 if not available.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def input_file_block_length(): Column = Column.fn("input_file_block_length")
-
- /**
- * Returns the start offset of the block being read, or -1 if not available.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def input_file_block_start(): Column = Column.fn("input_file_block_start")
-
- /**
- * Calls a method with reflection.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def reflect(cols: Column*): Column = Column.fn("reflect", cols: _*)
-
- /**
- * Calls a method with reflection.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def java_method(cols: Column*): Column = Column.fn("java_method", cols: _*)
-
- /**
- * This is a special version of `reflect` that performs the same operation, but returns a NULL
- * value instead of raising an error if the invoke method thrown exception.
- *
- * @group misc_funcs
- * @since 4.0.0
- */
- def try_reflect(cols: Column*): Column = Column.fn("try_reflect", cols: _*)
-
- /**
- * Returns the Spark version. The string contains 2 fields, the first being a release version
- * and the second being a git revision.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def version(): Column = Column.fn("version")
-
- /**
- * Return DDL-formatted type string for the data type of the input.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def typeof(col: Column): Column = Column.fn("typeof", col)
-
- /**
- * Separates `col1`, ..., `colk` into `n` rows. Uses column names col0, col1, etc. by default
- * unless specified otherwise.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def stack(cols: Column*): Column = Column.fn("stack", cols: _*)
-
- /**
- * Returns a random value with independent and identically distributed (i.i.d.) uniformly
- * distributed values in [0, 1).
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def random(seed: Column): Column = Column.fn("random", seed)
-
- /**
- * Returns a random value with independent and identically distributed (i.i.d.) uniformly
- * distributed values in [0, 1).
- *
- * @group math_funcs
- * @since 3.5.0
- */
- def random(): Column = Column.fn("random", lit(SparkClassUtils.random.nextLong))
-
- /**
- * Returns the bit position for the given input column.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def bitmap_bit_position(col: Column): Column =
- Column.fn("bitmap_bit_position", col)
-
- /**
- * Returns the bucket number for the given input column.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def bitmap_bucket_number(col: Column): Column =
- Column.fn("bitmap_bucket_number", col)
-
- /**
- * Returns a bitmap with the positions of the bits set from all the values from the input
- * column. The input column will most likely be bitmap_bit_position().
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bitmap_construct_agg(col: Column): Column =
- Column.fn("bitmap_construct_agg", col)
-
- /**
- * Returns the number of set bits in the input bitmap.
- *
- * @group misc_funcs
- * @since 3.5.0
- */
- def bitmap_count(col: Column): Column = Column.fn("bitmap_count", col)
-
- /**
- * Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column. The
- * input column should be bitmaps created from bitmap_construct_agg().
- *
- * @group agg_funcs
- * @since 3.5.0
- */
- def bitmap_or_agg(col: Column): Column = Column.fn("bitmap_or_agg", col)
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // String functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Computes the numeric value of the first character of the string column, and returns the
- * result as an int column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def ascii(e: Column): Column = Column.fn("ascii", e)
-
- /**
- * Computes the BASE64 encoding of a binary column and returns it as a string column. This is
- * the reverse of unbase64.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def base64(e: Column): Column = Column.fn("base64", e)
-
- /**
- * Calculates the bit length for the specified string column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def bit_length(e: Column): Column = Column.fn("bit_length", e)
-
- /**
- * Concatenates multiple input string columns together into a single string column, using the
- * given separator.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def concat_ws(sep: String, exprs: Column*): Column =
- Column.fn("concat_ws", lit(sep) +: exprs: _*)
-
- /**
- * Computes the first argument into a string from a binary using the provided character set (one
- * of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32'). If either
- * argument is null, the result will also be null.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def decode(value: Column, charset: String): Column =
- Column.fn("decode", value, lit(charset))
-
- /**
- * Computes the first argument into a binary from a string using the provided character set (one
- * of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32'). If either
- * argument is null, the result will also be null.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def encode(value: Column, charset: String): Column =
- Column.fn("encode", value, lit(charset))
-
- /**
- * Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with
- * HALF_EVEN round mode, and returns the result as a string column.
- *
- * If d is 0, the result has no decimal point or fractional part. If d is less than 0, the
- * result will be null.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def format_number(x: Column, d: Int): Column = Column.fn("format_number", x, lit(d))
-
- /**
- * Formats the arguments in printf-style and returns the result as a string column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- @scala.annotation.varargs
- def format_string(format: String, arguments: Column*): Column =
- Column.fn("format_string", lit(format) +: arguments: _*)
-
- /**
- * Returns a new string column by converting the first letter of each word to uppercase. Words
- * are delimited by whitespace.
- *
- * For example, "hello world" will become "Hello World".
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def initcap(e: Column): Column = Column.fn("initcap", e)
-
- /**
- * Locate the position of the first occurrence of substr column in the given string. Returns
- * null if either of the arguments are null.
- *
- * @note
- * The position is not zero based, but 1 based index. Returns 0 if substr could not be found
- * in str.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def instr(str: Column, substring: String): Column = Column.fn("instr", str, lit(substring))
-
- /**
- * Computes the character length of a given string or number of bytes of a binary string. The
- * length of character strings include the trailing spaces. The length of binary strings
- * includes binary zeros.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def length(e: Column): Column = Column.fn("length", e)
-
- /**
- * Computes the character length of a given string or number of bytes of a binary string. The
- * length of character strings include the trailing spaces. The length of binary strings
- * includes binary zeros.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def len(e: Column): Column = Column.fn("len", e)
-
- /**
- * Converts a string column to lower case.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def lower(e: Column): Column = Column.fn("lower", e)
-
- /**
- * Computes the Levenshtein distance of the two given string columns if it's less than or equal
- * to a given threshold.
- * @return
- * result distance, or -1
- * @group string_funcs
- * @since 3.5.0
- */
- def levenshtein(l: Column, r: Column, threshold: Int): Column =
- Column.fn("levenshtein", l, r, lit(threshold))
-
- /**
- * Computes the Levenshtein distance of the two given string columns.
- * @group string_funcs
- * @since 3.4.0
- */
- def levenshtein(l: Column, r: Column): Column = Column.fn("levenshtein", l, r)
-
- /**
- * Locate the position of the first occurrence of substr.
- *
- * @note
- * The position is not zero based, but 1 based index. Returns 0 if substr could not be found
- * in str.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def locate(substr: String, str: Column): Column = Column.fn("locate", lit(substr), str)
-
- /**
- * Locate the position of the first occurrence of substr in a string column, after position pos.
- *
- * @note
- * The position is not zero based, but 1 based index. returns 0 if substr could not be found
- * in str.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def locate(substr: String, str: Column, pos: Int): Column =
- Column.fn("locate", lit(substr), str, lit(pos))
-
- /**
- * Left-pad the string column with pad to a length of len. If the string column is longer than
- * len, the return value is shortened to len characters.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def lpad(str: Column, len: Int, pad: String): Column =
- Column.fn("lpad", str, lit(len), lit(pad))
-
- /**
- * Left-pad the binary column with pad to a byte length of len. If the binary column is longer
- * than len, the return value is shortened to len bytes.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def lpad(str: Column, len: Int, pad: Array[Byte]): Column =
- Column.fn("lpad", str, lit(len), lit(pad))
-
- /**
- * Trim the spaces from left end for the specified string value.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def ltrim(e: Column): Column = Column.fn("ltrim", e)
-
- /**
- * Trim the specified character string from left end for the specified string column.
- * @group string_funcs
- * @since 3.4.0
- */
- def ltrim(e: Column, trimString: String): Column = Column.fn("ltrim", lit(trimString), e)
-
- /**
- * Calculates the byte length for the specified string column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def octet_length(e: Column): Column = Column.fn("octet_length", e)
-
- /**
- * Returns true if `str` matches `regexp`, or false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def rlike(str: Column, regexp: Column): Column = Column.fn("rlike", str, regexp)
-
- /**
- * Returns true if `str` matches `regexp`, or false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def regexp(str: Column, regexp: Column): Column = Column.fn("regexp", str, regexp)
-
- /**
- * Returns true if `str` matches `regexp`, or false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def regexp_like(str: Column, regexp: Column): Column = Column.fn("regexp_like", str, regexp)
-
- /**
- * Returns a count of the number of times that the regular expression pattern `regexp` is
- * matched in the string `str`.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_count(str: Column, regexp: Column): Column = Column.fn("regexp_count", str, regexp)
-
- /**
- * Extract a specific group matched by a Java regex, from the specified string column. If the
- * regex did not match, or the specified group did not match, an empty string is returned. if
- * the specified group index exceeds the group count of regex, an IllegalArgumentException will
- * be thrown.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def regexp_extract(e: Column, exp: String, groupIdx: Int): Column =
- Column.fn("regexp_extract", e, lit(exp), lit(groupIdx))
-
- /**
- * Extract all strings in the `str` that match the `regexp` expression and corresponding to the
- * first regex group index.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_extract_all(str: Column, regexp: Column): Column =
- Column.fn("regexp_extract_all", str, regexp)
-
- /**
- * Extract all strings in the `str` that match the `regexp` expression and corresponding to the
- * regex group index.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_extract_all(str: Column, regexp: Column, idx: Column): Column =
- Column.fn("regexp_extract_all", str, regexp, idx)
-
- /**
- * Replace all substrings of the specified string value that match regexp with rep.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def regexp_replace(e: Column, pattern: String, replacement: String): Column =
- regexp_replace(e, lit(pattern), lit(replacement))
-
- /**
- * Replace all substrings of the specified string value that match regexp with rep.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def regexp_replace(e: Column, pattern: Column, replacement: Column): Column =
- Column.fn("regexp_replace", e, pattern, replacement)
-
- /**
- * Returns the substring that matches the regular expression `regexp` within the string `str`.
- * If the regular expression is not found, the result is null.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_substr(str: Column, regexp: Column): Column = Column.fn("regexp_substr", str, regexp)
-
- /**
- * Searches a string for a regular expression and returns an integer that indicates the
- * beginning position of the matched substring. Positions are 1-based, not 0-based. If no match
- * is found, returns 0.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_instr(str: Column, regexp: Column): Column = Column.fn("regexp_instr", str, regexp)
-
- /**
- * Searches a string for a regular expression and returns an integer that indicates the
- * beginning position of the matched substring. Positions are 1-based, not 0-based. If no match
- * is found, returns 0.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def regexp_instr(str: Column, regexp: Column, idx: Column): Column =
- Column.fn("regexp_instr", str, regexp, idx)
-
- /**
- * Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse
- * of base64.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def unbase64(e: Column): Column = Column.fn("unbase64", e)
-
- /**
- * Right-pad the string column with pad to a length of len. If the string column is longer than
- * len, the return value is shortened to len characters.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def rpad(str: Column, len: Int, pad: String): Column =
- Column.fn("rpad", str, lit(len), lit(pad))
-
- /**
- * Right-pad the binary column with pad to a byte length of len. If the binary column is longer
- * than len, the return value is shortened to len bytes.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def rpad(str: Column, len: Int, pad: Array[Byte]): Column =
- Column.fn("rpad", str, lit(len), lit(pad))
-
- /**
- * Repeats a string column n times, and returns it as a new string column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def repeat(str: Column, n: Int): Column = Column.fn("repeat", str, lit(n))
-
- /**
- * Repeats a string column n times, and returns it as a new string column.
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def repeat(str: Column, n: Column): Column = Column.fn("repeat", str, n)
-
- /**
- * Trim the spaces from right end for the specified string value.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def rtrim(e: Column): Column = Column.fn("rtrim", e)
-
- /**
- * Trim the specified character string from right end for the specified string column.
- * @group string_funcs
- * @since 3.4.0
- */
- def rtrim(e: Column, trimString: String): Column = Column.fn("rtrim", lit(trimString), e)
-
- /**
- * Returns the soundex code for the specified expression.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def soundex(e: Column): Column = Column.fn("soundex", e)
-
- /**
- * Splits str around matches of the given pattern.
- *
- * @param str
- * a string expression to split
- * @param pattern
- * a string representing a regular expression. The regex string should be a Java regular
- * expression.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def split(str: Column, pattern: String): Column = Column.fn("split", str, lit(pattern))
-
- /**
- * Splits str around matches of the given pattern.
- *
- * @param str
- * a string expression to split
- * @param pattern
- * a column of string representing a regular expression. The regex string should be a Java
- * regular expression.
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def split(str: Column, pattern: Column): Column = Column.fn("split", str, pattern)
-
- /**
- * Splits str around matches of the given pattern.
- *
- * @param str
- * a string expression to split
- * @param pattern
- * a string representing a regular expression. The regex string should be a Java regular
- * expression.
- * @param limit
- * an integer expression which controls the number of times the regex is applied. <ul>
- * <li>limit greater than 0: The resulting array's length will not be more than limit, and the
- * resulting array's last entry will contain all input beyond the last matched regex.</li>
- * <li>limit less than or equal to 0: `regex` will be applied as many times as possible, and
- * the resulting array can be of any size.</li> </ul>
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def split(str: Column, pattern: String, limit: Int): Column =
- Column.fn("split", str, lit(pattern), lit(limit))
-
- /**
- * Splits str around matches of the given pattern.
- *
- * @param str
- * a string expression to split
- * @param pattern
- * a column of string representing a regular expression. The regex string should be a Java
- * regular expression.
- * @param limit
- * a column of integer expression which controls the number of times the regex is applied.
- * <ul> <li>limit greater than 0: The resulting array's length will not be more than limit,
- * and the resulting array's last entry will contain all input beyond the last matched
- * regex.</li> <li>limit less than or equal to 0: `regex` will be applied as many times as
- * possible, and the resulting array can be of any size.</li> </ul>
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def split(str: Column, pattern: Column, limit: Column): Column =
- Column.fn("split", str, pattern, limit)
-
- /**
- * Substring starts at `pos` and is of length `len` when str is String type or returns the slice
- * of byte array that starts at `pos` in byte and is of length `len` when str is Binary type
- *
- * @note
- * The position is not zero based, but 1 based index.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def substring(str: Column, pos: Int, len: Int): Column =
- Column.fn("substring", str, lit(pos), lit(len))
-
- /**
- * Substring starts at `pos` and is of length `len` when str is String type or returns the slice
- * of byte array that starts at `pos` in byte and is of length `len` when str is Binary type
- *
- * @note
- * The position is not zero based, but 1 based index.
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def substring(str: Column, pos: Column, len: Column): Column =
- Column.fn("substring", str, pos, len)
-
- /**
- * Returns the substring from string str before count occurrences of the delimiter delim. If
- * count is positive, everything the left of the final delimiter (counting from left) is
- * returned. If count is negative, every to the right of the final delimiter (counting from the
- * right) is returned. substring_index performs a case-sensitive match when searching for delim.
- *
- * @group string_funcs
- */
- def substring_index(str: Column, delim: String, count: Int): Column =
- Column.fn("substring_index", str, lit(delim), lit(count))
-
- /**
- * Overlay the specified portion of `src` with `replace`, starting from byte position `pos` of
- * `src` and proceeding for `len` bytes.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def overlay(src: Column, replace: Column, pos: Column, len: Column): Column =
- Column.fn("overlay", src, replace, pos, len)
-
- /**
- * Overlay the specified portion of `src` with `replace`, starting from byte position `pos` of
- * `src`.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def overlay(src: Column, replace: Column, pos: Column): Column =
- Column.fn("overlay", src, replace, pos)
-
- /**
- * Splits a string into arrays of sentences, where each sentence is an array of words.
- * @group string_funcs
- * @since 3.4.0
- */
- def sentences(string: Column, language: Column, country: Column): Column =
- Column.fn("sentences", string, language, country)
-
- /**
- * Splits a string into arrays of sentences, where each sentence is an array of words. The
- * default locale is used.
- * @group string_funcs
- * @since 3.4.0
- */
- def sentences(string: Column): Column = Column.fn("sentences", string)
-
- /**
- * Translate any character in the src by a character in replaceString. The characters in
- * replaceString correspond to the characters in matchingString. The translate will happen when
- * any character in the string matches the character in the `matchingString`.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def translate(src: Column, matchingString: String, replaceString: String): Column =
- Column.fn("translate", src, lit(matchingString), lit(replaceString))
-
- /**
- * Trim the spaces from both ends for the specified string column.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def trim(e: Column): Column = Column.fn("trim", e)
-
- /**
- * Trim the specified character from both ends for the specified string column.
- * @group string_funcs
- * @since 3.4.0
- */
- def trim(e: Column, trimString: String): Column = Column.fn("trim", lit(trimString), e)
-
- /**
- * Converts a string column to upper case.
- *
- * @group string_funcs
- * @since 3.4.0
- */
- def upper(e: Column): Column = Column.fn("upper", e)
-
- /**
- * Converts the input `e` to a binary value based on the supplied `f`. The `f` can be a
- * case-insensitive string literal of "hex", "utf-8", "utf8", or "base64". By default, the
- * binary format for conversion is "hex" if `fmt` is omitted. The function returns NULL if at
- * least one of the input parameters is NULL.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def to_binary(e: Column, f: Column): Column = Column.fn("to_binary", e, f)
-
- /**
- * Converts the input `e` to a binary value based on the format "hex". The function returns NULL
- * if at least one of the input parameters is NULL.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def to_binary(e: Column): Column = Column.fn("to_binary", e)
-
- // scalastyle:off line.size.limit
- /**
- * Convert `e` to a string based on the `format`. Throws an exception if the conversion fails.
- *
- * @param e
- * A column of number to be converted
- * @param format
- * The format can consist of the following characters, case insensitive: <ul> <li> '0' or '9':
- * Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string
- * matches a sequence of digits in the input value, generating a result string of the same
- * length as the corresponding sequence in the format string. The result string is left-padded
- * with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal
- * value, starts with 0, and is before the decimal point. Otherwise, it is padded with
- * spaces.</li> <li>'.' or 'D': Specifies the position of the decimal point (optional, only
- * allowed once).</li> <li>',' or 'G': Specifies the position of the grouping (thousands)
- * separator (,). There must be a 0 or 9 to the left and right of each grouping
- * separator.</li> <li>'$': Specifies the location of the $ currency sign. This character may
- * only be specified once.</li> <li>'S' or 'MI': Specifies the position of a '-' or '+' sign
- * (optional, only allowed once at the beginning or end of the format string). Note that 'S'
- * prints '+' for positive values but 'MI' prints a space.</li> <li>'PR': Only allowed at the
- * end of the format string; specifies that the result string will be wrapped by angle
- * brackets if the input value is negative.</li> </ul> If `e` is a datetime, `format` shall be
- * a valid datetime pattern, see <a
- * href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime
- * Patterns</a>. If `e` is a binary, it is converted to a string in one of the formats: <ul>
- * <li>'base64': a base 64 string.</li> <li>'hex': a string in the hexadecimal format.</li>
- * <li>'utf-8': the input binary is decoded to UTF-8 string.</li> </ul>
- *
- * @group string_funcs
- * @since 3.5.0
- */
- // scalastyle:on line.size.limit
- def to_char(e: Column, format: Column): Column = Column.fn("to_char", e, format)
-
- // scalastyle:off line.size.limit
- /**
- * Convert `e` to a string based on the `format`. Throws an exception if the conversion fails.
- *
- * @param e
- * A column of number to be converted
- * @param format
- * The format can consist of the following characters, case insensitive: <ul> <li> '0' or '9':
- * Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string
- * matches a sequence of digits in the input value, generating a result string of the same
- * length as the corresponding sequence in the format string. The result string is left-padded
- * with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal
- * value, starts with 0, and is before the decimal point. Otherwise, it is padded with
- * spaces.</li> <li>'.' or 'D': Specifies the position of the decimal point (optional, only
- * allowed once).</li> <li>',' or 'G': Specifies the position of the grouping (thousands)
- * separator (,). There must be a 0 or 9 to the left and right of each grouping
- * separator.</li> <li>'$': Specifies the location of the $ currency sign. This character may
- * only be specified once.</li> <li>'S' or 'MI': Specifies the position of a '-' or '+' sign
- * (optional, only allowed once at the beginning or end of the format string). Note that 'S'
- * prints '+' for positive values but 'MI' prints a space.</li> <li>'PR': Only allowed at the
- * end of the format string; specifies that the result string will be wrapped by angle
- * brackets if the input value is negative.</li> </ul> If `e` is a datetime, `format` shall be
- * a valid datetime pattern, see <a
- * href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime
- * Patterns</a>. If `e` is a binary, it is converted to a string in one of the formats: <ul>
- * <li>'base64': a base 64 string.</li> <li>'hex': a string in the hexadecimal format.</li>
- * <li>'utf-8': the input binary is decoded to UTF-8 string.</li> </ul>
- *
- * @group string_funcs
- * @since 3.5.0
- */
- // scalastyle:on line.size.limit
- def to_varchar(e: Column, format: Column): Column = Column.fn("to_varchar", e, format)
-
- /**
- * Convert string 'e' to a number based on the string format 'format'. Throws an exception if
- * the conversion fails.
- *
- * @param e
- * A column of string to be converted
- * @param format
- * The format can consist of the following characters, case insensitive: <ul><li> '0' or '9':
- * Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string
- * matches a sequence of digits in the input string. If the 0/9 sequence starts with 0 and is
- * before the decimal point, it can only match a digit sequence of the same size. Otherwise,
- * if the sequence starts with 9 or is after the decimal point, it can match a digit sequence
- * that has the same or smaller size.</li> <li>'.' or 'D': Specifies the position of the
- * decimal point (optional, only allowed once).</li> <li>',' or 'G': Specifies the position of
- * the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of
- * each grouping separator. 'expr' must match the grouping separator relevant for the size of
- * the number.</li> <li>'$': Specifies the location of the $ currency sign. This character may
- * only be specified once.</li> <li>'S' or 'MI': Specifies the position of a '-' or '+' sign
- * (optional, only allowed once at the beginning or end of the format string). Note that 'S'
- * allows '-' but 'MI' does not.</li> <li>'PR': Only allowed at the end of the format string;
- * specifies that 'expr' indicates a negative number with wrapping angled brackets.</li></ul>
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def to_number(e: Column, format: Column): Column = Column.fn("to_number", e, format)
-
- /**
- * Replaces all occurrences of `search` with `replace`.
- *
- * @param src
- * A column of string to be replaced
- * @param search
- * A column of string, If `search` is not found in `str`, `str` is returned unchanged.
- * @param replace
- * A column of string, If `replace` is not specified or is an empty string, nothing replaces
- * the string that is removed from `str`.
- * @group string_funcs
- * @since 3.5.0
- */
- def replace(src: Column, search: Column, replace: Column): Column =
- Column.fn("replace", src, search, replace)
-
- /**
- * Replaces all occurrences of `search` with `replace`.
- *
- * @param src
- * A column of string to be replaced
- * @param search
- * A column of string, If `search` is not found in `src`, `src` is returned unchanged.
- * @group string_funcs
- * @since 3.5.0
- */
- def replace(src: Column, search: Column): Column = Column.fn("replace", src, search)
-
- /**
- * Splits `str` by delimiter and return requested part of the split (1-based). If any input is
- * null, returns null. if `partNum` is out of range of split parts, returns empty string. If
- * `partNum` is 0, throws an error. If `partNum` is negative, the parts are counted backward
- * from the end of the string. If the `delimiter` is an empty string, the `str` is not split.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def split_part(str: Column, delimiter: Column, partNum: Column): Column =
- Column.fn("split_part", str, delimiter, partNum)
-
- /**
- * Returns the substring of `str` that starts at `pos` and is of length `len`, or the slice of
- * byte array that starts at `pos` and is of length `len`.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def substr(str: Column, pos: Column, len: Column): Column =
- Column.fn("substr", str, pos, len)
-
- /**
- * Returns the substring of `str` that starts at `pos`, or the slice of byte array that starts
- * at `pos`.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def substr(str: Column, pos: Column): Column = Column.fn("substr", str, pos)
-
- /**
- * Extracts a part from a URL.
- *
- * @group url_funcs
- * @since 3.5.0
- */
- def parse_url(url: Column, partToExtract: Column, key: Column): Column =
- Column.fn("parse_url", url, partToExtract, key)
-
- /**
- * Extracts a part from a URL.
- *
- * @group url_funcs
- * @since 3.5.0
- */
- def parse_url(url: Column, partToExtract: Column): Column =
- Column.fn("parse_url", url, partToExtract)
-
- /**
- * Formats the arguments in printf-style and returns the result as a string column.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def printf(format: Column, arguments: Column*): Column =
- Column.fn("printf", (format +: arguments): _*)
-
- /**
- * Decodes a `str` in 'application/x-www-form-urlencoded' format using a specific encoding
- * scheme.
- *
- * @group url_funcs
- * @since 3.5.0
- */
- def url_decode(str: Column): Column = Column.fn("url_decode", str)
-
- /**
- * This is a special version of `url_decode` that performs the same operation, but returns a
- * NULL value instead of raising an error if the decoding cannot be performed.
- *
- * @group url_funcs
- * @since 4.0.0
- */
- def try_url_decode(str: Column): Column = Column.fn("try_url_decode", str)
-
- /**
- * Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding
- * scheme.
- *
- * @group url_funcs
- * @since 3.5.0
- */
- def url_encode(str: Column): Column = Column.fn("url_encode", str)
-
- /**
- * Returns the position of the first occurrence of `substr` in `str` after position `start`. The
- * given `start` and return value are 1-based.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def position(substr: Column, str: Column, start: Column): Column =
- Column.fn("position", substr, str, start)
-
- /**
- * Returns the position of the first occurrence of `substr` in `str` after position `1`. The
- * return value are 1-based.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def position(substr: Column, str: Column): Column =
- Column.fn("position", substr, str)
-
- /**
- * Returns a boolean. The value is True if str ends with suffix. Returns NULL if either input
- * expression is NULL. Otherwise, returns False. Both str or suffix must be of STRING or BINARY
- * type.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def endswith(str: Column, suffix: Column): Column =
- Column.fn("endswith", str, suffix)
-
- /**
- * Returns a boolean. The value is True if str starts with prefix. Returns NULL if either input
- * expression is NULL. Otherwise, returns False. Both str or prefix must be of STRING or BINARY
- * type.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def startswith(str: Column, prefix: Column): Column =
- Column.fn("startswith", str, prefix)
-
- /**
- * Returns the ASCII character having the binary equivalent to `n`. If n is larger than 256 the
- * result is equivalent to char(n % 256)
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def char(n: Column): Column = Column.fn("char", n)
-
- /**
- * Removes the leading and trailing space characters from `str`.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def btrim(str: Column): Column = Column.fn("btrim", str)
-
- /**
- * Remove the leading and trailing `trim` characters from `str`.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def btrim(str: Column, trim: Column): Column = Column.fn("btrim", str, trim)
-
- /**
- * This is a special version of `to_binary` that performs the same operation, but returns a NULL
- * value instead of raising an error if the conversion cannot be performed.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def try_to_binary(e: Column, f: Column): Column = Column.fn("try_to_binary", e, f)
-
- /**
- * This is a special version of `to_binary` that performs the same operation, but returns a NULL
- * value instead of raising an error if the conversion cannot be performed.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def try_to_binary(e: Column): Column = Column.fn("try_to_binary", e)
-
- /**
- * Convert string `e` to a number based on the string format `format`. Returns NULL if the
- * string `e` does not match the expected format. The format follows the same semantics as the
- * to_number function.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def try_to_number(e: Column, format: Column): Column = Column.fn("try_to_number", e, format)
-
- /**
- * Returns the character length of string data or number of bytes of binary data. The length of
- * string data includes the trailing spaces. The length of binary data includes binary zeros.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def char_length(str: Column): Column = Column.fn("char_length", str)
-
- /**
- * Returns the character length of string data or number of bytes of binary data. The length of
- * string data includes the trailing spaces. The length of binary data includes binary zeros.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def character_length(str: Column): Column = Column.fn("character_length", str)
-
- /**
- * Returns the ASCII character having the binary equivalent to `n`. If n is larger than 256 the
- * result is equivalent to chr(n % 256)
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def chr(n: Column): Column = Column.fn("chr", n)
-
- /**
- * Returns a boolean. The value is True if right is found inside left. Returns NULL if either
- * input expression is NULL. Otherwise, returns False. Both left or right must be of STRING or
- * BINARY type.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def contains(left: Column, right: Column): Column = Column.fn("contains", left, right)
-
- /**
- * Returns the `n`-th input, e.g., returns `input2` when `n` is 2. The function returns NULL if
- * the index exceeds the length of the array and `spark.sql.ansi.enabled` is set to false. If
- * `spark.sql.ansi.enabled` is set to true, it throws ArrayIndexOutOfBoundsException for invalid
- * indices.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- @scala.annotation.varargs
- def elt(inputs: Column*): Column = Column.fn("elt", inputs: _*)
-
- /**
- * Returns the index (1-based) of the given string (`str`) in the comma-delimited list
- * (`strArray`). Returns 0, if the string was not found or if the given string (`str`) contains
- * a comma.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def find_in_set(str: Column, strArray: Column): Column = Column.fn("find_in_set", str, strArray)
-
- /**
- * Returns true if str matches `pattern` with `escapeChar`, null if any arguments are null,
- * false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def like(str: Column, pattern: Column, escapeChar: Column): Column =
- Column.fn("like", str, pattern, escapeChar)
-
- /**
- * Returns true if str matches `pattern` with `escapeChar`('\'), null if any arguments are null,
- * false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def like(str: Column, pattern: Column): Column = Column.fn("like", str, pattern)
-
- /**
- * Returns true if str matches `pattern` with `escapeChar` case-insensitively, null if any
- * arguments are null, false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def ilike(str: Column, pattern: Column, escapeChar: Column): Column =
- Column.fn("ilike", str, pattern, escapeChar)
-
- /**
- * Returns true if str matches `pattern` with `escapeChar`('\') case-insensitively, null if any
- * arguments are null, false otherwise.
- *
- * @group predicate_funcs
- * @since 3.5.0
- */
- def ilike(str: Column, pattern: Column): Column = Column.fn("ilike", str, pattern)
-
- /**
- * Returns `str` with all characters changed to lowercase.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def lcase(str: Column): Column = Column.fn("lcase", str)
-
- /**
- * Returns `str` with all characters changed to uppercase.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def ucase(str: Column): Column = Column.fn("ucase", str)
-
- /**
- * Returns the leftmost `len`(`len` can be string type) characters from the string `str`, if
- * `len` is less or equal than 0 the result is an empty string.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def left(str: Column, len: Column): Column = Column.fn("left", str, len)
-
- /**
- * Returns the rightmost `len`(`len` can be string type) characters from the string `str`, if
- * `len` is less or equal than 0 the result is an empty string.
- *
- * @group string_funcs
- * @since 3.5.0
- */
- def right(str: Column, len: Column): Column = Column.fn("right", str, len)
-
- /**
- * Masks the given string value. The function replaces characters with 'X' or 'x', and numbers
- * with 'n'. This can be useful for creating copies of tables with sensitive information
- * removed.
- *
- * @param input
- * string value to mask. Supported types: STRING, VARCHAR, CHAR
- * @group string_funcs
- * @since 3.5.0
- */
- def mask(input: Column): Column = Column.fn("mask", input)
-
- /**
- * Masks the given string value. The function replaces upper-case characters with specific
- * character, lower-case characters with 'x', and numbers with 'n'. This can be useful for
- * creating copies of tables with sensitive information removed.
- *
- * @param input
- * string value to mask. Supported types: STRING, VARCHAR, CHAR
- * @param upperChar
- * character to replace upper-case characters with. Specify NULL to retain original character.
- * @group string_funcs
- * @since 3.5.0
- */
- def mask(input: Column, upperChar: Column): Column =
- Column.fn("mask", input, upperChar)
-
- /**
- * Masks the given string value. The function replaces upper-case, lower-case characters with
- * specific characters, and numbers with 'n'. This can be useful for creating copies of tables
- * with sensitive information removed.
- *
- * @param input
- * string value to mask. Supported types: STRING, VARCHAR, CHAR
- * @param upperChar
- * character to replace upper-case characters with. Specify NULL to retain original character.
- * @param lowerChar
- * character to replace lower-case characters with. Specify NULL to retain original character.
- * @group string_funcs
- * @since 3.5.0
- */
- def mask(input: Column, upperChar: Column, lowerChar: Column): Column =
- Column.fn("mask", input, upperChar, lowerChar)
-
- /**
- * Masks the given string value. The function replaces upper-case, lower-case characters and
- * numbers with specific characters. This can be useful for creating copies of tables with
- * sensitive information removed.
- *
- * @param input
- * string value to mask. Supported types: STRING, VARCHAR, CHAR
- * @param upperChar
- * character to replace upper-case characters with. Specify NULL to retain original character.
- * @param lowerChar
- * character to replace lower-case characters with. Specify NULL to retain original character.
- * @param digitChar
- * character to replace digit characters with. Specify NULL to retain original character.
- * @group string_funcs
- * @since 3.5.0
- */
- def mask(input: Column, upperChar: Column, lowerChar: Column, digitChar: Column): Column =
- Column.fn("mask", input, upperChar, lowerChar, digitChar)
-
- /**
- * Masks the given string value. This can be useful for creating copies of tables with sensitive
- * information removed.
- *
- * @param input
- * string value to mask. Supported types: STRING, VARCHAR, CHAR
- * @param upperChar
- * character to replace upper-case characters with. Specify NULL to retain original character.
- * @param lowerChar
- * character to replace lower-case characters with. Specify NULL to retain original character.
- * @param digitChar
- * character to replace digit characters with. Specify NULL to retain original character.
- * @param otherChar
- * character to replace all other characters with. Specify NULL to retain original character.
- * @group string_funcs
- * @since 3.5.0
- */
- def mask(
- input: Column,
- upperChar: Column,
- lowerChar: Column,
- digitChar: Column,
- otherChar: Column): Column =
- Column.fn("mask", input, upperChar, lowerChar, digitChar, otherChar)
-
- /**
- * Marks a given column with specified collation.
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def collate(e: Column, collation: String): Column = Column.fn("collate", e, lit(collation))
-
- /**
- * Returns the collation name of a given column.
- *
- * @group string_funcs
- * @since 4.0.0
- */
- def collation(e: Column): Column = Column.fn("collation", e)
-
- //////////////////////////////////////////////////////////////////////////////////////////////
- // DateTime functions
- //////////////////////////////////////////////////////////////////////////////////////////////
-
- /**
- * Returns the date that is `numMonths` after `startDate`.
- *
- * @param startDate
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param numMonths
- * The number of months to add to `startDate`, can be negative to subtract months
- * @return
- * A date, or null if `startDate` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def add_months(startDate: Column, numMonths: Int): Column =
- add_months(startDate, lit(numMonths))
-
- /**
- * Returns the date that is `numMonths` after `startDate`.
- *
- * @param startDate
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param numMonths
- * A column of the number of months to add to `startDate`, can be negative to subtract months
- * @return
- * A date, or null if `startDate` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def add_months(startDate: Column, numMonths: Column): Column =
- Column.fn("add_months", startDate, numMonths)
-
- /**
- * Returns the current date at the start of query evaluation as a date column. All calls of
- * current_date within the same query return the same value.
- *
- * @group datetime_funcs
- * @since 3.5.0
- */
- def curdate(): Column = Column.fn("curdate")
-
- /**
- * Returns the current date at the start of query evaluation as a date column. All calls of
- * current_date within the same query return the same value.
- *
- * @group datetime_funcs
- * @since 3.4.0
- */
- def current_date(): Column = Column.fn("current_date")
-
- /**
- * Returns the current session local timezone.
- *
- * @group datetime_funcs
- * @since 3.5.0
- */
- def current_timezone(): Column = Column.fn("current_timezone")
-
- /**
- * Returns the current timestamp at the start of query evaluation as a timestamp column. All
- * calls of current_timestamp within the same query return the same value.
- *
- * @group datetime_funcs
- * @since 3.4.0
- */
- def current_timestamp(): Column = Column.fn("current_timestamp")
-
- /**
- * Returns the current timestamp at the start of query evaluation.
- *
- * @group datetime_funcs
- * @since 3.5.0
- */
- def now(): Column = Column.fn("now")
-
- /**
- * Returns the current timestamp without time zone at the start of query evaluation as a
- * timestamp without time zone column. All calls of localtimestamp within the same query return
- * the same value.
- *
- * @group datetime_funcs
- * @since 3.4.0
- */
- def localtimestamp(): Column = Column.fn("localtimestamp")
-
- /**
- * Converts a date/timestamp/string to a value of string in the format specified by the date
- * format given by the second argument.
- *
- * See <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html"> Datetime
- * Patterns</a> for valid date and time format patterns
- *
- * @param dateExpr
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param format
- * A pattern `dd.MM.yyyy` would return a string like `18.03.1993`
- * @return
- * A string, or null if `dateExpr` was a string that could not be cast to a timestamp
- * @note
- * Use specialized functions like [[year]] whenever possible as they benefit from a
- * specialized implementation.
- * @throws IllegalArgumentException
- * if the `format` pattern is invalid
- * @group datetime_funcs
- * @since 3.4.0
- */
- def date_format(dateExpr: Column, format: String): Column =
- Column.fn("date_format", dateExpr, lit(format))
-
- /**
- * Returns the date that is `days` days after `start`
- *
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param days
- * The number of days to add to `start`, can be negative to subtract days
- * @return
- * A date, or null if `start` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def date_add(start: Column, days: Int): Column = date_add(start, lit(days))
-
- /**
- * Returns the date that is `days` days after `start`
- *
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param days
- * A column of the number of days to add to `start`, can be negative to subtract days
- * @return
- * A date, or null if `start` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def date_add(start: Column, days: Column): Column = Column.fn("date_add", start, days)
-
- /**
- * Returns the date that is `days` days after `start`
- *
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param days
- * A column of the number of days to add to `start`, can be negative to subtract days
- * @return
- * A date, or null if `start` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.5.0
- */
- def dateadd(start: Column, days: Column): Column = Column.fn("dateadd", start, days)
-
- /**
- * Returns the date that is `days` days before `start`
- *
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param days
- * The number of days to subtract from `start`, can be negative to add days
- * @return
- * A date, or null if `start` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def date_sub(start: Column, days: Int): Column = date_sub(start, lit(days))
-
- /**
- * Returns the date that is `days` days before `start`
- *
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param days
- * A column of the number of days to subtract from `start`, can be negative to add days
- * @return
- * A date, or null if `start` was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def date_sub(start: Column, days: Column): Column =
- Column.fn("date_sub", start, days)
-
- /**
- * Returns the number of days from `start` to `end`.
- *
- * Only considers the date part of the input. For example:
- * {{{
- * dateddiff("2018-01-10 00:00:00", "2018-01-09 23:59:59")
- * // returns 1
- * }}}
- *
- * @param end
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @return
- * An integer, or null if either `end` or `start` were strings that could not be cast to a
- * date. Negative if `end` is before `start`
- * @group datetime_funcs
- * @since 3.4.0
- */
- def datediff(end: Column, start: Column): Column = Column.fn("datediff", end, start)
-
- /**
- * Returns the number of days from `start` to `end`.
- *
- * Only considers the date part of the input. For example:
- * {{{
- * dateddiff("2018-01-10 00:00:00", "2018-01-09 23:59:59")
- * // returns 1
- * }}}
- *
- * @param end
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @return
- * An integer, or null if either `end` or `start` were strings that could not be cast to a
- * date. Negative if `end` is before `start`
- * @group datetime_funcs
- * @since 3.5.0
- */
- def date_diff(end: Column, start: Column): Column = Column.fn("date_diff", end, start)
-
- /**
- * Create date from the number of `days` since 1970-01-01.
- *
- * @group datetime_funcs
- * @since 3.5.0
- */
- def date_from_unix_date(days: Column): Column = Column.fn("date_from_unix_date", days)
-
- /**
- * Extracts the year as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def year(e: Column): Column = Column.fn("year", e)
-
- /**
- * Extracts the quarter as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def quarter(e: Column): Column = Column.fn("quarter", e)
-
- /**
- * Extracts the month as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def month(e: Column): Column = Column.fn("month", e)
-
- /**
- * Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1
- * for a Sunday through to 7 for a Saturday
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def dayofweek(e: Column): Column = Column.fn("dayofweek", e)
-
- /**
- * Extracts the day of the month as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def dayofmonth(e: Column): Column = Column.fn("dayofmonth", e)
-
- /**
- * Extracts the day of the month as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.5.0
- */
- def day(e: Column): Column = Column.fn("day", e)
-
- /**
- * Extracts the day of the year as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def dayofyear(e: Column): Column = Column.fn("dayofyear", e)
-
- /**
- * Extracts the hours as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def hour(e: Column): Column = Column.fn("hour", e)
-
- /**
- * Extracts a part of the date/timestamp or interval source.
- *
- * @param field
- * selects which part of the source should be extracted.
- * @param source
- * a date/timestamp or interval column from where `field` should be extracted.
- * @return
- * a part of the date/timestamp or interval source
- * @group datetime_funcs
- * @since 3.5.0
- */
- def extract(field: Column, source: Column): Column = {
- Column.fn("extract", field, source)
- }
-
- /**
- * Extracts a part of the date/timestamp or interval source.
- *
- * @param field
- * selects which part of the source should be extracted, and supported string values are as
- * same as the fields of the equivalent function `extract`.
- * @param source
- * a date/timestamp or interval column from where `field` should be extracted.
- * @return
- * a part of the date/timestamp or interval source
- * @group datetime_funcs
- * @since 3.5.0
- */
- def date_part(field: Column, source: Column): Column = {
- Column.fn("date_part", field, source)
- }
-
- /**
- * Extracts a part of the date/timestamp or interval source.
- *
- * @param field
- * selects which part of the source should be extracted, and supported string values are as
- * same as the fields of the equivalent function `extract`.
- * @param source
- * a date/timestamp or interval column from where `field` should be extracted.
- * @return
- * a part of the date/timestamp or interval source
- * @group datetime_funcs
- * @since 3.5.0
- */
- def datepart(field: Column, source: Column): Column = {
- Column.fn("datepart", field, source)
- }
-
- /**
- * Returns the last day of the month which the given date belongs to. For example, input
- * "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015.
- *
- * @param e
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @return
- * A date, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def last_day(e: Column): Column = Column.fn("last_day", e)
-
- /**
- * Extracts the minutes as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def minute(e: Column): Column = Column.fn("minute", e)
-
- /**
- * Returns the day of the week for date/timestamp (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).
- *
- * @group datetime_funcs
- * @since 3.5.0
- */
- def weekday(e: Column): Column = Column.fn("weekday", e)
-
- /**
- * @return
- * A date created from year, month and day fields.
- * @group datetime_funcs
- * @since 3.4.0
- */
- def make_date(year: Column, month: Column, day: Column): Column =
- Column.fn("make_date", year, month, day)
-
- /**
- * Returns number of months between dates `start` and `end`.
- *
- * A whole number is returned if both inputs have the same day of month or both are the last day
- * of their respective months. Otherwise, the difference is calculated assuming 31 days per
- * month.
- *
- * For example:
- * {{{
- * months_between("2017-11-14", "2017-07-14") // returns 4.0
- * months_between("2017-01-01", "2017-01-10") // returns 0.29032258
- * months_between("2017-06-01", "2017-06-16 12:00:00") // returns -0.5
- * }}}
- *
- * @param end
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param start
- * A date, timestamp or string. If a string, the data must be in a format that can cast to a
- * timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @return
- * A double, or null if either `end` or `start` were strings that could not be cast to a
- * timestamp. Negative if `end` is before `start`
- * @group datetime_funcs
- * @since 3.4.0
- */
- def months_between(end: Column, start: Column): Column =
- Column.fn("months_between", end, start)
-
- /**
- * Returns number of months between dates `end` and `start`. If `roundOff` is set to true, the
- * result is rounded off to 8 digits; it is not rounded otherwise.
- * @group datetime_funcs
- * @since 3.4.0
- */
- def months_between(end: Column, start: Column, roundOff: Boolean): Column =
- Column.fn("months_between", end, start, lit(roundOff))
-
- /**
- * Returns the first date which is later than the value of the `date` column that is on the
- * specified day of the week.
- *
- * For example, `next_day('2015-07-27', "Sunday")` returns 2015-08-02 because that is the first
- * Sunday after 2015-07-27.
- *
- * @param date
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param dayOfWeek
- * Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"
- * @return
- * A date, or null if `date` was a string that could not be cast to a date or if `dayOfWeek`
- * was an invalid value
- * @group datetime_funcs
- * @since 3.4.0
- */
- def next_day(date: Column, dayOfWeek: String): Column = next_day(date, lit(dayOfWeek))
-
- /**
- * Returns the first date which is later than the value of the `date` column that is on the
- * specified day of the week.
- *
- * For example, `next_day('2015-07-27', "Sunday")` returns 2015-08-02 because that is the first
- * Sunday after 2015-07-27.
- *
- * @param date
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param dayOfWeek
- * A column of the day of week. Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu",
- * "Fri", "Sat", "Sun"
- * @return
- * A date, or null if `date` was a string that could not be cast to a date or if `dayOfWeek`
- * was an invalid value
- * @group datetime_funcs
- * @since 3.4.0
- */
- def next_day(date: Column, dayOfWeek: Column): Column =
- Column.fn("next_day", date, dayOfWeek)
-
- /**
- * Extracts the seconds as an integer from a given date/timestamp/string.
- * @return
- * An integer, or null if the input was a string that could not be cast to a timestamp
- * @group datetime_funcs
- * @since 3.4.0
- */
- def second(e: Column): Column = Column.fn("second", e)
-
- /**
- * Extracts the week number as an integer from a given date/timestamp/string.
- *
- * A week is considered to start on a Monday and week 1 is the first week with more than 3 days,
- * as defined by ISO 8601
- *
- * @return
- * An integer, or null if the input was a string that could not be cast to a date
- * @group datetime_funcs
- * @since 3.4.0
- */
- def weekofyear(e: Column): Column = Column.fn("weekofyear", e)
-
- /**
- * Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string
- * representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd
- * HH:mm:ss format.
- *
- * @param ut
- * A number of a type that is castable to a long, such as string or integer. Can be negative
- * for timestamps before the unix epoch
- * @return
- * A string, or null if the input was a string that could not be cast to a long
- * @group datetime_funcs
- * @since 3.4.0
- */
- def from_unixtime(ut: Column): Column = Column.fn("from_unixtime", ut)
-
- /**
- * Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string
- * representing the timestamp of that moment in the current system time zone in the given
- * format.
- *
- * See <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html"> Datetime
- * Patterns</a> for valid date and time format patterns
- *
- * @param ut
- * A number of a type that is castable to a long, such as string or integer. Can be negative
- * for timestamps before the unix epoch
- * @param f
- * A date time pattern that the input will be formatted to
- * @return
- * A string, or null if `ut` was a string that could not be cast to a long or `f` was an
- * invalid date time pattern
- * @group datetime_funcs
- * @since 3.4.0
- */
- def from_unixtime(ut: Column, f: String): Column =
- Column.fn("from_unixtime", ut, lit(f))
-
- /**
- * Returns the current Unix timestamp (in seconds) as a long.
- *
- * @note
- * All calls of `unix_timestamp` within the same query return the same value (i.e. the current
- * timestamp is calculated at the start of query evaluation).
- *
- * @group datetime_funcs
- * @since 3.4.0
- */
- def unix_timestamp(): Column = unix_timestamp(current_timestamp())
-
- /**
- * Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the
- * default timezone and the default locale.
- *
- * @param s
- * A date, timestamp or string. If a string, the data must be in the `yyyy-MM-dd HH:mm:ss`
- * format
- * @return
- * A long, or null if the input was a string not of the correct format
- * @group datetime_funcs
- * @since 3.4.0
- */
- def unix_timestamp(s: Column): Column = Column.fn("unix_timestamp", s)
-
- /**
- * Converts time string with given pattern to Unix timestamp (in seconds).
- *
- * See <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html"> Datetime
- * Patterns</a> for valid date and time format patterns
- *
- * @param s
- * A date, timestamp or string. If a string, the data must be in a format that can be cast to
- * a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
- * @param p
- * A date time pattern detailing the format of `s` when `s` is a string
- * @return
- * A long, or null if `s` was a string that could not be cast to a date or `p` was an invalid
- * format
- * @group datetime_funcs
... 6905 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org