You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/01 15:35:12 UTC
(datafusion) 01/01: added support for MapFromEntries (#21720)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-21720-a8761a625dc72c979e7156cdd21a6f335a189f96
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit f46a4a431b6c46d1d0e53f00c088002c05adbd95
Author: Krishna Sudarshan J <75...@users.noreply.github.com>
AuthorDate: Mon Jun 1 21:04:49 2026 +0530
added support for MapFromEntries (#21720)
## Which issue does this PR close?
Closes # (none: prerequisite for
https://github.com/apache/datafusion-comet/issues/2706; follow-up to
#17779 and #19274).
## Rationale for this change
Spark's default mapKeyDedupPolicy is EXCEPTION, raising
SparkRuntimeException with error class DUPLICATED_MAP_KEY on duplicate
map keys. The existing Spark map_from_entries / map_from_arrays UDFs
silently kept the last occurrence, forcing downstream engines like
datafusion-comet to fall back to Spark.
## What changes are included in this PR?
Duplicate map keys in Spark map_from_entries, map_from_arrays, and
str_to_map now raise [DUPLICATED_MAP_KEY] Duplicate map key {key} was
found, matching Spark's default behaviour and error class.
## Are these changes tested?
Yes, via sqllogictest assertions covering the new error across the
affected Spark map UDFs.
## Are there any user-facing changes?
Yes. Duplicate keys now raise [DUPLICATED_MAP_KEY] under the default
policy instead of silently collapsing to the last occurrence. No new
config keys, no API changes.
---------
Co-authored-by: Krishna Sudarshan J <75...@users.noreply.github.com>
---
datafusion/common/src/config.rs | 70 +++++++
.../spark/src/function/map/map_from_arrays.rs | 11 +-
.../spark/src/function/map/map_from_entries.rs | 11 +-
datafusion/spark/src/function/map/str_to_map.rs | 116 +++++++----
datafusion/spark/src/function/map/utils.rs | 220 +++++++++++++++++----
.../sqllogictest/test_files/information_schema.slt | 2 +
.../test_files/spark/map/map_from_arrays.slt | 59 +++++-
.../test_files/spark/map/map_from_entries.slt | 59 +++++-
.../test_files/spark/map/str_to_map.slt | 53 ++++-
docs/source/library-user-guide/upgrading/55.0.0.md | 31 +++
docs/source/user-guide/configs.md | 1 +
11 files changed, 545 insertions(+), 88 deletions(-)
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 9d960e3bf6..e4a3cea709 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -456,6 +456,53 @@ impl Display for SpillCompression {
}
}
+/// Policy for handling duplicate keys in Spark-compatible map-construction
+/// functions (`map_from_arrays`, `map_from_entries`, `str_to_map`). Mirrors
+/// Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961).
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub enum MapKeyDedupPolicy {
+ /// Raise `[DUPLICATED_MAP_KEY]` at runtime on any duplicate key.
+ #[default]
+ Exception,
+ /// Keep the last occurrence of each duplicate key.
+ LastWin,
+}
+
+impl FromStr for MapKeyDedupPolicy {
+ type Err = DataFusionError;
+
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
+ match s.to_ascii_uppercase().as_str() {
+ "EXCEPTION" => Ok(Self::Exception),
+ "LAST_WIN" => Ok(Self::LastWin),
+ other => Err(DataFusionError::Configuration(format!(
+ "Invalid MapKeyDedupPolicy: {other}. Expected one of: EXCEPTION, LAST_WIN"
+ ))),
+ }
+ }
+}
+
+impl ConfigField for MapKeyDedupPolicy {
+ fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
+ v.some(key, self, description)
+ }
+
+ fn set(&mut self, _: &str, value: &str) -> Result<()> {
+ *self = MapKeyDedupPolicy::from_str(value)?;
+ Ok(())
+ }
+}
+
+impl Display for MapKeyDedupPolicy {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let str = match self {
+ Self::Exception => "EXCEPTION",
+ Self::LastWin => "LAST_WIN",
+ };
+ write!(f, "{str}")
+ }
+}
+
impl From<SpillCompression> for Option<CompressionType> {
fn from(c: SpillCompression) -> Self {
match c {
@@ -1499,6 +1546,24 @@ impl<'a> TryFrom<&'a FormatOptions> for arrow::util::display::FormatOptions<'a>
}
}
+config_namespace! {
+ /// Options controlling DataFusion's Spark-compatibility layer (functions
+ /// under `datafusion/spark`). Keys here mirror their `spark.sql.*`
+ /// equivalents in Apache Spark.
+ pub struct SparkOptions {
+ /// Policy for handling duplicate keys in Spark-compatible map-construction
+ /// functions (`map_from_arrays`, `map_from_entries`, `str_to_map`).
+ ///
+ /// Mirrors Spark's
+ /// [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961):
+ /// - `EXCEPTION` (default): raise `[DUPLICATED_MAP_KEY]` at runtime on any duplicate key.
+ /// - `LAST_WIN`: keep the last occurrence of each duplicate key.
+ ///
+ /// Values are case-insensitive.
+ pub map_key_dedup_policy: MapKeyDedupPolicy, default = MapKeyDedupPolicy::Exception
+ }
+}
+
/// A key value pair, with a corresponding description
#[derive(Debug, Clone, Hash, PartialEq, Eq)]
pub struct ConfigEntry {
@@ -1530,6 +1595,8 @@ pub struct ConfigOptions {
pub extensions: Extensions,
/// Formatting options when printing batches
pub format: FormatOptions,
+ /// Spark-compatibility options (functions under `datafusion/spark`)
+ pub spark: SparkOptions,
}
impl ConfigField for ConfigOptions {
@@ -1540,6 +1607,7 @@ impl ConfigField for ConfigOptions {
self.explain.visit(v, "datafusion.explain", "");
self.sql_parser.visit(v, "datafusion.sql_parser", "");
self.format.visit(v, "datafusion.format", "");
+ self.spark.visit(v, "datafusion.spark", "");
}
fn set(&mut self, key: &str, value: &str) -> Result<()> {
@@ -1552,6 +1620,7 @@ impl ConfigField for ConfigOptions {
"explain" => self.explain.set(rem, value),
"sql_parser" => self.sql_parser.set(rem, value),
"format" => self.format.set(rem, value),
+ "spark" => self.spark.set(rem, value),
_ => _config_err!("Config value \"{key}\" not found on ConfigOptions"),
}
}
@@ -1591,6 +1660,7 @@ impl ConfigField for ConfigOptions {
"explain" => self.explain.reset(rem),
"sql_parser" => self.sql_parser.reset(rem),
"format" => self.format.reset(rem),
+ "spark" => self.spark.reset(rem),
other => _config_err!("Config value \"{other}\" not found on ConfigOptions"),
}
}
diff --git a/datafusion/spark/src/function/map/map_from_arrays.rs b/datafusion/spark/src/function/map/map_from_arrays.rs
index 692e837d00..92dea2720f 100644
--- a/datafusion/spark/src/function/map/map_from_arrays.rs
+++ b/datafusion/spark/src/function/map/map_from_arrays.rs
@@ -22,6 +22,7 @@ use crate::function::map::utils::{
use arrow::array::{Array, ArrayRef, NullArray};
use arrow::compute::kernels::cast;
use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::config::MapKeyDedupPolicy;
use datafusion_common::utils::take_function_args;
use datafusion_common::{Result, internal_err};
use datafusion_expr::{
@@ -81,11 +82,16 @@ impl ScalarUDFImpl for MapFromArrays {
}
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
- make_scalar_function(map_from_arrays_inner, vec![])(&args.args)
+ let last_value_wins =
+ args.config_options.spark.map_key_dedup_policy == MapKeyDedupPolicy::LastWin;
+ make_scalar_function(
+ move |args: &[ArrayRef]| map_from_arrays_inner(args, last_value_wins),
+ vec![],
+ )(&args.args)
}
}
-fn map_from_arrays_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
+fn map_from_arrays_inner(args: &[ArrayRef], last_value_wins: bool) -> Result<ArrayRef> {
let [keys, values] = take_function_args("map_from_arrays", args)?;
if *keys.data_type() == DataType::Null || *values.data_type() == DataType::Null {
@@ -105,6 +111,7 @@ fn map_from_arrays_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
&get_list_offsets(values)?,
keys.nulls(),
values.nulls(),
+ last_value_wins,
)
}
diff --git a/datafusion/spark/src/function/map/map_from_entries.rs b/datafusion/spark/src/function/map/map_from_entries.rs
index facf9f8c53..69ce352694 100644
--- a/datafusion/spark/src/function/map/map_from_entries.rs
+++ b/datafusion/spark/src/function/map/map_from_entries.rs
@@ -24,6 +24,7 @@ use crate::function::map::utils::{
use arrow::array::{Array, ArrayRef, NullBufferBuilder, StructArray};
use arrow::buffer::NullBuffer;
use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::config::MapKeyDedupPolicy;
use datafusion_common::utils::take_function_args;
use datafusion_common::{Result, exec_err, internal_err};
use datafusion_expr::{
@@ -101,11 +102,16 @@ impl ScalarUDFImpl for MapFromEntries {
}
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
- make_scalar_function(map_from_entries_inner, vec![])(&args.args)
+ let last_value_wins =
+ args.config_options.spark.map_key_dedup_policy == MapKeyDedupPolicy::LastWin;
+ make_scalar_function(
+ move |args: &[ArrayRef]| map_from_entries_inner(args, last_value_wins),
+ vec![],
+ )(&args.args)
}
}
-fn map_from_entries_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
+fn map_from_entries_inner(args: &[ArrayRef], last_value_wins: bool) -> Result<ArrayRef> {
let [entries] = take_function_args("map_from_entries", args)?;
let entries_offsets = get_list_offsets(entries)?;
let entries_values = get_list_values(entries)?;
@@ -148,6 +154,7 @@ fn map_from_entries_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
&entries_offsets,
None,
res_nulls.as_ref(),
+ last_value_wins,
)
}
diff --git a/datafusion/spark/src/function/map/str_to_map.rs b/datafusion/spark/src/function/map/str_to_map.rs
index d0f4cf03cd..abb4bd0476 100644
--- a/datafusion/spark/src/function/map/str_to_map.rs
+++ b/datafusion/spark/src/function/map/str_to_map.rs
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use arrow::array::{
@@ -33,6 +33,7 @@ use datafusion_expr::{
};
use crate::function::map::utils::map_type_from_key_value_types;
+use datafusion_common::config::MapKeyDedupPolicy;
const DEFAULT_PAIR_DELIM: &str = ",";
const DEFAULT_KV_DELIM: &str = ":";
@@ -48,11 +49,10 @@ const DEFAULT_KV_DELIM: &str = ":";
/// - keyValueDelim: Delimiter between key and value (default: ':')
///
/// # Duplicate Key Handling
-/// Uses EXCEPTION behavior (Spark 3.0+ default): errors on duplicate keys.
-/// See `spark.sql.mapKeyDedupPolicy`:
-/// <https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4502-L4511>
-///
-/// TODO: Support configurable `spark.sql.mapKeyDedupPolicy` (LAST_WIN) in a follow-up PR.
+/// Mirrors Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4502-L4511),
+/// wired through DataFusion's `datafusion.spark.map_key_dedup_policy`:
+/// - `EXCEPTION` (default): error on duplicate keys.
+/// - `LAST_WIN`: keep the last occurrence of each duplicate key.
#[derive(Debug, PartialEq, Eq, Hash)]
pub struct SparkStrToMap {
signature: Signature,
@@ -102,22 +102,32 @@ impl ScalarUDFImpl for SparkStrToMap {
}
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let last_value_wins =
+ args.config_options.spark.map_key_dedup_policy == MapKeyDedupPolicy::LastWin;
let arrays: Vec<ArrayRef> = ColumnarValue::values_to_arrays(&args.args)?;
- let result = str_to_map_inner(&arrays)?;
+ let result = str_to_map_inner(&arrays, last_value_wins)?;
Ok(ColumnarValue::Array(result))
}
}
-fn str_to_map_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
+fn str_to_map_inner(args: &[ArrayRef], last_value_wins: bool) -> Result<ArrayRef> {
match args.len() {
1 => match args[0].data_type() {
- DataType::Utf8 => str_to_map_impl(as_string_array(&args[0])?, None, None),
- DataType::LargeUtf8 => {
- str_to_map_impl(as_large_string_array(&args[0])?, None, None)
- }
- DataType::Utf8View => {
- str_to_map_impl(as_string_view_array(&args[0])?, None, None)
+ DataType::Utf8 => {
+ str_to_map_impl(as_string_array(&args[0])?, None, None, last_value_wins)
}
+ DataType::LargeUtf8 => str_to_map_impl(
+ as_large_string_array(&args[0])?,
+ None,
+ None,
+ last_value_wins,
+ ),
+ DataType::Utf8View => str_to_map_impl(
+ as_string_view_array(&args[0])?,
+ None,
+ None,
+ last_value_wins,
+ ),
other => exec_err!(
"Unsupported data type {other:?} for str_to_map, \
expected Utf8, LargeUtf8, or Utf8View"
@@ -128,16 +138,19 @@ fn str_to_map_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
as_string_array(&args[0])?,
Some(as_string_array(&args[1])?),
None,
+ last_value_wins,
),
(DataType::LargeUtf8, DataType::LargeUtf8) => str_to_map_impl(
as_large_string_array(&args[0])?,
Some(as_large_string_array(&args[1])?),
None,
+ last_value_wins,
),
(DataType::Utf8View, DataType::Utf8View) => str_to_map_impl(
as_string_view_array(&args[0])?,
Some(as_string_view_array(&args[1])?),
None,
+ last_value_wins,
),
(t1, t2) => exec_err!(
"Unsupported data types ({t1:?}, {t2:?}) for str_to_map, \
@@ -153,12 +166,14 @@ fn str_to_map_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
as_string_array(&args[0])?,
Some(as_string_array(&args[1])?),
Some(as_string_array(&args[2])?),
+ last_value_wins,
),
(DataType::LargeUtf8, DataType::LargeUtf8, DataType::LargeUtf8) => {
str_to_map_impl(
as_large_string_array(&args[0])?,
Some(as_large_string_array(&args[1])?),
Some(as_large_string_array(&args[2])?),
+ last_value_wins,
)
}
(DataType::Utf8View, DataType::Utf8View, DataType::Utf8View) => {
@@ -166,6 +181,7 @@ fn str_to_map_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
as_string_view_array(&args[0])?,
Some(as_string_view_array(&args[1])?),
Some(as_string_view_array(&args[2])?),
+ last_value_wins,
)
}
(t1, t2, t3) => exec_err!(
@@ -181,6 +197,7 @@ fn str_to_map_impl<'a, V: StringArrayType<'a> + Copy>(
text_array: V,
pair_delim_array: Option<V>,
kv_delim_array: Option<V>,
+ last_value_wins: bool,
) -> Result<ArrayRef> {
let num_rows = text_array.len();
@@ -206,6 +223,10 @@ fn str_to_map_impl<'a, V: StringArrayType<'a> + Copy>(
);
let mut seen_keys = HashSet::new();
+ // LAST_WIN buffers pairs to support in-place value overwrite at the key's
+ // first-seen position — matches Spark's `ArrayBasedMapBuilder`.
+ let mut pairs: Vec<(&str, Option<&str>)> = Vec::new();
+ let mut key_positions: HashMap<&str, usize> = HashMap::new();
for row_idx in 0..num_rows {
if combined_nulls.as_ref().is_some_and(|n| n.is_null(row_idx)) {
map_builder.append(false)?;
@@ -226,31 +247,56 @@ fn str_to_map_impl<'a, V: StringArrayType<'a> + Copy>(
continue;
}
- seen_keys.clear();
- for pair in text.split(pair_delim) {
- if pair.is_empty() {
- continue;
+ if last_value_wins {
+ pairs.clear();
+ key_positions.clear();
+ for pair in text.split(pair_delim) {
+ if pair.is_empty() {
+ continue;
+ }
+ let mut kv_iter = pair.splitn(2, kv_delim);
+ let key = kv_iter.next().unwrap_or("");
+ let value = kv_iter.next();
+ match key_positions.get(key) {
+ Some(&idx) => pairs[idx].1 = value,
+ None => {
+ key_positions.insert(key, pairs.len());
+ pairs.push((key, value));
+ }
+ }
+ }
+ for (key, value) in &pairs {
+ map_builder.keys().append_value(key);
+ match value {
+ Some(v) => map_builder.values().append_value(v),
+ None => map_builder.values().append_null(),
+ }
}
+ } else {
+ seen_keys.clear();
+ for pair in text.split(pair_delim) {
+ if pair.is_empty() {
+ continue;
+ }
- let mut kv_iter = pair.splitn(2, kv_delim);
- let key = kv_iter.next().unwrap_or("");
- let value = kv_iter.next();
+ let mut kv_iter = pair.splitn(2, kv_delim);
+ let key = kv_iter.next().unwrap_or("");
+ let value = kv_iter.next();
- // TODO: Support LAST_WIN policy via spark.sql.mapKeyDedupPolicy config
- // EXCEPTION policy: error on duplicate keys (Spark 3.0+ default)
- if !seen_keys.insert(key) {
- return exec_err!(
- "Duplicate map key '{key}' was found, please check the input data. \
- If you want to remove the duplicated keys, you can set \
- spark.sql.mapKeyDedupPolicy to \"LAST_WIN\" so that the key \
- inserted at last takes precedence."
- );
- }
+ if !seen_keys.insert(key) {
+ return exec_err!(
+ "[DUPLICATED_MAP_KEY] Duplicate map key '{key}' was found, \
+ please check the input data. To allow duplicate keys with \
+ last-value-wins semantics, set \
+ `datafusion.spark.map_key_dedup_policy` to `LAST_WIN`."
+ );
+ }
- map_builder.keys().append_value(key);
- match value {
- Some(v) => map_builder.values().append_value(v),
- None => map_builder.values().append_null(),
+ map_builder.keys().append_value(key);
+ match value {
+ Some(v) => map_builder.values().append_value(v),
+ None => map_builder.values().append_null(),
+ }
}
}
map_builder.append(true)?;
diff --git a/datafusion/spark/src/function/map/utils.rs b/datafusion/spark/src/function/map/utils.rs
index f5fff0c4b4..fa6b2a960d 100644
--- a/datafusion/spark/src/function/map/utils.rs
+++ b/datafusion/spark/src/function/map/utils.rs
@@ -16,12 +16,14 @@
// under the License.
use std::borrow::Cow;
-use std::collections::HashSet;
+use std::collections::HashMap;
use std::sync::Arc;
-use arrow::array::{Array, ArrayRef, AsArray, BooleanBuilder, MapArray, StructArray};
+use arrow::array::{
+ Array, ArrayRef, AsArray, BooleanBuilder, Int32Array, MapArray, StructArray,
+};
use arrow::buffer::{NullBuffer, OffsetBuffer};
-use arrow::compute::filter;
+use arrow::compute::{filter, take};
use arrow::datatypes::{DataType, Field, Fields};
use datafusion_common::{Result, ScalarValue, exec_err};
@@ -111,13 +113,13 @@ pub fn map_type_from_key_value_types(
/// So the inputs can be [`ListArray`](`arrow::array::ListArray`)/[`LargeListArray`](`arrow::array::LargeListArray`)/[`FixedSizeListArray`](`arrow::array::FixedSizeListArray`)<br>
/// To preserve the row info, [`offsets`](arrow::array::ListArray::offsets) and [`nulls`](arrow::array::ListArray::nulls) for both keys and values need to be provided<br>
/// [`FixedSizeListArray`](`arrow::array::FixedSizeListArray`) has no `offsets`, so they can be generated as a cumulative sum of it's `Size`
-/// 2. Spark provides [spark.sql.mapKeyDedupPolicy](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961)
-/// to handle duplicate keys<br>
-/// For now, configurable functions are not supported by Datafusion<br>
-/// So more permissive `LAST_WIN` option is used in this implementation (instead of `EXCEPTION`)<br>
-/// `EXCEPTION` behaviour can still be achieved externally in cost of performance:<br>
-/// `when(array_length(array_distinct(keys)) == array_length(keys), constructed_map)`<br>
-/// `.otherwise(raise_error("duplicate keys occurred during map construction"))`
+/// 2. Duplicate-key handling mirrors Spark's
+/// [spark.sql.mapKeyDedupPolicy](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961)
+/// and is driven by `last_value_wins`:
+/// - `false` (Spark's default `EXCEPTION`): raise `[DUPLICATED_MAP_KEY]` on any duplicate.
+/// - `true` (`LAST_WIN`): keep the last occurrence of each duplicate key.
+///
+/// Callers wire this from `datafusion.spark.map_key_dedup_policy`.
pub fn map_from_keys_values_offsets_nulls(
flat_keys: &ArrayRef,
flat_values: &ArrayRef,
@@ -125,6 +127,7 @@ pub fn map_from_keys_values_offsets_nulls(
values_offsets: &[i32],
keys_nulls: Option<&NullBuffer>,
values_nulls: Option<&NullBuffer>,
+ last_value_wins: bool,
) -> Result<ArrayRef> {
let (keys, values, offsets) = map_deduplicate_keys(
flat_keys,
@@ -133,6 +136,7 @@ pub fn map_from_keys_values_offsets_nulls(
values_offsets,
keys_nulls,
values_nulls,
+ last_value_wins,
)?;
let nulls = NullBuffer::union(keys_nulls, values_nulls);
@@ -155,6 +159,7 @@ fn map_deduplicate_keys(
values_offsets: &[i32],
keys_nulls: Option<&NullBuffer>,
values_nulls: Option<&NullBuffer>,
+ last_value_wins: bool,
) -> Result<(ArrayRef, ArrayRef, OffsetBuffer<i32>)> {
let offsets_len = keys_offsets.len();
let mut new_offsets = Vec::with_capacity(offsets_len);
@@ -171,8 +176,14 @@ fn map_deduplicate_keys(
let mut new_last_offset = 0;
new_offsets.push(new_last_offset);
+ // Mirror Spark's `ArrayBasedMapBuilder`: the first occurrence of a key
+ // fixes its position in the output; under LAST_WIN a later duplicate
+ // overwrites that slot's value. `keys_mask` selects the first-seen keys,
+ // `value_indices` records the source index in `flat_values` to materialize
+ // for each output slot (updated in place on overwrite).
let mut keys_mask_builder = BooleanBuilder::new();
- let mut values_mask_builder = BooleanBuilder::new();
+ let mut value_indices: Vec<i32> = Vec::new();
+ let mut key_to_output_idx: HashMap<ScalarValue, usize> = HashMap::new();
for (row_idx, (next_keys_offset, next_values_offset)) in keys_offsets
.iter()
.zip(values_offsets.iter())
@@ -182,9 +193,6 @@ fn map_deduplicate_keys(
let num_keys_entries = *next_keys_offset as usize - cur_keys_offset;
let num_values_entries = *next_values_offset as usize - cur_values_offset;
- let mut keys_mask_one = vec![false; num_keys_entries];
- let mut values_mask_one = vec![false; num_values_entries];
-
let key_is_valid = keys_nulls.is_none_or(|buf| buf.is_valid(row_idx));
let value_is_valid = values_nulls.is_none_or(|buf| buf.is_valid(row_idx));
@@ -193,43 +201,175 @@ fn map_deduplicate_keys(
return exec_err!(
"map_deduplicate_keys: keys and values lists in the same row must have equal lengths"
);
- } else if num_keys_entries != 0 {
- let mut seen_keys = HashSet::new();
-
- for cur_entry_idx in (0..num_keys_entries).rev() {
- let key = ScalarValue::try_from_array(
- &flat_keys,
- cur_keys_offset + cur_entry_idx,
- )?
- .compacted();
- if seen_keys.contains(&key) {
- // TODO: implement configuration and logic for spark.sql.mapKeyDedupPolicy=EXCEPTION (this is default spark-config)
- // exec_err!("invalid argument: duplicate keys in map")
- // https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961
- } else {
- // This code implements deduplication logic for spark.sql.mapKeyDedupPolicy=LAST_WIN (this is NOT default spark-config)
- keys_mask_one[cur_entry_idx] = true;
- values_mask_one[cur_entry_idx] = true;
- seen_keys.insert(key);
- new_last_offset += 1;
+ }
+ key_to_output_idx.clear();
+ for cur_entry_idx in 0..num_keys_entries {
+ let key = ScalarValue::try_from_array(
+ &flat_keys,
+ cur_keys_offset + cur_entry_idx,
+ )?
+ .compacted();
+ let abs_value_idx = (cur_values_offset + cur_entry_idx) as i32;
+
+ if let Some(&output_idx) = key_to_output_idx.get(&key) {
+ if last_value_wins {
+ value_indices[output_idx] = abs_value_idx;
+ keys_mask_builder.append_value(false);
+ continue;
}
+ return exec_err!(
+ "[DUPLICATED_MAP_KEY] Duplicate map key {key} was found, \
+ please check the input data. To allow duplicate keys with \
+ last-value-wins semantics, set \
+ `datafusion.spark.map_key_dedup_policy` to `LAST_WIN`."
+ );
}
+ keys_mask_builder.append_value(true);
+ key_to_output_idx.insert(key, value_indices.len());
+ value_indices.push(abs_value_idx);
+ new_last_offset += 1;
}
} else {
- // the result entry is NULL
- // both current row offsets are skipped
- // keys or values in the current row are marked false in the masks
+ // The result entry is NULL — no keys/values emitted. Still pad the
+ // mask so it stays aligned with `flat_keys`.
+ keys_mask_builder.append_n(num_keys_entries, false);
}
- keys_mask_builder.append_array(&keys_mask_one.into());
- values_mask_builder.append_array(&values_mask_one.into());
new_offsets.push(new_last_offset);
cur_keys_offset += num_keys_entries;
cur_values_offset += num_values_entries;
}
let keys_mask = keys_mask_builder.finish();
- let values_mask = values_mask_builder.finish();
let needed_keys = filter(&flat_keys, &keys_mask)?;
- let needed_values = filter(&flat_values, &values_mask)?;
+ let value_indices_array = Int32Array::from(value_indices);
+ let needed_values = take(&flat_values, &value_indices_array, None)?;
let offsets = OffsetBuffer::new(new_offsets.into());
Ok((needed_keys, needed_values, offsets))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow::array::{Int32Array, StringArray};
+
+ fn int32_utf8_inputs(
+ keys: Vec<i32>,
+ values: Vec<Option<&str>>,
+ ) -> (ArrayRef, ArrayRef) {
+ let keys: ArrayRef = Arc::new(Int32Array::from(keys));
+ let values: ArrayRef = Arc::new(StringArray::from(values));
+ (keys, values)
+ }
+
+ #[test]
+ fn happy_path_two_rows_no_duplicates() {
+ let (keys, values) =
+ int32_utf8_inputs(vec![1, 2, 3], vec![Some("a"), Some("b"), Some("c")]);
+ let offsets = [0i32, 2, 3];
+
+ let result = map_from_keys_values_offsets_nulls(
+ &keys, &values, &offsets, &offsets, None, None, false,
+ )
+ .unwrap();
+
+ let map = result.as_map();
+ assert_eq!(map.len(), 2);
+ assert_eq!(map.value_offsets(), &[0, 2, 3]);
+ }
+
+ #[test]
+ fn single_row_duplicate_errors_under_exception() {
+ let (keys, values) =
+ int32_utf8_inputs(vec![1, 2, 1], vec![Some("a"), Some("b"), Some("c")]);
+ let offsets = [0i32, 3];
+
+ let err = map_from_keys_values_offsets_nulls(
+ &keys, &values, &offsets, &offsets, None, None, false,
+ )
+ .unwrap_err()
+ .to_string();
+
+ assert!(err.contains("[DUPLICATED_MAP_KEY]"), "{err}");
+ assert!(err.contains("map_key_dedup_policy"), "{err}");
+ }
+
+ #[test]
+ fn last_win_keeps_final_occurrence() {
+ let (keys, values) = int32_utf8_inputs(
+ vec![1, 2, 1, 3, 2],
+ vec![Some("a"), Some("b"), Some("c"), Some("d"), Some("e")],
+ );
+ let offsets = [0i32, 5];
+
+ let result = map_from_keys_values_offsets_nulls(
+ &keys, &values, &offsets, &offsets, None, None, true,
+ )
+ .unwrap();
+
+ let map = result.as_map();
+ assert_eq!(map.len(), 1);
+ // 5 entries in, 3 unique keys -> offsets [0, 3]
+ assert_eq!(map.value_offsets(), &[0, 3]);
+ }
+
+ #[test]
+ fn duplicate_in_later_row_still_errors() {
+ let (keys, values) = int32_utf8_inputs(
+ vec![1, 2, 1, 1],
+ vec![Some("a"), Some("b"), Some("x"), Some("y")],
+ );
+ let offsets = [0i32, 2, 4];
+
+ let err = map_from_keys_values_offsets_nulls(
+ &keys, &values, &offsets, &offsets, None, None, false,
+ )
+ .unwrap_err()
+ .to_string();
+
+ assert!(err.contains("[DUPLICATED_MAP_KEY]"), "{err}");
+ }
+
+ #[test]
+ fn empty_row_does_not_trigger_dedup() {
+ let (keys, values) = int32_utf8_inputs(vec![], vec![]);
+ let offsets = [0i32, 0];
+
+ let result = map_from_keys_values_offsets_nulls(
+ &keys, &values, &offsets, &offsets, None, None, false,
+ )
+ .unwrap();
+
+ let map = result.as_map();
+ assert_eq!(map.len(), 1);
+ assert_eq!(map.value_offsets(), &[0, 0]);
+ }
+
+ #[test]
+ fn null_row_is_skipped_and_not_checked() {
+ // Row 0 is NULL (keys null). Its duplicate keys should be ignored;
+ // row 1 is a clean row.
+ let (keys, values) = int32_utf8_inputs(
+ vec![1, 1, 2, 3],
+ vec![Some("dup-a"), Some("dup-b"), Some("x"), Some("y")],
+ );
+ let offsets = [0i32, 2, 4];
+ let keys_nulls = NullBuffer::from(vec![false, true]);
+
+ let result = map_from_keys_values_offsets_nulls(
+ &keys,
+ &values,
+ &offsets,
+ &offsets,
+ Some(&keys_nulls),
+ None,
+ false,
+ )
+ .unwrap();
+
+ let map = result.as_map();
+ assert_eq!(map.len(), 2);
+ // First row is NULL (no entries emitted), second row keeps both entries.
+ assert_eq!(map.value_offsets(), &[0, 0, 2]);
+ assert!(map.is_null(0));
+ assert!(!map.is_null(1));
+ }
+}
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 387ef2262e..ec2055e5ad 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -342,6 +342,7 @@ datafusion.runtime.max_temp_directory_size 100G
datafusion.runtime.memory_limit unlimited
datafusion.runtime.metadata_cache_limit 50M
datafusion.runtime.temp_directory NULL
+datafusion.spark.map_key_dedup_policy EXCEPTION
datafusion.sql_parser.collect_spans false
datafusion.sql_parser.default_null_ordering nulls_max
datafusion.sql_parser.dialect generic
@@ -493,6 +494,7 @@ datafusion.runtime.max_temp_directory_size 100G Maximum temporary file directory
datafusion.runtime.memory_limit unlimited Maximum memory limit for query execution. Supports suffixes K (kilobytes), M (megabytes), and G (gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
datafusion.runtime.metadata_cache_limit 50M Maximum memory to use for file metadata cache such as Parquet metadata. Supports suffixes K (kilobytes), M (megabytes), and G (gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
datafusion.runtime.temp_directory NULL The path to the temporary file directory.
+datafusion.spark.map_key_dedup_policy EXCEPTION Policy for handling duplicate keys in Spark-compatible map-construction functions (`map_from_arrays`, `map_from_entries`, `str_to_map`). Mirrors Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961): - `EXCEPTION` (default): raise `[DUPLICATED_MAP_KEY]` at runtime on any duplicate key. - `LAST_WIN`: [...]
datafusion.sql_parser.collect_spans false When set to true, the source locations relative to the original SQL query (i.e. [`Span`](https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html)) will be collected and recorded in the logical plan nodes.
datafusion.sql_parser.default_null_ordering nulls_max Specifies the default null ordering for query results. There are 4 options: - `nulls_max`: Nulls appear last in ascending order. - `nulls_min`: Nulls appear first in ascending order. - `nulls_first`: Nulls always be first in any order. - `nulls_last`: Nulls always be last in any order. By default, `nulls_max` is used to follow Postgres's behavior. postgres rule: <https://www.postgresql.org/docs/current/queries-order.html>
datafusion.sql_parser.dialect generic Configure the SQL dialect used by DataFusion's parser; supported values include: Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB and Databricks.
diff --git a/datafusion/sqllogictest/test_files/spark/map/map_from_arrays.slt b/datafusion/sqllogictest/test_files/spark/map/map_from_arrays.slt
index a26b0435c9..7e501a3162 100644
--- a/datafusion/sqllogictest/test_files/spark/map/map_from_arrays.slt
+++ b/datafusion/sqllogictest/test_files/spark/map/map_from_arrays.slt
@@ -118,11 +118,25 @@ SELECT
----
{outer_key1: {inner_a: 1, inner_b: 2}, outer_key2: {inner_x: 10, inner_y: 20, inner_z: 30}}
-# Test with duplicate keys
-query ?
+# Test with duplicate keys: raises DUPLICATED_MAP_KEY under Spark's default policy
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key true was found
SELECT map_from_arrays(array(true, false, true), array('a', NULL, 'b'));
-----
-{false: NULL, true: b}
+
+# Integer keys with a duplicate also raise DUPLICATED_MAP_KEY.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key 1 was found
+SELECT map_from_arrays(array(1, 2, 1), array('a', 'b', 'c'));
+
+# String keys with a duplicate also raise DUPLICATED_MAP_KEY.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key k was found
+SELECT map_from_arrays(array('k', 'k', 'k'), array(1, 2, 3));
+
+# Multi-row: a clean row and a duplicate row still errors.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key 1 was found
+SELECT map_from_arrays(a, b)
+FROM values
+ (array[1, 2], array['a', 'b']),
+ (array[1, 1], array['x', 'y'])
+AS tab(a, b);
# Tests with different list types
query ?
@@ -134,3 +148,40 @@ query ?
SELECT map_from_arrays(arrow_cast(array('a', 'b', 'c'), 'FixedSizeList(3, Utf8)'), arrow_cast(array(1, 2, 3), 'LargeList(Int32)'));
----
{a: 1, b: 2, c: 3}
+
+# LAST_WIN policy: duplicates are allowed; later occurrences overwrite earlier ones.
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'LAST_WIN';
+
+query ?
+SELECT map_from_arrays(array(1, 2, 1), array('a', 'b', 'c'));
+----
+{1: c, 2: b}
+
+query ?
+SELECT map_from_arrays(array('k', 'k', 'k'), array(1, 2, 3));
+----
+{k: 3}
+
+query ?
+SELECT map_from_arrays(array(true, false, true), array('a', NULL, 'b'));
+----
+{true: b, false: NULL}
+
+# Multi-row mix under LAST_WIN: clean, duplicate, empty and NULL rows all work.
+query ?
+SELECT map_from_arrays(a, b)
+FROM values
+ (array[1, 2], array['a', 'b']),
+ (array[1, 1], array['x', 'y']),
+ (array[], array[]),
+ (NULL, NULL)
+AS tab(a, b);
+----
+{1: a, 2: b}
+{1: y}
+{}
+NULL
+
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'EXCEPTION';
diff --git a/datafusion/sqllogictest/test_files/spark/map/map_from_entries.slt b/datafusion/sqllogictest/test_files/spark/map/map_from_entries.slt
index 19b46886a0..21f41f5ad9 100644
--- a/datafusion/sqllogictest/test_files/spark/map/map_from_entries.slt
+++ b/datafusion/sqllogictest/test_files/spark/map/map_from_entries.slt
@@ -151,8 +151,8 @@ SELECT
----
{outer_key1: {inner_a: 1, inner_b: 2}, outer_key2: {inner_x: 10, inner_y: 20, inner_z: 30}}
-# Test with duplicate keys
-query ?
+# Test with duplicate keys: raises DUPLICATED_MAP_KEY under Spark's default policy
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key true was found
SELECT map_from_entries(array(
struct(true, 'a'),
struct(false, 'b'),
@@ -160,5 +160,58 @@ SELECT map_from_entries(array(
struct(false, cast(NULL as string)),
struct(true, 'd')
));
+
+# Integer keys with a duplicate also raise DUPLICATED_MAP_KEY.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key 1 was found
+SELECT map_from_entries(array(struct(1, 'a'), struct(2, 'b'), struct(1, 'c')));
+
+# String keys with triple occurrence also raise DUPLICATED_MAP_KEY.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key k was found
+SELECT map_from_entries(array(struct('k', 1), struct('k', 2), struct('k', 3)));
+
+# Multi-row: a clean row followed by a duplicate row still errors.
+query error DataFusion error: Execution error: \[DUPLICATED_MAP_KEY\] Duplicate map key 1 was found
+SELECT map_from_entries(data)
+FROM values
+ (array[struct(1, 'a'), struct(2, 'b')]),
+ (array[struct(1, 'x'), struct(1, 'y')])
+AS tab(data);
+
+# LAST_WIN policy: duplicates are allowed; later occurrences overwrite earlier ones.
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'LAST_WIN';
+
+query ?
+SELECT map_from_entries(array(
+ struct(true, 'a'),
+ struct(false, 'b'),
+ struct(true, 'c'),
+ struct(false, cast(NULL as string)),
+ struct(true, 'd')
+));
----
-{false: NULL, true: d}
+{true: d, false: NULL}
+
+query ?
+SELECT map_from_entries(array(struct(1, 'a'), struct(2, 'b'), struct(1, 'c')));
+----
+{1: c, 2: b}
+
+query ?
+SELECT map_from_entries(array(struct('k', 1), struct('k', 2), struct('k', 3)));
+----
+{k: 3}
+
+# Multi-row mix under LAST_WIN: clean row + duplicate row both succeed.
+query ?
+SELECT map_from_entries(data)
+FROM values
+ (array[struct(1, 'a'), struct(2, 'b')]),
+ (array[struct(1, 'x'), struct(1, 'y')])
+AS tab(data);
+----
+{1: a, 2: b}
+{1: y}
+
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'EXCEPTION';
diff --git a/datafusion/sqllogictest/test_files/spark/map/str_to_map.slt b/datafusion/sqllogictest/test_files/spark/map/str_to_map.slt
index 30d1672aef..68d856d854 100644
--- a/datafusion/sqllogictest/test_files/spark/map/str_to_map.slt
+++ b/datafusion/sqllogictest/test_files/spark/map/str_to_map.slt
@@ -64,11 +64,25 @@ SELECT str_to_map('a=1&b=2&c=3', '&', '=');
{a: 1, b: 2, c: 3}
# Duplicate keys: EXCEPTION policy (Spark 3.0+ default)
-# TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config is supported
statement error
Duplicate map key
SELECT str_to_map('a:1,b:2,a:3');
+# Triple+ occurrences of the same key still raise DUPLICATED_MAP_KEY.
+statement error
+Duplicate map key 'a'
+SELECT str_to_map('a:1,a:2,a:3');
+
+# Duplicate where one occurrence is missing the kv_delim (value = NULL) still errors.
+statement error
+Duplicate map key 'a'
+SELECT str_to_map('a,b:2,a:3');
+
+# Multi-row input: a clean row followed by a duplicate row fails on the duplicate row.
+statement error
+Duplicate map key 'a'
+SELECT str_to_map(col) FROM (VALUES ('a:1,b:2'), ('a:3,a:4')) AS t(col);
+
# Additional tests (DataFusion-specific)
# NULL input returns NULL
@@ -111,4 +125,39 @@ SELECT str_to_map(col1, col2, col3) FROM (VALUES ('a=1,b=2', ',', '='), ('x#9',
----
{a: 1, b: 2}
{x: 9}
-NULL
\ No newline at end of file
+NULL
+
+# LAST_WIN policy: duplicates are allowed; later occurrences overwrite earlier ones.
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'LAST_WIN';
+
+query ?
+SELECT str_to_map('a:1,b:2,a:3');
+----
+{a: 3, b: 2}
+
+query ?
+SELECT str_to_map('a:1,a:2,a:3');
+----
+{a: 3}
+
+# Missing kv_delim: the later occurrence overwrites the value at the key's
+# first-seen position.
+query ?
+SELECT str_to_map('a:1,b:2,a');
+----
+{a: NULL, b: 2}
+
+# Multi-row: both clean and duplicate rows succeed under LAST_WIN.
+query ?
+SELECT str_to_map(col) FROM (VALUES ('a:1,b:2'), ('a:3,a:4')) AS t(col);
+----
+{a: 1, b: 2}
+{a: 4}
+
+statement ok
+set datafusion.spark.map_key_dedup_policy = 'EXCEPTION';
+
+# Invalid policy values are rejected at SET time with a clear message.
+statement error DataFusion error: Invalid or Unsupported Configuration: Invalid MapKeyDedupPolicy: BOGUS\. Expected one of: EXCEPTION, LAST_WIN
+set datafusion.spark.map_key_dedup_policy = 'BOGUS';
\ No newline at end of file
diff --git a/docs/source/library-user-guide/upgrading/55.0.0.md b/docs/source/library-user-guide/upgrading/55.0.0.md
index 54bacbdff2..d3988c33a4 100644
--- a/docs/source/library-user-guide/upgrading/55.0.0.md
+++ b/docs/source/library-user-guide/upgrading/55.0.0.md
@@ -96,3 +96,34 @@ as a supertrait:
- pub trait QueryPlanner: Debug
+ pub trait QueryPlanner: Any + Debug
```
+
+### Spark map functions now reject duplicate keys by default
+
+The Spark-compatibility map-construction functions (`map_from_arrays`,
+`map_from_entries`, `str_to_map`) now raise `[DUPLICATED_MAP_KEY]` at runtime
+when constructing a map that contains duplicate keys. This matches the default
+of Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4502-L4511).
+
+A new config option, `datafusion.spark.map_key_dedup_policy`, controls the
+behavior:
+
+- `EXCEPTION` (default): raise on any duplicate key.
+- `LAST_WIN`: keep the last occurrence of each duplicate key. The key stays at
+ its first-seen position with the value from its last occurrence (matching
+ Spark's `ArrayBasedMapBuilder`).
+
+**Who is affected:**
+
+- Queries calling `map_from_arrays` or `str_to_map` on data that contains
+ duplicate keys. Previously these functions either tolerated duplicates
+ silently or raised a non-configurable error.
+
+**Migration guide:**
+
+To restore lenient duplicate-key handling, set the policy to `LAST_WIN`:
+
+```sql
+SET datafusion.spark.map_key_dedup_policy = 'LAST_WIN';
+```
+
+See [PR #21720](https://github.com/apache/datafusion/pull/21720) for details.
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index e0e2a5d21c..2d16e9ae3a 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -204,6 +204,7 @@ The following configuration settings are available:
| datafusion.format.time_format | %H:%M:%S%.f | Time format for time arrays [...]
| datafusion.format.duration_format | pretty | Duration format. Can be either `"pretty"` or `"ISO8601"` [...]
| datafusion.format.types_info | false | Show types in visual representation batches [...]
+| datafusion.spark.map_key_dedup_policy | EXCEPTION | Policy for handling duplicate keys in Spark-compatible map-construction functions (`map_from_arrays`, `map_from_entries`, `str_to_map`). Mirrors Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961): - `EXCEPTION` (default): raise `[DUPLICATED_M [...]
You can also reset configuration options to default settings via SQL using the `RESET` command. For
example, to set and reset `datafusion.execution.batch_size`:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org