You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Blizzara (via GitHub)" <gi...@apache.org> on 2024/07/19 12:11:56 UTC
[PR] Avo/substrait map literals [datafusion]
Blizzara opened a new pull request, #11547:
URL: https://github.com/apache/datafusion/pull/11547
## Which issue does this PR close?
Related to Map epic https://github.com/apache/datafusion/issues/11434
## Rationale for this change
Substrait didn't support Map literals, since we previously didn't have Map ScalarValues. https://github.com/apache/datafusion/pull/11224 implemented ScalarValues (thanks!), so now we can add them into Substrait as well.
There were also couple gaps left by the implementation that I encountered while testing this, so I fixed those as well.
Also, there was a bug in the from_substrait_literal for lists containing structs, which I realized while implementing the map support.
## What changes are included in this PR?
- Implement Substrait roundtrip support for Map literals, incl. null and empty maps.
- Fix field name handling for Substrait lists containing structs
- Add Map support to `ScalarValue::iter_to_array` and `create_hashes`
## Are these changes tested?
Tested with new roundtrip tests for the Map literals, and unit tests for hashing.
Also an existing UT for list literals is extended to cover the multiple structs values case.
I'd have added a sql roundtrip test for Map as well, but it seems that the `MAP` command turns into a ScalarFunction rather than ScalarValue. Maybe we'd need to run some round of optimizer to fold it, but I wonder why that is different from the `STRUCT` command which does turn into a struct literal?
I.e. doing `roundtrip("VALUES (MAP(['k1', 'k2'], [true, CAST(NULL AS BOOLEAN)]))").await?;` results in `Error: Substrait("Only literal types can be aliased in Virtual Tables, got: ScalarFunction")`, while similar code for STRUCT works fine.
## Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1685012930
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -302,3 +302,11 @@ SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), a
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
+
+
+query ?
+VALUES (MAP(['a'], [1])), (MAP(['b'], [2])), (MAP(['c', 'a'], [3, 1]))
Review Comment:
Without the changes in `scalar/mod.rs`, this would error:
```
External error: query failed: DataFusion error: Internal error: Unsupported creation of Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) array from ScalarValue Some(Map([{}])).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] VALUES(MAP([], []))
at test_files/map.slt:307
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Avo/substrait map literals [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1684290508
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -302,3 +302,9 @@ SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), a
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
+
+
+query ?
+VALUES(MAP([], []))
Review Comment:
Withtout the changes in `scalar/mod.rs`, this would error:
```
External error: query failed: DataFusion error: Internal error: Unsupported creation of Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) array from ScalarValue Some(Map([{}])).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] VALUES(MAP([], []))
at test_files/map.slt:307
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#issuecomment-2244058251
> I wonder if @goldmedal you might have some time to review this PR as well?
Sure, I'll review this tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#issuecomment-2240244642
@alamb thanks for merging the other PR, this is now rebased on top and also ready for review! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1688306249
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -692,6 +732,64 @@ mod tests {
assert_eq!(hashes[0], hashes[1]);
}
+ #[test]
+ // Tests actual values of hashes, which are different if forcing collisions
+ #[cfg(not(feature = "force_hash_collisions"))]
+ fn create_hashes_for_map_arrays() {
+ let mut builder =
+ MapBuilder::new(None, StringBuilder::new(), Int32Builder::new());
+ // Row 0
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 1
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 2
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(12);
+ builder.append(true).unwrap();
+ // Row 3
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key3");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 4
+ builder.keys().append_value("key1");
+ builder.values().append_value(10);
+ builder.append(true).unwrap();
+ // Row 5
+ builder.keys().append_value("key1");
+ builder.values().append_null();
+ builder.append(true).unwrap();
+ // Row 6
+ builder.append(true).unwrap();
+ // Row 7
+ builder.keys().append_value("key1");
+ builder.values().append_value(10);
+ builder.append(false).unwrap();
+
+ let array = Arc::new(builder.finish()) as ArrayRef;
+
+ let random_state = RandomState::with_seeds(0, 0, 0, 0);
+ let mut hashes = vec![0; array.len()];
+ create_hashes(&[array], &random_state, &mut hashes).unwrap();
+ assert_eq!(hashes[0], hashes[1]); // same value
+ assert_ne!(hashes[0], hashes[2]); // different key
+ assert_ne!(hashes[0], hashes[3]); // different value
Review Comment:
Yes, great catch! I must have confused myself, or changed it after writing 😅 fixed in https://github.com/apache/datafusion/pull/11547/commits/62149fa992bf07ae6ca9b78f283541fcf1293339, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#issuecomment-2239027784
This and https://github.com/apache/datafusion/pull/11510 will conflict, once either one is merged I'll be happy to fix the other one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11547:
URL: https://github.com/apache/datafusion/pull/11547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1688326679
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1785,6 +1798,98 @@ fn from_substrait_literal(
}
}
}
+ Some(LiteralType::Map(m)) => {
+ // Each entry should start the name index from the same value, then we increase it
+ // once at the end
+ let mut entry_name_idx = *name_idx;
+ let entries = m
+ .key_values
+ .iter()
+ .map(|kv| {
+ entry_name_idx = *name_idx;
+ let key_sv = from_substrait_literal(
+ kv.key.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ let value_sv = from_substrait_literal(
+ kv.value.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ ScalarStructBuilder::new()
Review Comment:
Hmh, yeah I think I'll leave it as-is for now, as this code is relevant only for local relations (ie data encoded in the substrait message itself) I don't expect it to be very performance-sensitive, the data scale should hopefully always be small...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1685017353
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1785,6 +1798,98 @@ fn from_substrait_literal(
}
}
}
+ Some(LiteralType::Map(m)) => {
+ // Each entry should start the name index from the same value, then we increase it
+ // once at the end
+ let mut entry_name_idx = *name_idx;
+ let entries = m
+ .key_values
+ .iter()
+ .map(|kv| {
+ entry_name_idx = *name_idx;
+ let key_sv = from_substrait_literal(
+ kv.key.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ let value_sv = from_substrait_literal(
+ kv.value.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ ScalarStructBuilder::new()
Review Comment:
this was the most high-level way of creating the map I could think of, lmk if you have better ideas!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1688090369
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1785,6 +1798,98 @@ fn from_substrait_literal(
}
}
}
+ Some(LiteralType::Map(m)) => {
+ // Each entry should start the name index from the same value, then we increase it
+ // once at the end
+ let mut entry_name_idx = *name_idx;
+ let entries = m
+ .key_values
+ .iter()
+ .map(|kv| {
+ entry_name_idx = *name_idx;
+ let key_sv = from_substrait_literal(
+ kv.key.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ let value_sv = from_substrait_literal(
+ kv.value.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ ScalarStructBuilder::new()
Review Comment:
> this was the most high-level way of creating the map I could think of, lmk if you have better ideas!
It looks good to me. If you intend to be more efficient, I suggest referring to how `make_map_batch_internal` creates a `MapArray`.
https://github.com/apache/datafusion/blob/77311a5896272c7ed252d8cd53d48ec6ea7c0ccf/datafusion/functions-array/src/map.rs#L80
You need to partition the key and value pairs into two arrays, and build the `MapArray` based on them. Maybe you can refer to `plan_make_map`.
https://github.com/apache/datafusion/blob/77311a5896272c7ed252d8cd53d48ec6ea7c0ccf/datafusion/functions-array/src/planner.rs#L104
However, I think it just some improvements. We don't need to do that in this PR.
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -692,6 +732,64 @@ mod tests {
assert_eq!(hashes[0], hashes[1]);
}
+ #[test]
+ // Tests actual values of hashes, which are different if forcing collisions
+ #[cfg(not(feature = "force_hash_collisions"))]
+ fn create_hashes_for_map_arrays() {
+ let mut builder =
+ MapBuilder::new(None, StringBuilder::new(), Int32Builder::new());
+ // Row 0
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 1
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 2
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key2");
+ builder.values().append_value(10);
+ builder.values().append_value(12);
+ builder.append(true).unwrap();
+ // Row 3
+ builder.keys().append_value("key1");
+ builder.keys().append_value("key3");
+ builder.values().append_value(10);
+ builder.values().append_value(11);
+ builder.append(true).unwrap();
+ // Row 4
+ builder.keys().append_value("key1");
+ builder.values().append_value(10);
+ builder.append(true).unwrap();
+ // Row 5
+ builder.keys().append_value("key1");
+ builder.values().append_null();
+ builder.append(true).unwrap();
+ // Row 6
+ builder.append(true).unwrap();
+ // Row 7
+ builder.keys().append_value("key1");
+ builder.values().append_value(10);
+ builder.append(false).unwrap();
+
+ let array = Arc::new(builder.finish()) as ArrayRef;
+
+ let random_state = RandomState::with_seeds(0, 0, 0, 0);
+ let mut hashes = vec![0; array.len()];
+ create_hashes(&[array], &random_state, &mut hashes).unwrap();
+ assert_eq!(hashes[0], hashes[1]); // same value
+ assert_ne!(hashes[0], hashes[2]); // different key
+ assert_ne!(hashes[0], hashes[3]); // different value
Review Comment:
```suggestion
assert_ne!(hashes[0], hashes[2]); // different value
assert_ne!(hashes[0], hashes[3]); // different key
```
I guess the comments are wrong.
The difference between `Row 0` and `Row 2` is the value of `key2`: `{'key2': 11}` in Row 0 and `{'key2': 12}` in Row 2.
However, the difference between `Row 0` and `Row 3` is the key: `key2` in Row 0 and `key3` in Row 3.
Did I say that correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1686995057
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -692,6 +730,48 @@ mod tests {
assert_eq!(hashes[0], hashes[1]);
}
+ #[test]
+ // Tests actual values of hashes, which are different if forcing collisions
+ #[cfg(not(feature = "force_hash_collisions"))]
+ fn create_hashes_for_map_arrays() {
Review Comment:
It would help me undertand / verify this test if you could use a `MapBuilder` or add a comment showing what MapArray was being built
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -1773,6 +1773,7 @@ impl ScalarValue {
}
DataType::List(_)
| DataType::LargeList(_)
+ | DataType::Map(_, _)
Review Comment:
this is fine I think because `ScalarValue::Map` is implemented as a 1 row array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1685018032
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1785,6 +1798,98 @@ fn from_substrait_literal(
}
}
}
+ Some(LiteralType::Map(m)) => {
+ // Each entry should start the name index from the same value, then we increase it
+ // once at the end
+ let mut entry_name_idx = *name_idx;
+ let entries = m
+ .key_values
+ .iter()
+ .map(|kv| {
+ entry_name_idx = *name_idx;
+ let key_sv = from_substrait_literal(
+ kv.key.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ let value_sv = from_substrait_literal(
+ kv.value.as_ref().unwrap(),
+ extensions,
+ dfs_names,
+ &mut entry_name_idx,
+ )?;
+ ScalarStructBuilder::new()
Review Comment:
also/alternatively, should I add this into ScalarValue? they could sit next to ScalarValue::new_list etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1685003113
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -236,6 +236,40 @@ fn hash_struct_array(
Ok(())
}
+fn hash_map_array(
+ array: &MapArray,
+ random_state: &RandomState,
+ hashes_buffer: &mut [u64],
+) -> Result<()> {
+ let nulls = array.nulls();
+ let offsets = array.offsets();
+
+ // Create hashes for each entry in each row
+ let mut values_hashes = vec![0u64; array.entries().len()];
+ create_hashes(array.entries().columns(), random_state, &mut values_hashes)?;
+
+ // Combine the hashes for entries on each row with each other and previous hash for that row
Review Comment:
I adapted this logic from combining List and Struct hashing, the result seemed to make sense to me, but I'm not 100% confident in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1684290508
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -302,3 +302,9 @@ SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), a
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
+
+
+query ?
+VALUES(MAP([], []))
Review Comment:
Withtout the changes in `scalar/mod.rs`, this would error:
```
External error: query failed: DataFusion error: Internal error: Unsupported creation of Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) array from ScalarValue Some(Map([{}])).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] VALUES(MAP([], []))
at test_files/map.slt:307
```
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -302,3 +302,11 @@ SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), a
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
{POST: 41, HEAD: 33, PATCH: 30}
+
+
+query ?
+VALUES (MAP(['a'], [1])), (MAP(['b'], [2])), (MAP(['c', 'a'], [3, 1]))
Review Comment:
Withtout the changes in `scalar/mod.rs`, this would error:
```
External error: query failed: DataFusion error: Internal error: Unsupported creation of Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) array from ScalarValue Some(Map([{}])).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] VALUES(MAP([], []))
at test_files/map.slt:307
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1685011486
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -1773,6 +1773,7 @@ impl ScalarValue {
}
DataType::List(_)
| DataType::LargeList(_)
+ | DataType::Map(_, _)
Review Comment:
this works at least for the test case, given it re-uses arrow::compute::concat I'd hope it does the right thing overall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#issuecomment-2245720581
Thanks @goldmedal and @Blizzara -- 👌 very nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Map literals in Substrait consumer and producer [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #11547:
URL: https://github.com/apache/datafusion/pull/11547#discussion_r1687639877
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -692,6 +730,48 @@ mod tests {
assert_eq!(hashes[0], hashes[1]);
}
+ #[test]
+ // Tests actual values of hashes, which are different if forcing collisions
+ #[cfg(not(feature = "force_hash_collisions"))]
+ fn create_hashes_for_map_arrays() {
Review Comment:
done! https://github.com/apache/datafusion/pull/11547/commits/06b7d3cc8a96b7d1b38277de14326303b3b47401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org