You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/23 01:27:42 UTC
[GitHub] [arrow-rs] viirya opened a new pull request #1476: Fix generate_non_canonical_map_case
viirya opened a new pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476
# Which issue does this PR close?
<!---
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #1475.
# Rationale for this change
<!---
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!---
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!---
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #1476: Fix generate_non_canonical_map_case, fix `MapArray` equality
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a change in pull request #1476: Fix generate_non_canonical_map_case
Posted by GitBox <gi...@apache.org>.
tustvold commented on a change in pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476#discussion_r834338546
##########
File path: arrow/src/array/equal/list.rs
##########
@@ -131,17 +159,46 @@ pub(super) fn list_equal<T: OffsetSizeTrait>(
lengths_equal(
&lhs_offsets[lhs_start..lhs_start + len],
&rhs_offsets[rhs_start..rhs_start + len],
- ) && equal_range(
- lhs_values,
- rhs_values,
- child_lhs_nulls.as_ref(),
- child_rhs_nulls.as_ref(),
- lhs_offsets[lhs_start].to_usize().unwrap(),
- rhs_offsets[rhs_start].to_usize().unwrap(),
- (lhs_offsets[lhs_start + len] - lhs_offsets[lhs_start])
- .to_usize()
- .unwrap(),
- )
+ ) && {
+ match lhs.data_type() {
+ DataType::Map(_, _) => {
+ // Don't use `equal_range` which calls `utils::base_equal` that checks
Review comment:
Same as above
##########
File path: arrow/src/array/equal/utils.rs
##########
@@ -66,7 +66,25 @@ pub(super) fn equal_nulls(
#[inline]
pub(super) fn base_equal(lhs: &ArrayData, rhs: &ArrayData) -> bool {
- lhs.data_type() == rhs.data_type() && lhs.len() == rhs.len()
+ let equal_type = match (lhs.data_type(), rhs.data_type()) {
+ (DataType::Map(l_field, l_sorted), DataType::Map(r_field, r_sorted)) => {
+ let field_equal = match (l_field.data_type(), r_field.data_type()) {
+ (DataType::Struct(l_fields), DataType::Struct(r_fields))
+ if l_fields.len() == 2 && r_fields.len() == 2 =>
+ {
+ // We don't enforce the equality of field names
+ l_fields.get(0).unwrap().data_type()
Review comment:
Should this also check nullability and metadata? I also wonder if this logic is better contained within the equality logic for DataType itself :thinking:
##########
File path: integration-testing/src/bin/arrow-json-integration-test.rs
##########
@@ -121,7 +164,7 @@ fn validate(arrow_name: &str, json_name: &str, verbose: bool) -> Result<()> {
let arrow_schema = arrow_reader.schema().as_ref().to_owned();
// compare schemas
- if json_file.schema != arrow_schema {
+ if canonicalize_schema(&json_file.schema) != canonicalize_schema(&arrow_schema) {
Review comment:
Something feels a bit off about this logic living at the edges, i.e. the interface. If the field names of a DataType::Map really don't matter, I would expect this to either not be stored, or at least stored as an implementation detail...
##########
File path: arrow/src/array/equal/list.rs
##########
@@ -58,22 +61,47 @@ fn offset_value_equal<T: OffsetSizeTrait>(
lhs_pos: usize,
rhs_pos: usize,
len: usize,
+ data_type: &DataType,
) -> bool {
let lhs_start = lhs_offsets[lhs_pos].to_usize().unwrap();
let rhs_start = rhs_offsets[rhs_pos].to_usize().unwrap();
let lhs_len = lhs_offsets[lhs_pos + len] - lhs_offsets[lhs_pos];
let rhs_len = rhs_offsets[rhs_pos + len] - rhs_offsets[rhs_pos];
- lhs_len == rhs_len
- && equal_range(
- lhs_values,
- rhs_values,
- lhs_nulls,
- rhs_nulls,
- lhs_start,
- rhs_start,
- lhs_len.to_usize().unwrap(),
- )
+ lhs_len == rhs_len && {
+ match data_type {
+ DataType::Map(_, _) => {
+ // Don't use `equal_range` which calls `utils::base_equal` that checks
Review comment:
I'm probably missing something but as far as I can tell you have modified base_equal to not check the names, and so I'm not sure why this special case is still needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #1476: Fix generate_non_canonical_map_case
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476#discussion_r833712963
##########
File path: integration-testing/src/bin/arrow-json-integration-test.rs
##########
@@ -107,6 +109,47 @@ fn arrow_to_json(arrow_name: &str, json_name: &str, verbose: bool) -> Result<()>
Ok(())
}
+fn canonicalize_schema(schema: &Schema) -> Schema {
Review comment:
is the issue that other arrow implementations create different names for the "key" and "value" fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a change in pull request #1476: Fix generate_non_canonical_map_case
Posted by GitBox <gi...@apache.org>.
tustvold commented on a change in pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476#discussion_r834532928
##########
File path: arrow/src/array/equal/list.rs
##########
@@ -58,22 +61,47 @@ fn offset_value_equal<T: OffsetSizeTrait>(
lhs_pos: usize,
rhs_pos: usize,
len: usize,
+ data_type: &DataType,
) -> bool {
let lhs_start = lhs_offsets[lhs_pos].to_usize().unwrap();
let rhs_start = rhs_offsets[rhs_pos].to_usize().unwrap();
let lhs_len = lhs_offsets[lhs_pos + len] - lhs_offsets[lhs_pos];
let rhs_len = rhs_offsets[rhs_pos + len] - rhs_offsets[rhs_pos];
- lhs_len == rhs_len
- && equal_range(
- lhs_values,
- rhs_values,
- lhs_nulls,
- rhs_nulls,
- lhs_start,
- rhs_start,
- lhs_len.to_usize().unwrap(),
- )
+ lhs_len == rhs_len && {
+ match data_type {
+ DataType::Map(_, _) => {
+ // Don't use `equal_range` which calls `utils::base_equal` that checks
Review comment:
Aah I see the problem, darn... :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a change in pull request #1476: Fix generate_non_canonical_map_case
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476#discussion_r834516183
##########
File path: arrow/src/array/equal/list.rs
##########
@@ -58,22 +61,47 @@ fn offset_value_equal<T: OffsetSizeTrait>(
lhs_pos: usize,
rhs_pos: usize,
len: usize,
+ data_type: &DataType,
) -> bool {
let lhs_start = lhs_offsets[lhs_pos].to_usize().unwrap();
let rhs_start = rhs_offsets[rhs_pos].to_usize().unwrap();
let lhs_len = lhs_offsets[lhs_pos + len] - lhs_offsets[lhs_pos];
let rhs_len = rhs_offsets[rhs_pos + len] - rhs_offsets[rhs_pos];
- lhs_len == rhs_len
- && equal_range(
- lhs_values,
- rhs_values,
- lhs_nulls,
- rhs_nulls,
- lhs_start,
- rhs_start,
- lhs_len.to_usize().unwrap(),
- )
+ lhs_len == rhs_len && {
+ match data_type {
+ DataType::Map(_, _) => {
+ // Don't use `equal_range` which calls `utils::base_equal` that checks
Review comment:
Hmm, we don't implement `Eq` for `DataType`, do you mean we should do it? Even we implement it, here we are comparing child data of the map array, so its data type is `Struct(vec!["key", "value"])`. So we still need to specially treat map type here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a change in pull request #1476: Fix generate_non_canonical_map_case
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #1476:
URL: https://github.com/apache/arrow-rs/pull/1476#discussion_r833773674
##########
File path: integration-testing/src/bin/arrow-json-integration-test.rs
##########
@@ -107,6 +109,47 @@ fn arrow_to_json(arrow_name: &str, json_name: &str, verbose: bool) -> Result<()>
Ok(())
}
+fn canonicalize_schema(schema: &Schema) -> Schema {
Review comment:
The generated json has fields named other than "key" and 'value". I think this is why it is non canonical map case. But I'm not sure which arrow implementations can actually produce such map arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org