[GitHub] [arrow-datafusion] alamb commented on a change in pull request #619: RFC: Do not prune out unnecessary columns with unqualified references

[GitBox <gi...@apache.org>]

alamb commented on a change in pull request #619:
URL: https://github.com/apache/arrow-datafusion/pull/619#discussion_r658862205



##########
File path: datafusion/src/optimizer/projection_push_down.rs
##########
@@ -75,9 +78,12 @@ fn get_projected_schema(
     //
     // we discard non-existing columns because some column names are not part of the schema,
     // e.g. when the column derives from an aggregation
-    let mut projection: Vec<usize> = required_columns
+    //
+    // Use BTreeSet to remove potential duplicates (e.g. union) as
+    // well as to sort the projection to ensure deterministic behavior
+    let mut projection: BTreeSet<usize> = required_columns

Review comment:
       When I did not use a `BTreeSet` one of the `UNION ALL` tests failed due to a duplicate column being projected. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org