You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by GitBox <gi...@apache.org> on 2021/12/15 13:47:37 UTC

[GitHub] [orc] pgaref opened a new pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

pgaref opened a new pull request #972:
URL: https://github.com/apache/orc/pull/972


   Change-Id: I3687491a8e430609374ba259d721e98bf4359ba8
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. File a JIRA issue first and use it as a prefix of your PR title, e.g., `ORC-001: Fix ABC`.
     2. Use your PR title to summarize what this PR proposes instead of describing the problem.
     3. Make PR title and description complete because these will be the permanent commit log.
     4. If possible, provide a concise and reproducible example to reproduce the issue for a faster review.
     5. If the PR is unfinished, use GitHub PR Draft feature.
   -->
   
   ### What changes were proposed in this pull request?
   When trying to bump Apache Hive to 1.7 I noticed query failures related to SArgs pushdown.
   In more detail, Hive may push Partition column filters that are not part of the columns, until 1.6 recently these columns were ignored, while 1.7 throws IllegalArgumentException. 
   
   
   
   ### Why are the changes needed?
   Align findColumns behaviour between 1.6 and 1.7 release
   
   
   ### How was this patch tested?
   Existing tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun edited a comment on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995489232


   BTW, let me try to test the status of `branch-1.7` and roll a new Apache ORC release for Apache Hive, @pgaref .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun commented on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995306538


   cc @pavibhai and @omalley 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun commented on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995304947


   I cherry-picked this to branch-1.7 for Apache ORC 1.7.2 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun commented on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995488966


   Is it okay when we handle multiple ORC files in Hive schema evolutions?
   > I was thinking that throwing an exception when a SArg column is not found is probably a better approach than just logging.
   
   In many cases, Hive partitions might have different schemas. The simplest case is having new columns additionally in new partitions. If a user run a query for all partitions, SArg columns can have new columns which old partitions don't have.
   
   Apache Spark checks the physical schema when we open a file and try to adjust the missing columns. Given this PR's description, Apache Hive doesn't, right?
   
   For me, throwing an exception could be too intrusive and the AS-IS status (`-1`) would be enough.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun commented on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995489232


   BTW, let me try to test the status of `branch-1.7` and roll a new release for Apache Hive, @pgaref .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] dongjoon-hyun merged pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun merged pull request #972:
URL: https://github.com/apache/orc/pull/972


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [orc] pgaref commented on pull request #972: ORC-1059: Align findColumns behaviour between 1.6 and 1.7 release

Posted by GitBox <gi...@apache.org>.
pgaref commented on pull request #972:
URL: https://github.com/apache/orc/pull/972#issuecomment-995471135


   Thanks for the quick review @dongjoon-hyun ! 
   This will definitely help us bump to 1.7 quickly (e.g., next 1.7 release) but I was thinking that throwing an exception when a SArg column is not found is probably a better approach than just logging.
   
   I suggest we keep this in mind for 1.8 (and switch back to that behaviour) where we could align downstream consumers like Hive to only push SArgs for columns that are part of the schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org