You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by GitBox <gi...@apache.org> on 2020/10/05 06:17:04 UTC
[GitHub] [orc] dongjoon-hyun opened a new pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
dongjoon-hyun opened a new pull request #548:
URL: https://github.com/apache/orc/pull/548
### What changes were proposed in this pull request?
Apache ORC 1.6.0 ~ 1.6.5 cannot handle predicate pushdown case-insensitively.
### Why are the changes needed?
This is a performance regression from Apache ORC 1.5.x. Without this patch, ORC 1.6.x cannot pass SPARK-32622 (case sensitivity in predicate pushdown) unit test case.
### How was this patch tested?
Pass the newly added unit test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] omalley closed pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
omalley closed pull request #548:
URL: https://github.com/apache/orc/pull/548
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #548:
URL: https://github.com/apache/orc/pull/548#issuecomment-703422903
Could you review this PR please, @omalley ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] omalley commented on pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
omalley commented on pull request #548:
URL: https://github.com/apache/orc/pull/548#issuecomment-703824527
Ok, I just made the signature change and committed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on a change in pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #548:
URL: https://github.com/apache/orc/pull/548#discussion_r499364939
##########
File path: java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
##########
@@ -112,6 +112,16 @@ public void testFindColumn() throws Exception {
assertEquals(3, RecordReaderImpl.findColumns(evo, "e"));
}
+ @Test
+ public void testFindColumnCaseInsensitively() throws Exception {
Review comment:
This passed in `branch-1.5` with no change, too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #548:
URL: https://github.com/apache/orc/pull/548#issuecomment-703875528
Thank you so much, @omalley !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on a change in pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #548:
URL: https://github.com/apache/orc/pull/548#discussion_r499364939
##########
File path: java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
##########
@@ -112,6 +112,16 @@ public void testFindColumn() throws Exception {
assertEquals(3, RecordReaderImpl.findColumns(evo, "e"));
}
+ @Test
+ public void testFindColumnCaseInsensitively() throws Exception {
Review comment:
This passed in `branch-1.5` without any change.
##########
File path: java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
##########
@@ -112,6 +112,16 @@ public void testFindColumn() throws Exception {
assertEquals(3, RecordReaderImpl.findColumns(evo, "e"));
}
+ @Test
+ public void testFindColumnCaseInsensitively() throws Exception {
Review comment:
This passed in `branch-1.5` with no change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] omalley commented on a change in pull request #548: ORC-670. RecordReaderImpl.findColumns should respect orc.schema.evolution.case.sensitive
Posted by GitBox <gi...@apache.org>.
omalley commented on a change in pull request #548:
URL: https://github.com/apache/orc/pull/548#discussion_r499766335
##########
File path: java/core/src/java/org/apache/orc/TypeDescription.java
##########
@@ -818,8 +818,12 @@ public TypeDescription findSubtype(int goal) {
* @return the subtype
*/
public TypeDescription findSubtype(String columnName) {
+ return findSubtype(columnName, true);
+ }
+
+ public TypeDescription findSubtype(String columnName, Boolean isSchemaEvolutionCaseAware) {
Review comment:
This should be boolean.
##########
File path: java/core/src/java/org/apache/orc/impl/ParserUtils.java
##########
@@ -246,6 +246,12 @@ public static TypeDescription parseType(ParserUtils.StringPosition source) {
public static TypeDescription findSubtype(TypeDescription schema,
ParserUtils.StringPosition source) {
+ return findSubtype(schema, source, true);
+ }
+
+ public static TypeDescription findSubtype(TypeDescription schema,
+ ParserUtils.StringPosition source,
+ Boolean isSchemaEvolutionCaseAware) {
Review comment:
This should be boolean instead of Boolean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org