You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "guiyanakuang (via GitHub)" <gi...@apache.org> on 2023/08/23 09:18:31 UTC

[GitHub] [orc] guiyanakuang opened a new pull request, #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

guiyanakuang opened a new pull request, #1596:
URL: https://github.com/apache/orc/pull/1596

   
   ### What changes were proposed in this pull request?
   In this pull request, a conditional check for `footer.hasWriter()` is introduced before calling `footer.getWriter()` in the `getWriterVersion()` method. If the writer field is not present in the footer, the method will default to `UNKNOWN`.
   
   ### Why are the changes needed?
   Previously, the `getWriterVersion()` method did not check for the presence of the writer field in the footer. This led to a default value of 0 being used for `footer.getWriter()`, resulting in an incorrect WriterVersion being returned.
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   Added UT.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303236851


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   Technically, this read-path change doesn't match with our writer behavior because we throws `IllegalArgumentException` explicitly.
   
   https://github.com/apache/orc/blob/873d00c1f42753e9522cc488db59b1c979299eea/java/core/src/java/org/apache/orc/OrcFile.java#L749-L752



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] guiyanakuang commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303243866


##########
java/core/src/test/org/apache/orc/TestVectorOrcFile.java:
##########
@@ -4133,7 +4133,7 @@ public void testZeroByteOrcFile(Version fileFormat) throws Exception {
     assertEquals(0, reader.getStripes().size());
     assertEquals(0, reader.getStatistics().length);
     assertEquals(0, reader.getMetadataKeys().size());
-    assertEquals(OrcFile.CURRENT_WRITER, reader.getWriterVersion());
+    assertEquals(OrcFile.WriterVersion.FUTURE, reader.getWriterVersion());

Review Comment:
   Shouldn't the writer version of a zero byte orc file be unknown?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303281893


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   No, it seems that you are confused with the literal `FUTURE` version. The literal `FUTURE` version means `infinite` in Math, @guiyanakuang .



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] guiyanakuang commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1308147612


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   That being the case, close this PR and keep the status quo



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] guiyanakuang commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303258282


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   I don't think so; A low version writer certainly can't specify that the orc written is a futrue version. But a reader of a low version should be able to read an orc file created by a higher version, even if that higher version is unknown in the low-code implementation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303280374


##########
java/core/src/test/org/apache/orc/TestVectorOrcFile.java:
##########
@@ -4133,7 +4133,7 @@ public void testZeroByteOrcFile(Version fileFormat) throws Exception {
     assertEquals(0, reader.getStripes().size());
     assertEquals(0, reader.getStatistics().length);
     assertEquals(0, reader.getMetadataKeys().size());
-    assertEquals(OrcFile.CURRENT_WRITER, reader.getWriterVersion());
+    assertEquals(OrcFile.WriterVersion.FUTURE, reader.getWriterVersion());

Review Comment:
   No, it's defined as a valid file always. So, it should be the up-to-date version, which is `CURRENT_WRITER`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] guiyanakuang closed pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang closed pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`
URL: https://github.com/apache/orc/pull/1596


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] wgtmac commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303184871


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -31,6 +31,7 @@
 import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.List;
+import static org.apache.orc.OrcFile.WriterImplementation.UNKNOWN;

Review Comment:
   nit: should we avoid this static import?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303281893


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   No, it seems that you are confused with the literal `FUTURE` version. The literal `FUTURE` version means `infinite` in Math, @guiyanakuang .
   
   https://github.com/apache/orc/blob/873d00c1f42753e9522cc488db59b1c979299eea/java/core/src/java/org/apache/orc/OrcFile.java#L84C60-L84C60



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] guiyanakuang commented on pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on PR #1596:
URL: https://github.com/apache/orc/pull/1596#issuecomment-1690301989

   > Thank you for making a PR, @guiyanakuang .
   > 
   > Although I understand the issue and intention, I'm not sure if this PR makes Apache ORC safer or more robust because I'm not sure that the AS-IS code works correctly when the writer version is `FUTURE`. There are a few conditional code based on the writer versions, aren't there?
   > 
   > https://github.com/apache/orc/blob/873d00c1f42753e9522cc488db59b1c979299eea/java/core/src/java/org/apache/orc/OrcFile.java#L1056-L1059
   > 
   > https://github.com/apache/orc/blob/84cefd50be4651c37d31fab7581a9a14786d1237/java/core/src/java/org/apache/orc/OrcFile.java#L236-L239
   
   When the writer's version is `FUTURE`. there are actually two possibilities
   
   1: The orc file doesn't specify a writer, so it's unknown.
   
   2: The orc file file specifies a writer, but the version is outside the scope of the current impl (i.e., the orc file was created by a higher version of the impl).
   
   In order to prevent loss of info (e.g., new features in statistics), we prevent the use of a lower version of an impl to merge an orc file created by a higher version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303286993


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   I believe UNSTABLE_PRE_2_0 and FUTURE are for developer testing, and not valid for the production ORC files.
   https://github.com/apache/orc/blob/873d00c1f42753e9522cc488db59b1c979299eea/java/core/src/java/org/apache/orc/OrcFile.java#L71-L85



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303281893


##########
java/core/src/java/org/apache/orc/impl/OrcTail.java:
##########
@@ -121,8 +121,10 @@ public OrcProto.PostScript getPostScript() {
   public OrcFile.WriterVersion getWriterVersion() {
     OrcProto.PostScript ps = fileTail.getPostscript();
     OrcProto.Footer footer = fileTail.getFooter();
-    OrcFile.WriterImplementation writer =
-        OrcFile.WriterImplementation.from(footer.getWriter());
+    OrcFile.WriterImplementation writer = OrcFile.WriterImplementation.UNKNOWN;
+    if (footer.hasWriter()) {
+      writer = OrcFile.WriterImplementation.from(footer.getWriter());
+    }

Review Comment:
   No, it seems that you are confused with the literal `FUTURE` version. The literal `FUTURE` version means `Int.MavValue` in Math, @guiyanakuang .
   
   https://github.com/apache/orc/blob/873d00c1f42753e9522cc488db59b1c979299eea/java/core/src/java/org/apache/orc/OrcFile.java#L84



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1596: ORC-1490: Fix bug in `getWriterVersion` due to unchecked `footer.hasWriter()`

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1596:
URL: https://github.com/apache/orc/pull/1596#discussion_r1303233227


##########
java/core/src/test/org/apache/orc/TestVectorOrcFile.java:
##########
@@ -4133,7 +4133,7 @@ public void testZeroByteOrcFile(Version fileFormat) throws Exception {
     assertEquals(0, reader.getStripes().size());
     assertEquals(0, reader.getStatistics().length);
     assertEquals(0, reader.getMetadataKeys().size());
-    assertEquals(OrcFile.CURRENT_WRITER, reader.getWriterVersion());
+    assertEquals(OrcFile.WriterVersion.FUTURE, reader.getWriterVersion());

Review Comment:
   Also, this looks like a wrong behavior change to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org