You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "srbiswal (via GitHub)" <gi...@apache.org> on 2026/08/06 14:37:49 UTC
[PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
srbiswal opened a new pull request, #6680:
URL: https://github.com/apache/hive/pull/6680
For tables with skip.header.line.count/skip.footer.line.count, SkippingTextInputFormat called readLine() then getPos() on the same FSDataInputStream (in getCachedStartIndex and getCachedEndIndex). When the data contains a lone \r (not followed by \n), DataInputStream.readLine() pushes back its look-ahead byte by wrapping the stream in a non-Seekable PushbackInputStream. The following getPos() then throws ClassCastException during Tez split generation. LF, CRLF, and \r-at-EOF were unaffected.
Replace readLine()+getPos() in both methods with a ByteCountingLineReader that handles \n, \r\n, and lone \r, computing offsets from bytes consumed, so getPos() is never called on a mutated stream. Offset behavior is unchanged. As a safety net, makeSplitInternal now turns any unexpected error from header/footer detection into a clear, file-specific message instead of a raw ClassCastException
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests in `TestSkippingTextInputFormat`:
- **`testSkipFileSplitsLoneCR`** — reproduces the `ClassCastException` on a lone-CR file with `skip.header.line.count=1` (fails on master, passes with this fix).
- **`testSkipHeaderSplitOffsetsAcrossLineEndings`** — writes the same content with LF, lone-CR, and CRLF terminators and asserts exact split `getStart()`/`getLength()`, proving lone-CR matches LF and there is no boundary regression.
- **`testSkipFileSplitsLoneCRHeaderFooter`** — exercises the footer path (`getCachedEndIndex`) with lone-CR and skip header/footer, verifying the header and footer rows are skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5348555100
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764648988
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
Review Comment:
since you've added your own reader why not keep the API - .getPos().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5328343407
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5207304658
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "srbiswal (via GitHub)" <gi...@apache.org>.
srbiswal commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3767485220
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
Review Comment:
replaced ByteCountingLineReader with org.apache.hadoop.util.LineReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764648988
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
Review Comment:
since you've added your own reader why not keep the API - .getPos().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5342300659
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764693556
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
+ private final InputStream in;
+ private long bytesConsumed;
+ // Look-ahead byte past a '\r' (belongs to the next line, not yet counted); -1 = empty.
+ private int pushedBack = -1;
+
+ ByteCountingLineReader(InputStream in) {
+ this.in = in;
+ }
+
+ long getBytesConsumed() {
+ return bytesConsumed;
+ }
+
+ private int nextByte() throws IOException {
+ if (pushedBack != -1) {
Review Comment:
could we replace the pushedBack field with Java's existing PushbackInputStream ?
this.in = new PushbackInputStream(in, 1);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5269865014
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "srbiswal (via GitHub)" <gi...@apache.org>.
srbiswal commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3767476626
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -78,6 +79,10 @@ private FileSplit makeSplitInternal(Path file, long start, long length, String[]
} catch (IOException e) {
LOG.warn("Could not detect header/footer", e);
return new NullRowsInputFormat.DummyInputSplit(file);
+ } catch (RuntimeException e) {
+ // Report unexpected detection failures clearly instead of a cryptic cast.
+ throw new RuntimeException("Failed to detect header/footer boundaries for file "
Review Comment:
makes sense. removed the catch (RuntimeException)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5345546573
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764619563
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -78,6 +79,10 @@ private FileSplit makeSplitInternal(Path file, long start, long length, String[]
} catch (IOException e) {
LOG.warn("Could not detect header/footer", e);
return new NullRowsInputFormat.DummyInputSplit(file);
+ } catch (RuntimeException e) {
+ // Report unexpected detection failures clearly instead of a cryptic cast.
+ throw new RuntimeException("Failed to detect header/footer boundaries for file "
Review Comment:
i don't see much benefit in re-throwing same runtime exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6680:
URL: https://github.com/apache/hive/pull/6680#issuecomment-5325254330
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6680&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6680&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6680&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6680)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #6680:
URL: https://github.com/apache/hive/pull/6680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764693556
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
+ private final InputStream in;
+ private long bytesConsumed;
+ // Look-ahead byte past a '\r' (belongs to the next line, not yet counted); -1 = empty.
+ private int pushedBack = -1;
+
+ ByteCountingLineReader(InputStream in) {
+ this.in = in;
+ }
+
+ long getBytesConsumed() {
+ return bytesConsumed;
+ }
+
+ private int nextByte() throws IOException {
+ if (pushedBack != -1) {
Review Comment:
could we replace the pushedBack field with Java's existing PushbackInputStream ?
this.in = new PushbackInputStream(in, 1);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6680:
URL: https://github.com/apache/hive/pull/6680#discussion_r3764748959
##########
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java:
##########
@@ -193,17 +193,72 @@ private long getCachedEndIndex(Path path) throws IOException {
// there were not enough lines in the file to consume all footer rows.
endIndexForFile = Long.MIN_VALUE;
}
- } finally {
- if (fis != null) {
- fis.close();
- }
}
}
endIndexMap.put(path, endIndexForFile);
}
return endIndexForFile;
}
+ /**
+ * Reads lines while counting bytes consumed, so offsets can be computed without
+ * {@link FSDataInputStream#getPos()} -- which is unreliable after {@code readLine()}:
+ * a lone {@code '\r'} makes it swap in a non-Seekable {@code PushbackInputStream}, so
+ * the next {@code getPos()} throws {@code ClassCastException}. Handles {@code '\n'},
+ * {@code '\r\n'} and lone {@code '\r'}; after {@link #readLine()},
+ * {@link #getBytesConsumed()} is the offset just past the line's terminator.
+ */
+ static final class ByteCountingLineReader {
Review Comment:
doesn't hadoop already comes with similar util:
````
Text headerLine = new Text();
LineReader reader = new LineReader(fis);
int bytesRead = reader.readLine(headerLine);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org