You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2012/09/21 07:48:08 UTC
[jira] [Created] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Harsh J created HADOOP-8833:
-------------------------------
Summary: fs -text should make sure to call inputstream.seek(0) before using input stream
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
>From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // <=== index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461420#comment-13461420 ]
Hudson commented on HADOOP-8833:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1205 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/])
HADOOP-8833. fs -text should make sure to call inputstream.seek(0) before using input stream. Contributed by Tom White and Harsh J. (harsh) (Revision 1388869)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388869
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8833:
----------------------------
Attachment: HADOOP-8833.patch
This should fix it.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460489#comment-13460489 ]
Harsh J commented on HADOOP-8833:
---------------------------------
Oh, first gotta wait for jenkins again.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461410#comment-13461410 ]
Hudson commented on HADOOP-8833:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1174 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/])
HADOOP-8833. fs -text should make sure to call inputstream.seek(0) before using input stream. Contributed by Tom White and Harsh J. (harsh) (Revision 1388869)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388869
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461246#comment-13461246 ]
Harsh J commented on HADOOP-8833:
---------------------------------
I ran the new test locally - works on a Mac too so apparently no native dependency as I'd originally thought (due to deflate):
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hdfs.TestDFSShell
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.699 sec
{code}
Committing shortly. Thanks again Tom and Karthik.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8833:
----------------------------
Target Version/s: 3.0.0 (was: 2.0.2-alpha)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated HADOOP-8833:
-------------------------------------
Status: Open (was: Patch Available)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461251#comment-13461251 ]
Hudson commented on HADOOP-8833:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2818 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2818/])
HADOOP-8833. fs -text should make sure to call inputstream.seek(0) before using input stream. Contributed by Tom White and Harsh J. (harsh) (Revision 1388869)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388869
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated HADOOP-8833:
-------------------------------------
Status: Patch Available (was: Open)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-8833:
------------------------------
Attachment: HADOOP-8833.patch
+1 on the fix. I noticed that the test doesn't fail without the fix though. This is because BZip2Codec.BZip2CompressionInputStream.readStreamHeader() tolerates a missing (two-byte) header, so BZip2 files happen to work anyway. I've modified the test slightly to test a deflate-compressed file, and this one does fail without the seek fix.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461250#comment-13461250 ]
Hudson commented on HADOOP-8833:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2755 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2755/])
HADOOP-8833. fs -text should make sure to call inputstream.seek(0) before using input stream. Contributed by Tom White and Harsh J. (harsh) (Revision 1388869)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388869
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461256#comment-13461256 ]
Harsh J commented on HADOOP-8833:
---------------------------------
Committed to trunk. Pending commit to branch-2 (svn up seems to be going too slow here).
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated HADOOP-8833:
-------------------------------------
Status: Patch Available (was: Open)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8833:
----------------------------
Target Version/s: 3.0.0, 2.0.3-alpha (was: 3.0.0)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461257#comment-13461257 ]
Hudson commented on HADOOP-8833:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2777 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2777/])
HADOOP-8833. fs -text should make sure to call inputstream.seek(0) before using input stream. Contributed by Tom White and Harsh J. (harsh) (Revision 1388869)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388869
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8833:
----------------------------
Target Version/s: 2.0.2-alpha
Status: Patch Available (was: Open)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460399#comment-13460399 ]
Hadoop QA commented on HADOOP-8833:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546011/HADOOP-8833.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1490//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1490//console
This message is automatically generated.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated HADOOP-8833:
-------------------------------------
Attachment: HADOOP-8833.patch
Uploading the same patch again.
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460488#comment-13460488 ]
Harsh J commented on HADOOP-8833:
---------------------------------
Thanks Tom, I did wonder about that. Then though it to be my maven local repo cause I had first run test with fix installed. thanks for revising the patch. Committing to trunk and branch-2 now, but leaving open for 2.0.2 (gatekeeper has to grant).
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated HADOOP-8833:
-------------------------------------
Status: Open (was: Patch Available)
Cancelling patch to re-submit and kick Jenkins
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call
inputstream.seek(0) before using input stream
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8833:
----------------------------
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Target Version/s: (was: 3.0.0, 2.0.3-alpha)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> fs -text should make sure to call inputstream.seek(0) before using input stream
> -------------------------------------------------------------------------------
>
> Key: HADOOP-8833
> URL: https://issues.apache.org/jira/browse/HADOOP-8833
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.2-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
>
>
> From Muddy Dixon on HADOOP-8449:
> Hi
> We found the changes in order of switch and guard block in
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
> {code}
> Because of this change, return value of
> {code}
> codec.createInputStream(i)
> {code}
> is changed if codec exists.
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> // check codecs
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> switch(i.readShort()) {
> // cases
> }
> {code}
> New:
> {code}
> private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
> FSDataInputStream i = srcFs.open(p);
> switch(i.readShort()) { // <=== index (or pointer) processes!!
> // cases
> default: {
> // Check the type of compression instead, depending on Codec class's
> // own detection methods, based on the provided path.
> CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
> CompressionCodec codec = cf.getCodec(p);
> if (codec != null) {
> return codec.createInputStream(i);
> }
> break;
> }
> }
> // File is non-compressed, or not a file container we know.
> i.seek(0);
> return i;
> }
> {code}
> Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira