You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Anupam Seth (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 19:45:40 UTC
[jira] [Created] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Hadoop ignores old-style config options for enabling compressed output
----------------------------------------------------------------------
Key: HADOOP-7993
URL: https://issues.apache.org/jira/browse/HADOOP-7993
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 0.23.1
Reporter: Anupam Seth
Assignee: Anupam Seth
Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
mapred.output.compression.codec
- settings that work on 0.20 but not on 0.23
mapred.output.compress=true
mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
- settings that work on 0.23
mapreduce.output.fileoutputformat.compress=true
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
This breaks backwards compatibility and causes existing jobs to fail.
This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194641#comment-13194641 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #938 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/938/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236506
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194416#comment-13194416 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1673 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1673/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236506
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194214#comment-13194214 ]
Mahadev konar commented on HADOOP-7993:
---------------------------------------
Patch looks good me. Ill go ahead and commit this.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated HADOOP-7993:
----------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Anupam!
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated HADOOP-7993:
--------------------------------
Status: Patch Available (was: Open)
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194661#comment-13194661 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #151 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/151/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev) - Merging r1236506 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194423#comment-13194423 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Common-0.23-Commit #432 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/432/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev) - Merging r1236506 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194696#comment-13194696 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #173 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/173/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev) - Merging r1236506 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated HADOOP-7993:
--------------------------------
Status: Open (was: Patch Available)
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194026#comment-13194026 ]
Anupam Seth commented on HADOOP-7993:
-------------------------------------
Just to clarify the reason for proposing the code change the way it is in the patch is that upon investigation, our understanding was that the old-style config was being written to file by the JobSubmitter before the replacement with the keys listed in MR deprecated keys are invoked, e.g. by a get. Thus, when the AppMaster tried to read in the config coupled together with mapred-default which currently has this set to false, the old-style config was being overriden to be whatever was specified in the default file. As a solution, therefore, we force handling deprecation on all keys before the job.xml gets written out.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated HADOOP-7993:
----------------------------------
Fix Version/s: 0.23.1
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194675#comment-13194675 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #971 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/971/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236506
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193469#comment-13193469 ]
Hadoop QA commented on HADOOP-7993:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511912/HADOOP-7993_branch_0_23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 7 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/532//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/532//console
This message is automatically generated.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194425#comment-13194425 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1617 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1617/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236506
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated HADOOP-7993:
--------------------------------
Attachment: HADOOP-7993_branch_0_23.patch
Kindly review.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194424#comment-13194424 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #423 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/423/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev) - Merging r1236506 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193380#comment-13193380 ]
Robert Joseph Evans commented on HADOOP-7993:
---------------------------------------------
Overall the patch looks good to me, btu I do have a few comments.
In your test, it looks like test-config.xml is never deleted. It would be better if you could use a temp file. That way you don't have to worry about collisions between files already on disk, and you don't have to worry about the file going away at the end of the test. Or even better if you could use a ByteArrayOutputStream instead of writing to a file at all.
In Configuration.handleDeprecation() could you change all calls to LOG.info to LOG.debug or just to be comments. We don't need to it be printed out every time we write out a config file. Also it looks like all you need are the property keys. You might not need to clone the entire property object. Just call getProps().keys() or getProps().keySet().
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193328#comment-13193328 ]
Anupam Seth commented on HADOOP-7993:
-------------------------------------
The javadoc warnings are unrelated.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193326#comment-13193326 ]
Hadoop QA commented on HADOOP-7993:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511882/HADOOP-7993_branch_0_23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 7 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/531//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/531//console
This message is automatically generated.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194430#comment-13194430 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #448 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/448/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev) - Merging r1236506 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated HADOOP-7993:
--------------------------------
Attachment: HADOOP-7993_branch_0_23.patch
Thanks for the feedback Bobby! I have incorporated your comments in the updated patch.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated HADOOP-7993:
--------------------------------
Status: Patch Available (was: Open)
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config
options for enabling compressed output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194428#comment-13194428 ]
Hudson commented on HADOOP-7993:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #1601 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1601/])
HADOOP-7993. Hadoop ignores old-style config options for enabling compressed output. (Anupam Seth via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236506
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestDeprecatedKeys.java
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>
> Key: HADOOP-7993
> URL: https://issues.apache.org/jira/browse/HADOOP-7993
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.23.1
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Fix For: 0.23.1
>
> Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
>
>
> Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the old-style configs and can be fixed by handdling deprecation before the file is written out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira