You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramkumar Vadali (JIRA)" <ji...@apache.org> on 2010/05/26 08:04:33 UTC
[jira] Created: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
HAR files used for RAID parity need to have configurable partfile size
----------------------------------------------------------------------
Key: MAPREDUCE-1816
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/raid
Affects Versions: 0.20.1
Reporter: Ramkumar Vadali
Priority: Minor
RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Status: Open (was: Patch Available)
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Attachment: MAPREDUCE-1816.patch.1
Fixed unnecessary whitespace changes in README
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926065#action_12926065 ]
Hudson commented on MAPREDUCE-1816:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Assignee: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871954#action_12871954 ]
Hadoop QA commented on MAPREDUCE-1816:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445530/MAPREDUCE-1816.patch
against trunk revision 947758.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/205/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/205/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/205/console
This message is automatically generated.
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871531#action_12871531 ]
Rodrigo Schmidt commented on MAPREDUCE-1816:
--------------------------------------------
Hi Ram,
Your patch is changing spaces and tabs which makes it hard to check. Please change it.
You need some unit tests as well.
Thanks for working on this!
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Attachment: MAPREDUCE-1816.patch
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Status: Patch Available (was: Open)
Fix Version/s: 0.22.0
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-1816:
----------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I just committed this. Thanks Ram!
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Assignee: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884267#action_12884267 ]
Hadoop QA commented on MAPREDUCE-1816:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448457/MAPREDUCE-1816.patch.1
against trunk revision 959509.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/278/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/278/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/278/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/278/console
This message is automatically generated.
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885400#action_12885400 ]
Rodrigo Schmidt commented on MAPREDUCE-1816:
--------------------------------------------
+1
Patch looks good.
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Status: Patch Available (was: Open)
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Attachments: MAPREDUCE-1816.patch
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916342#action_12916342 ]
Rodrigo Schmidt commented on MAPREDUCE-1816:
--------------------------------------------
Do I need to +1 again? :)
+1
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Assignee: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Status: Open (was: Patch Available)
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar Vadali updated MAPREDUCE-1816:
---------------------------------------
Status: Patch Available (was: Open)
Assignee: Ramkumar Vadali
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Assignee: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1816) HAR files used for RAID parity
need to have configurable partfile size
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916263#action_12916263 ]
Ramkumar Vadali commented on MAPREDUCE-1816:
--------------------------------------------
Test results:
ant test:
All succeeded except TestUlimit, which is an unrelated failure.
ant test-patch:
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
I think it is OK to not have a unit-test for this patch since this only changes configuration parameters. I have tested this manually and verified that it works
> HAR files used for RAID parity need to have configurable partfile size
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1816
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1816
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.20.1
> Reporter: Ramkumar Vadali
> Assignee: Ramkumar Vadali
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1816.patch, MAPREDUCE-1816.patch.1
>
>
> RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
> The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.