You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2008/04/08 02:41:25 UTC
[jira] Created: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
------------------------------------------------------------------------------
Key: HADOOP-3203
URL: https://issues.apache.org/jira/browse/HADOOP-3203
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.15.3
Reporter: Chris Douglas
Fix For: 0.18.0
In TaskTracker::localizeJob:
{code}
// Get sizes of JobFile and JarFile
// sizes are -1 if they are not present.
FileSystem fileSystem = FileSystem.get(fConf);
FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
long jarFileSize = -1;
long jobFileSize = -1;
for(FileStatus stat : status) {
if (stat.getPath().toString().contains("job.xml")) {
jobFileSize = stat.getLen();
} else {
jobFileSize = -1;
}
if (stat.getPath().toString().contains("job.jar")) {
jarFileSize = stat.getLen();
} else {
jarFileSize = -1;
}
}
{code}
One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3203:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3203:
--------------------------------------------
Status: Open (was: Patch Available)
Cancelling patch to incorporate Nicholas's suggestion.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587121#action_12587121 ]
Amareshwari Sriramadasu commented on HADOOP-3203:
-------------------------------------------------
bq. you should use FileSystem.getFileStatus(Path) instead of FileSystem.listStatus(Path).
Using listStatus(Path) makes only one RPC. Do you think we should go for FileSystem.getFileStatus(Path) and make two RPCs?
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3203:
----------------------------------
Hadoop Flags: [Reviewed]
+1
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3203:
--------------------------------------------
Status: Patch Available (was: Open)
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596034#action_12596034 ]
Hadoop QA commented on HADOOP-3203:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381858/patch-3203.txt
against trunk revision 655410.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2451/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2451/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2451/console
This message is automatically generated.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587296#action_12587296 ]
Chris Douglas commented on HADOOP-3203:
---------------------------------------
bq. Using listStatus(Path) makes only one RPC. Do you think we should go for FileSystem.getFileStatus(Path) and make two RPCs?
This section of code should be called once per TaskTracker per Job, right? The difference between one RPC and two RPCs here is insufficient to measure, let alone effect a performance bottleneck or load we should worry about. Since the 5x expansion of the jar is a pessimistic estimate, the config should scarcely matter, but given HADOOP-3048 and the possibility of serialized objects in the config, it's worth checking its size, too.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3203:
--------------------------------------------
Status: Patch Available (was: Open)
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586620#action_12586620 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3203:
------------------------------------------------
- Why does it do stat.getPath().toString().contains("job.xml") but not new Path(parent, "job.xml")?
Also If the parent path contains "job.xml", all stat will be matched.
- Do we really need jobFileSize or jarFileSize? It seems working fine even the computation in the for-loop is wrong.
See also https://issues.apache.org/jira/browse/HADOOP-3182?focusedCommentId=12585877#action_12585877
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587272#action_12587272 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3203:
------------------------------------------------
If we use FileSystem.listStatus(Path), it requires 'r' permission in the parent dir. It seems that this additional permission requirement is not necessary.
For saving RPCs, do we really need the size of job.xml? In most cases, the file size is < 20kB.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3203:
--------------------------------------------
Attachment: patch-3203.txt
Patch fixing the bug in calculating jarFileSize and jobFileSize.
Also incorporated Nicholas's suggestion in comparison.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586894#action_12586894 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3203:
------------------------------------------------
In this case, you should use FileSystem.getFileStatus(Path) instead of FileSystem.listStatus(Path).
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3203:
--------------------------------------------
Attachment: patch-3203.txt
Patch incorporating review comments.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586662#action_12586662 ]
Amareshwari Sriramadasu commented on HADOOP-3203:
-------------------------------------------------
bq Do we really need jobFileSize or jarFileSize? It seems working fine even the computation in the for-loop is wrong.
We need them to make sure the LocalDirAllocator checks for size before allocating. If size is -1, LocalDirAllocator does not to check. thatswhy, things were fine.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu reassigned HADOOP-3203:
-----------------------------------------------
Assignee: Amareshwari Sriramadasu
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't
provide the correct size to LocalDirAllocator
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597118#action_12597118 ]
Hudson commented on HADOOP-3203:
--------------------------------
Integrated in Hadoop-trunk #492 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/492/])
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt, patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.