You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Libo Yu <yu...@hotmail.com> on 2014/05/13 04:10:44 UTC
Is it a bug from CombineFileInputFormat?
Hi,
This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is
immediately used at line N if the file is not splitable. Is that right?
private static class OneFileInfo {
private long fileSize; // size of the file
private OneBlockInfo[] blocks; // all blocks in this file
OneFileInfo(Path path, Configuration conf,
boolean isSplitable,
HashMap<String, List<OneBlockInfo>> rackToBlocks,
HashMap<OneBlockInfo, String[]> blockToNodes,
HashMap<String, List<OneBlockInfo>> nodeToBlocks,
HashMap<String, Set<String>> rackToNodes,
long maxSize)
throws IOException {
this.fileSize = 0;
// get block locations from file system
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
BlockLocation[] locations = fs.getFileBlockLocations(stat, 0,
stat.getLen());
// create a list of all block and their locations
if (locations == null) {
blocks = new OneBlockInfo[0];
} else {
if(locations.length == 0) {
locations = new BlockLocation[] { new BlockLocation() }; <-----------------------------M
}
if (!isSplitable) {
// if the file is not splitable, just create the one block with
// full file length
blocks = new OneBlockInfo[1];
fileSize = stat.getLen();
blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]
.getHosts(), locations[0].getTopologyPaths()); <----------------------------------N
} else {
RE: Is it a bug from CombineFileInputFormat?
Posted by java8964 <ja...@hotmail.com>.
Why do you say so? What problem you got from this code?
Yong
From: yu_libo@hotmail.com
To: user@hadoop.apache.org
Subject: Is it a bug from CombineFileInputFormat?
Date: Mon, 12 May 2014 22:10:44 -0400
Hi,
This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is
immediately used at line N if the file is not splitable. Is that right?
private static class OneFileInfo {
private long fileSize; // size of the file
private OneBlockInfo[] blocks; // all blocks in this file
OneFileInfo(Path path, Configuration conf,
boolean isSplitable,
HashMap<String, List<OneBlockInfo>> rackToBlocks,
HashMap<OneBlockInfo, String[]> blockToNodes,
HashMap<String, List<OneBlockInfo>> nodeToBlocks,
HashMap<String, Set<String>> rackToNodes,
long maxSize)
throws IOException {
this.fileSize = 0;
// get block locations from file system
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
BlockLocation[] locations = fs.getFileBlockLocations(stat, 0,
stat.getLen());
// create a list of all block and their locations
if (locations == null) {
blocks = new OneBlockInfo[0];
} else {
if(locations.length == 0) {
locations = new BlockLocation[] { new BlockLocation() }; <-----------------------------M
}
if (!isSplitable) {
// if the file is not splitable, just create the one block with
// full file length
blocks = new OneBlockInfo[1];
fileSize = stat.getLen();
blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]
.getHosts(), locations[0].getTopologyPaths()); <----------------------------------N
} else {
RE: Is it a bug from CombineFileInputFormat?
Posted by java8964 <ja...@hotmail.com>.
Why do you say so? What problem you got from this code?
Yong
From: yu_libo@hotmail.com
To: user@hadoop.apache.org
Subject: Is it a bug from CombineFileInputFormat?
Date: Mon, 12 May 2014 22:10:44 -0400
Hi,
This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is
immediately used at line N if the file is not splitable. Is that right?
private static class OneFileInfo {
private long fileSize; // size of the file
private OneBlockInfo[] blocks; // all blocks in this file
OneFileInfo(Path path, Configuration conf,
boolean isSplitable,
HashMap<String, List<OneBlockInfo>> rackToBlocks,
HashMap<OneBlockInfo, String[]> blockToNodes,
HashMap<String, List<OneBlockInfo>> nodeToBlocks,
HashMap<String, Set<String>> rackToNodes,
long maxSize)
throws IOException {
this.fileSize = 0;
// get block locations from file system
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
BlockLocation[] locations = fs.getFileBlockLocations(stat, 0,
stat.getLen());
// create a list of all block and their locations
if (locations == null) {
blocks = new OneBlockInfo[0];
} else {
if(locations.length == 0) {
locations = new BlockLocation[] { new BlockLocation() }; <-----------------------------M
}
if (!isSplitable) {
// if the file is not splitable, just create the one block with
// full file length
blocks = new OneBlockInfo[1];
fileSize = stat.getLen();
blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]
.getHosts(), locations[0].getTopologyPaths()); <----------------------------------N
} else {
RE: Is it a bug from CombineFileInputFormat?
Posted by java8964 <ja...@hotmail.com>.
Why do you say so? What problem you got from this code?
Yong
From: yu_libo@hotmail.com
To: user@hadoop.apache.org
Subject: Is it a bug from CombineFileInputFormat?
Date: Mon, 12 May 2014 22:10:44 -0400
Hi,
This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is
immediately used at line N if the file is not splitable. Is that right?
private static class OneFileInfo {
private long fileSize; // size of the file
private OneBlockInfo[] blocks; // all blocks in this file
OneFileInfo(Path path, Configuration conf,
boolean isSplitable,
HashMap<String, List<OneBlockInfo>> rackToBlocks,
HashMap<OneBlockInfo, String[]> blockToNodes,
HashMap<String, List<OneBlockInfo>> nodeToBlocks,
HashMap<String, Set<String>> rackToNodes,
long maxSize)
throws IOException {
this.fileSize = 0;
// get block locations from file system
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
BlockLocation[] locations = fs.getFileBlockLocations(stat, 0,
stat.getLen());
// create a list of all block and their locations
if (locations == null) {
blocks = new OneBlockInfo[0];
} else {
if(locations.length == 0) {
locations = new BlockLocation[] { new BlockLocation() }; <-----------------------------M
}
if (!isSplitable) {
// if the file is not splitable, just create the one block with
// full file length
blocks = new OneBlockInfo[1];
fileSize = stat.getLen();
blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]
.getHosts(), locations[0].getTopologyPaths()); <----------------------------------N
} else {
RE: Is it a bug from CombineFileInputFormat?
Posted by java8964 <ja...@hotmail.com>.
Why do you say so? What problem you got from this code?
Yong
From: yu_libo@hotmail.com
To: user@hadoop.apache.org
Subject: Is it a bug from CombineFileInputFormat?
Date: Mon, 12 May 2014 22:10:44 -0400
Hi,
This is a private static inner class from CombineFilenputFormat.java
When "locations' length is 0", line M creates a new location which is
immediately used at line N if the file is not splitable. Is that right?
private static class OneFileInfo {
private long fileSize; // size of the file
private OneBlockInfo[] blocks; // all blocks in this file
OneFileInfo(Path path, Configuration conf,
boolean isSplitable,
HashMap<String, List<OneBlockInfo>> rackToBlocks,
HashMap<OneBlockInfo, String[]> blockToNodes,
HashMap<String, List<OneBlockInfo>> nodeToBlocks,
HashMap<String, Set<String>> rackToNodes,
long maxSize)
throws IOException {
this.fileSize = 0;
// get block locations from file system
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
BlockLocation[] locations = fs.getFileBlockLocations(stat, 0,
stat.getLen());
// create a list of all block and their locations
if (locations == null) {
blocks = new OneBlockInfo[0];
} else {
if(locations.length == 0) {
locations = new BlockLocation[] { new BlockLocation() }; <-----------------------------M
}
if (!isSplitable) {
// if the file is not splitable, just create the one block with
// full file length
blocks = new OneBlockInfo[1];
fileSize = stat.getLen();
blocks[0] = new OneBlockInfo(path, 0, fileSize, locations[0]
.getHosts(), locations[0].getTopologyPaths()); <----------------------------------N
} else {