You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Xie Gang <xi...@gmail.com> on 2017/10/26 04:34:22 UTC
Block invalid IOException causes the DFSClient domain socket being disabled
Hi,
We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain socket
is disabled when datanode throw block invalid exception.
The block is invalidated for some reason on datanote and it's OK. Then
DFSClient tries to access this block on this datanode via domain socket.
This triggers a IOExcetion. On DFSClient side, when get a IOExcetion and
error code 'ERROR', it disables the domain socket and fails back to TCP.
and the worst is that it seems never recover the socket.
I think this is a defect and with such "block invalid" exception, we should
not disable the domain socket because the is nothing wrong about the domain
socket service.
And thoughts?
The code:
private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer,
Slot slot) throws IOException {
ShortCircuitCache cache = clientContext.getShortCircuitCache();
final DataOutputStream out =
new DataOutputStream(new BufferedOutputStream(peer.getOutputStream()));
SlotId slotId = slot == null ? null : slot.getSlotId();
new Sender(out).requestShortCircuitFds(block, token, slotId, 1);
DataInputStream in = new DataInputStream(peer.getInputStream());
BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(
PBHelper.vintPrefixed(in));
DomainSocket sock = peer.getDomainSocket();
switch (resp.getStatus()) {
case SUCCESS:
byte buf[] = new byte[1];
FileInputStream fis[] = new FileInputStream[2];
sock.recvFileInputStreams(fis, buf, 0, buf.length);
ShortCircuitReplica replica = null;
try {
ExtendedBlockId key =
new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId());
replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,
Time.monotonicNow(), slot);
} catch (IOException e) {
// This indicates an error reading from disk, or a format error. Since
// it's not a socket communication problem, we return null rather than
// throwing an exception.
LOG.warn(this + ": error creating ShortCircuitReplica.", e);
return null;
} finally {
if (replica == null) {
IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]);
}
}
return new ShortCircuitReplicaInfo(replica);
case ERROR_UNSUPPORTED:
if (!resp.hasShortCircuitAccessVersion()) {
LOG.warn("short-circuit read access is disabled for " +
"DataNode " + datanode + ". reason: " + resp.getMessage());
clientContext.getDomainSocketFactory()
.disableShortCircuitForPath(pathInfo.getPath());
} else {
LOG.warn("short-circuit read access for the file " +
fileName + " is disabled for DataNode " + datanode +
". reason: " + resp.getMessage());
}
return null;
case ERROR_ACCESS_TOKEN:
String msg = "access control error while " +
"attempting to set up short-circuit access to " +
fileName + resp.getMessage();
if (LOG.isDebugEnabled()) {
LOG.debug(this + ":" + msg);
}
return new ShortCircuitReplicaInfo(new InvalidToken(msg));
default:
LOG.warn(this + ": unknown response code " + resp.getStatus() +
" while attempting to set up short-circuit access. " +
resp.getMessage());
clientContext.getDomainSocketFactory()
.disableShortCircuitForPath(pathInfo.getPath());
return null;
}
--
Xie Gang
Re: Block invalid IOException causes the DFSClient domain socket
being disabled
Posted by John Zhuge <jo...@gmail.com>.
Please go ahead.
On Thu, Oct 26, 2017 at 6:12 PM, Xie Gang <xi...@gmail.com> wrote:
> Shall I create the jira directly?
>
> On Thu, Oct 26, 2017 at 12:34 PM, Xie Gang <xi...@gmail.com> wrote:
>
> > Hi,
> >
> > We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain
> > socket is disabled when datanode throw block invalid exception.
> >
> > The block is invalidated for some reason on datanote and it's OK. Then
> > DFSClient tries to access this block on this datanode via domain socket.
> > This triggers a IOExcetion. On DFSClient side, when get a IOExcetion and
> > error code 'ERROR', it disables the domain socket and fails back to TCP.
> > and the worst is that it seems never recover the socket.
> >
> > I think this is a defect and with such "block invalid" exception, we
> > should not disable the domain socket because the is nothing wrong about
> the
> > domain socket service.
> >
> > And thoughts?
> >
> > The code:
> >
> > private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer,
> > Slot slot) throws IOException {
> > ShortCircuitCache cache = clientContext.getShortCircuitCache();
> > final DataOutputStream out =
> > new DataOutputStream(new BufferedOutputStream(peer.
> getOutputStream()));
> > SlotId slotId = slot == null ? null : slot.getSlotId();
> > new Sender(out).requestShortCircuitFds(block, token, slotId, 1);
> > DataInputStream in = new DataInputStream(peer.getInputStream());
> > BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(
> > PBHelper.vintPrefixed(in));
> > DomainSocket sock = peer.getDomainSocket();
> > switch (resp.getStatus()) {
> > case SUCCESS:
> > byte buf[] = new byte[1];
> > FileInputStream fis[] = new FileInputStream[2];
> > sock.recvFileInputStreams(fis, buf, 0, buf.length);
> > ShortCircuitReplica replica = null;
> > try {
> > ExtendedBlockId key =
> > new ExtendedBlockId(block.getBlockId(),
> block.getBlockPoolId());
> > replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,
> > Time.monotonicNow(), slot);
> > } catch (IOException e) {
> > // This indicates an error reading from disk, or a format error.
> Since
> > // it's not a socket communication problem, we return null rather
> than
> > // throwing an exception.
> > LOG.warn(this + ": error creating ShortCircuitReplica.", e);
> > return null;
> > } finally {
> > if (replica == null) {
> > IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]);
> > }
> > }
> > return new ShortCircuitReplicaInfo(replica);
> > case ERROR_UNSUPPORTED:
> > if (!resp.hasShortCircuitAccessVersion()) {
> > LOG.warn("short-circuit read access is disabled for " +
> > "DataNode " + datanode + ". reason: " + resp.getMessage());
> > clientContext.getDomainSocketFactory()
> > .disableShortCircuitForPath(pathInfo.getPath());
> > } else {
> > LOG.warn("short-circuit read access for the file " +
> > fileName + " is disabled for DataNode " + datanode +
> > ". reason: " + resp.getMessage());
> > }
> > return null;
> > case ERROR_ACCESS_TOKEN:
> > String msg = "access control error while " +
> > "attempting to set up short-circuit access to " +
> > fileName + resp.getMessage();
> > if (LOG.isDebugEnabled()) {
> > LOG.debug(this + ":" + msg);
> > }
> > return new ShortCircuitReplicaInfo(new InvalidToken(msg));
> > default:
> > LOG.warn(this + ": unknown response code " + resp.getStatus() +
> > " while attempting to set up short-circuit access. " +
> > resp.getMessage());
> > clientContext.getDomainSocketFactory()
> > .disableShortCircuitForPath(pathInfo.getPath());
> > return null;
> > }
> >
> >
> >
> > --
> > Xie Gang
> >
>
>
>
> --
> Xie Gang
>
--
John
Re: Block invalid IOException causes the DFSClient domain socket
being disabled
Posted by Xie Gang <xi...@gmail.com>.
Shall I create the jira directly?
On Thu, Oct 26, 2017 at 12:34 PM, Xie Gang <xi...@gmail.com> wrote:
> Hi,
>
> We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain
> socket is disabled when datanode throw block invalid exception.
>
> The block is invalidated for some reason on datanote and it's OK. Then
> DFSClient tries to access this block on this datanode via domain socket.
> This triggers a IOExcetion. On DFSClient side, when get a IOExcetion and
> error code 'ERROR', it disables the domain socket and fails back to TCP.
> and the worst is that it seems never recover the socket.
>
> I think this is a defect and with such "block invalid" exception, we
> should not disable the domain socket because the is nothing wrong about the
> domain socket service.
>
> And thoughts?
>
> The code:
>
> private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer,
> Slot slot) throws IOException {
> ShortCircuitCache cache = clientContext.getShortCircuitCache();
> final DataOutputStream out =
> new DataOutputStream(new BufferedOutputStream(peer.getOutputStream()));
> SlotId slotId = slot == null ? null : slot.getSlotId();
> new Sender(out).requestShortCircuitFds(block, token, slotId, 1);
> DataInputStream in = new DataInputStream(peer.getInputStream());
> BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(
> PBHelper.vintPrefixed(in));
> DomainSocket sock = peer.getDomainSocket();
> switch (resp.getStatus()) {
> case SUCCESS:
> byte buf[] = new byte[1];
> FileInputStream fis[] = new FileInputStream[2];
> sock.recvFileInputStreams(fis, buf, 0, buf.length);
> ShortCircuitReplica replica = null;
> try {
> ExtendedBlockId key =
> new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId());
> replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,
> Time.monotonicNow(), slot);
> } catch (IOException e) {
> // This indicates an error reading from disk, or a format error. Since
> // it's not a socket communication problem, we return null rather than
> // throwing an exception.
> LOG.warn(this + ": error creating ShortCircuitReplica.", e);
> return null;
> } finally {
> if (replica == null) {
> IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]);
> }
> }
> return new ShortCircuitReplicaInfo(replica);
> case ERROR_UNSUPPORTED:
> if (!resp.hasShortCircuitAccessVersion()) {
> LOG.warn("short-circuit read access is disabled for " +
> "DataNode " + datanode + ". reason: " + resp.getMessage());
> clientContext.getDomainSocketFactory()
> .disableShortCircuitForPath(pathInfo.getPath());
> } else {
> LOG.warn("short-circuit read access for the file " +
> fileName + " is disabled for DataNode " + datanode +
> ". reason: " + resp.getMessage());
> }
> return null;
> case ERROR_ACCESS_TOKEN:
> String msg = "access control error while " +
> "attempting to set up short-circuit access to " +
> fileName + resp.getMessage();
> if (LOG.isDebugEnabled()) {
> LOG.debug(this + ":" + msg);
> }
> return new ShortCircuitReplicaInfo(new InvalidToken(msg));
> default:
> LOG.warn(this + ": unknown response code " + resp.getStatus() +
> " while attempting to set up short-circuit access. " +
> resp.getMessage());
> clientContext.getDomainSocketFactory()
> .disableShortCircuitForPath(pathInfo.getPath());
> return null;
> }
>
>
>
> --
> Xie Gang
>
--
Xie Gang
Re: Block invalid IOException causes the DFSClient domain socket
being disabled
Posted by John Zhuge <jo...@gmail.com>.
Hi Xie Gang,
Thanks for reporting the issue. How was the block invalidated on DN?
I recently created a unit test that might trigger a somewhat similar
symptom:
1. Read a 4000-byte file in MiniDFS. It should go thru domain socket.
2. Call input stream unbuffer. It will close the block reader.
3. Somehow evict the replica from ShortCircuitCache. This may occur
quickly on a busy node. The unit test triggers the behavior by setting
cache's max total size 0.
4. Append the file. It will cause the meta file to be changed. The block
file remains the same in this test.
5. Re-read the file. The logs show a TCP block reader used. Saw this log:
WARN impl.BlockReaderFactory
> (BlockReaderFactory.java:requestFileDescriptors(647)) -
> BlockReaderFactory(fileName=/test_file,
> block=BP-1017908210-192.168.0.103-1508995502008:blk_1073741825_1001):
> unknown response code ERROR while attempting to set up short-circuit
> access. No data exists for block
> BP-1017908210-192.168.0.103-1508995502008:blk_1073741825_1001
The unit test patch:
https://issues.apache.org/jira/secure/attachment/12893637/HDFS-12528.000.patch
Thanks,
John
On Wed, Oct 25, 2017 at 9:34 PM, Xie Gang <xi...@gmail.com> wrote:
> Hi,
>
> We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain socket
> is disabled when datanode throw block invalid exception.
>
> The block is invalidated for some reason on datanote and it's OK. Then
> DFSClient tries to access this block on this datanode via domain socket.
> This triggers a IOExcetion. On DFSClient side, when get a IOExcetion and
> error code 'ERROR', it disables the domain socket and fails back to TCP.
> and the worst is that it seems never recover the socket.
>
> I think this is a defect and with such "block invalid" exception, we should
> not disable the domain socket because the is nothing wrong about the domain
> socket service.
>
> And thoughts?
>
> The code:
>
> private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer,
> Slot slot) throws IOException {
> ShortCircuitCache cache = clientContext.getShortCircuitCache();
> final DataOutputStream out =
> new DataOutputStream(new BufferedOutputStream(peer.
> getOutputStream()));
> SlotId slotId = slot == null ? null : slot.getSlotId();
> new Sender(out).requestShortCircuitFds(block, token, slotId, 1);
> DataInputStream in = new DataInputStream(peer.getInputStream());
> BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(
> PBHelper.vintPrefixed(in));
> DomainSocket sock = peer.getDomainSocket();
> switch (resp.getStatus()) {
> case SUCCESS:
> byte buf[] = new byte[1];
> FileInputStream fis[] = new FileInputStream[2];
> sock.recvFileInputStreams(fis, buf, 0, buf.length);
> ShortCircuitReplica replica = null;
> try {
> ExtendedBlockId key =
> new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId());
> replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,
> Time.monotonicNow(), slot);
> } catch (IOException e) {
> // This indicates an error reading from disk, or a format error.
> Since
> // it's not a socket communication problem, we return null rather
> than
> // throwing an exception.
> LOG.warn(this + ": error creating ShortCircuitReplica.", e);
> return null;
> } finally {
> if (replica == null) {
> IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]);
> }
> }
> return new ShortCircuitReplicaInfo(replica);
> case ERROR_UNSUPPORTED:
> if (!resp.hasShortCircuitAccessVersion()) {
> LOG.warn("short-circuit read access is disabled for " +
> "DataNode " + datanode + ". reason: " + resp.getMessage());
> clientContext.getDomainSocketFactory()
> .disableShortCircuitForPath(pathInfo.getPath());
> } else {
> LOG.warn("short-circuit read access for the file " +
> fileName + " is disabled for DataNode " + datanode +
> ". reason: " + resp.getMessage());
> }
> return null;
> case ERROR_ACCESS_TOKEN:
> String msg = "access control error while " +
> "attempting to set up short-circuit access to " +
> fileName + resp.getMessage();
> if (LOG.isDebugEnabled()) {
> LOG.debug(this + ":" + msg);
> }
> return new ShortCircuitReplicaInfo(new InvalidToken(msg));
> default:
> LOG.warn(this + ": unknown response code " + resp.getStatus() +
> " while attempting to set up short-circuit access. " +
> resp.getMessage());
> clientContext.getDomainSocketFactory()
> .disableShortCircuitForPath(pathInfo.getPath());
> return null;
> }
>
>
>
> --
> Xie Gang
>
--
John