You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/06/15 08:52:25 UTC
[GitHub] [ozone] ChenSammi opened a new pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
ChenSammi opened a new pull request #2337:
URL: https://github.com/apache/ozone/pull/2337
https://issues.apache.org/jira/browse/HDDS-5265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] github-actions[bot] commented on pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#issuecomment-966713761
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r656911267
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
##########
@@ -827,12 +829,25 @@ ContainerCommandResponseProto handleGetSmallFile(
ChunkBuffer data = chunkManager.readChunk(kvContainer, blockID,
chunkInfo, dispatcherContext);
dataBuffers.addAll(data.toByteStringList(byteBufferToByteString));
- chunkInfoProto = chunk;
+ chunk.getChecksumData().getChecksumsList().stream().forEach(
+ (ck -> checksumBuilder.addChecksums(ck)));
Review comment:
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] github-actions[bot] closed pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #2337:
URL: https://github.com/apache/ozone/pull/2337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r656914552
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
##########
@@ -382,10 +382,10 @@ private XceiverClientReply sendCommandWithRetry(
} catch (IOException e) {
ioException = e;
responseProto = null;
- LOG.debug("Failed to execute command {} on datanode {}",
+ LOG.error("Failed to execute command {} on datanode {}",
Review comment:
Will revert the log level change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#issuecomment-871960594
@adoroszlai @bshashikant for the code review. New commits are uploaded to address the comments.
The failed integration test is about TestSCMInstallSnapshot, which is not relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#issuecomment-948697977
/pending `TestKeyInputStream` is still timing out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r656893104
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
##########
@@ -382,10 +382,10 @@ private XceiverClientReply sendCommandWithRetry(
} catch (IOException e) {
ioException = e;
responseProto = null;
- LOG.debug("Failed to execute command {} on datanode {}",
+ LOG.error("Failed to execute command {} on datanode {}",
Review comment:
This may lead of verbose logging on client logs/console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r724779761
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##########
@@ -109,45 +111,83 @@
private int chunkIndexOfPrevPosition;
private final Function<BlockID, Pipeline> refreshPipelineFunction;
+ private boolean smallBlock = false;
+ private final OzoneClientConfig clientConfig;
+ @SuppressWarnings("parameternumber")
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
XceiverClientFactory xceiverClientFactory,
- Function<BlockID, Pipeline> refreshPipelineFunction) {
+ Function<BlockID, Pipeline> refreshPipelineFunction,
+ OzoneClientConfig clientConfig) {
this.blockID = blockId;
this.length = blockLen;
this.pipeline = pipeline;
this.token = token;
this.verifyChecksum = verifyChecksum;
this.xceiverClientFactory = xceiverClientFactory;
this.refreshPipelineFunction = refreshPipelineFunction;
+ if (clientConfig != null) {
+ this.smallBlock = (length <= clientConfig.getSmallBlockThreshold());
+ }
+ this.clientConfig = clientConfig;
+ }
+
+ public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory,
+ Function<BlockID, Pipeline> refreshPipelineFunction) {
+ this(blockId, blockLen, pipeline, token, verifyChecksum,
+ xceiverClientFactory, refreshPipelineFunction, null);
}
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
- Token<OzoneBlockTokenIdentifier> token,
- boolean verifyChecksum,
- XceiverClientFactory xceiverClientFactory
- ) {
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory) {
this(blockId, blockLen, pipeline, token, verifyChecksum,
- xceiverClientFactory, null);
+ xceiverClientFactory, null, null);
}
/**
* Initialize the BlockInputStream. Get the BlockData (list of chunks) from
* the Container and create the ChunkInputStreams for each Chunk in the Block.
*/
public synchronized void initialize() throws IOException {
-
- // Pre-check that the stream has not been intialized already
+ // Pre-check that the stream has not been initialized already
if (initialized) {
return;
}
+ // Initialize pipeline and client.
+ // irrespective of the container state, we will always read via Standalone
+ // protocol.
+ if (pipeline.getType() != HddsProtos.ReplicationType.STAND_ALONE) {
+ pipeline = Pipeline.newBuilder(pipeline)
+ .setReplicationConfig(new StandaloneReplicationConfig(
+ ReplicationConfig
+ .getLegacyFactor(pipeline.getReplicationConfig())))
+ .build();
+ }
+ acquireClient();
+
List<ChunkInfo> chunks;
- try {
- chunks = getChunkInfos();
- } catch (ContainerNotFoundException ioEx) {
- refreshPipeline(ioEx);
- chunks = getChunkInfos();
+ if (smallBlock) {
+ ChunkInfo chunkInfo = ChunkInfo.newBuilder()
+ .setChunkName(blockID.getLocalID() + "_chunk_" + (chunkIndex + 1))
+ .setOffset(0L).setLen(length)
+ .setChecksumData(ContainerProtos.ChecksumData.newBuilder()
+ .setBytesPerChecksum(clientConfig.getBytesPerChecksum())
+ .setType(clientConfig.getChecksumType())
+ .build())
+ .build();
+ chunks = new ArrayList<>();
+ chunks.add(chunkInfo);
+ } else {
+ try {
+ chunks = getChunkInfos();
+ } catch (ContainerNotFoundException ioEx) {
+ refreshPipeline(ioEx);
Review comment:
@ChenSammi thanks for the update. Now if pipeline is refreshed, new client gets created, but old client seems to be leaked. I guess here we could simply call `handleReadError` instead of `refreshPipeline` to release old client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#issuecomment-874741582
Thanks @ChenSammi for updating the patch.
> The failed integration test is about TestSCMInstallSnapshot, which is not relevant.
Fork for `TestKeyInputStream` timed out in the most recent run. Can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#issuecomment-921660029
`TestKeyInputStream` is still timing out:
https://github.com/adoroszlai/hadoop-ozone/runs/3624602677#step:4:7973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r655872160
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##########
@@ -1315,7 +1315,7 @@ private OzoneInputStream createInputStream(
if (feInfo == null) {
LengthInputStream lengthInputStream = KeyInputStream
.getFromOmKeyInfo(keyInfo, xceiverClientManager,
- clientConfig.isChecksumVerify(), retryFunction);
+ clientConfig.isChecksumVerify(), retryFunction, clientConfig);
Review comment:
Parameter `verifyChecksum` could be removed by getting it via `clientConfig.isChecksumVerify()` only in `BlockInputStream` constructor.
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##########
@@ -109,45 +111,83 @@
private int chunkIndexOfPrevPosition;
private final Function<BlockID, Pipeline> refreshPipelineFunction;
+ private boolean smallBlock = false;
+ private final OzoneClientConfig clientConfig;
+ @SuppressWarnings("parameternumber")
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
XceiverClientFactory xceiverClientFactory,
- Function<BlockID, Pipeline> refreshPipelineFunction) {
+ Function<BlockID, Pipeline> refreshPipelineFunction,
+ OzoneClientConfig clientConfig) {
this.blockID = blockId;
this.length = blockLen;
this.pipeline = pipeline;
this.token = token;
this.verifyChecksum = verifyChecksum;
this.xceiverClientFactory = xceiverClientFactory;
this.refreshPipelineFunction = refreshPipelineFunction;
+ if (clientConfig != null) {
+ this.smallBlock = (length <= clientConfig.getSmallBlockThreshold());
+ }
+ this.clientConfig = clientConfig;
+ }
+
+ public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory,
+ Function<BlockID, Pipeline> refreshPipelineFunction) {
+ this(blockId, blockLen, pipeline, token, verifyChecksum,
+ xceiverClientFactory, refreshPipelineFunction, null);
}
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
- Token<OzoneBlockTokenIdentifier> token,
- boolean verifyChecksum,
- XceiverClientFactory xceiverClientFactory
- ) {
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory) {
this(blockId, blockLen, pipeline, token, verifyChecksum,
- xceiverClientFactory, null);
+ xceiverClientFactory, null, null);
Review comment:
Can we pass `new OzoneClientConfig()` instead of `null`? This way we can avoid null checks and use the default configuration seamlessly.
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##########
@@ -109,45 +111,83 @@
private int chunkIndexOfPrevPosition;
private final Function<BlockID, Pipeline> refreshPipelineFunction;
+ private boolean smallBlock = false;
+ private final OzoneClientConfig clientConfig;
+ @SuppressWarnings("parameternumber")
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
XceiverClientFactory xceiverClientFactory,
- Function<BlockID, Pipeline> refreshPipelineFunction) {
+ Function<BlockID, Pipeline> refreshPipelineFunction,
+ OzoneClientConfig clientConfig) {
this.blockID = blockId;
this.length = blockLen;
this.pipeline = pipeline;
this.token = token;
this.verifyChecksum = verifyChecksum;
this.xceiverClientFactory = xceiverClientFactory;
this.refreshPipelineFunction = refreshPipelineFunction;
+ if (clientConfig != null) {
+ this.smallBlock = (length <= clientConfig.getSmallBlockThreshold());
+ }
+ this.clientConfig = clientConfig;
+ }
+
+ public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory,
+ Function<BlockID, Pipeline> refreshPipelineFunction) {
+ this(blockId, blockLen, pipeline, token, verifyChecksum,
+ xceiverClientFactory, refreshPipelineFunction, null);
}
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
- Token<OzoneBlockTokenIdentifier> token,
- boolean verifyChecksum,
- XceiverClientFactory xceiverClientFactory
- ) {
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory) {
this(blockId, blockLen, pipeline, token, verifyChecksum,
- xceiverClientFactory, null);
+ xceiverClientFactory, null, null);
}
/**
* Initialize the BlockInputStream. Get the BlockData (list of chunks) from
* the Container and create the ChunkInputStreams for each Chunk in the Block.
*/
public synchronized void initialize() throws IOException {
-
- // Pre-check that the stream has not been intialized already
+ // Pre-check that the stream has not been initialized already
if (initialized) {
return;
}
+ // Initialize pipeline and client.
+ // irrespective of the container state, we will always read via Standalone
+ // protocol.
+ if (pipeline.getType() != HddsProtos.ReplicationType.STAND_ALONE) {
+ pipeline = Pipeline.newBuilder(pipeline)
+ .setReplicationConfig(new StandaloneReplicationConfig(
+ ReplicationConfig
+ .getLegacyFactor(pipeline.getReplicationConfig())))
+ .build();
+ }
+ acquireClient();
+
List<ChunkInfo> chunks;
- try {
- chunks = getChunkInfos();
- } catch (ContainerNotFoundException ioEx) {
- refreshPipeline(ioEx);
- chunks = getChunkInfos();
+ if (smallBlock) {
+ ChunkInfo chunkInfo = ChunkInfo.newBuilder()
+ .setChunkName(blockID.getLocalID() + "_chunk_" + (chunkIndex + 1))
+ .setOffset(0L).setLen(length)
+ .setChecksumData(ContainerProtos.ChecksumData.newBuilder()
+ .setBytesPerChecksum(clientConfig.getBytesPerChecksum())
+ .setType(clientConfig.getChecksumType())
+ .build())
+ .build();
+ chunks = new ArrayList<>();
+ chunks.add(chunkInfo);
+ } else {
+ try {
+ chunks = getChunkInfos();
+ } catch (ContainerNotFoundException ioEx) {
+ refreshPipeline(ioEx);
Review comment:
`refreshPipeline` may update the pipeline, but client still references the old one. New one may need to be acquired. Also, new pipeline needs to be converted to standalone, too.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestInputStreamBase.java
##########
@@ -85,12 +87,22 @@
public Timeout timeout = Timeout.seconds(300);
@Parameterized.Parameters
- public static Iterable<Object[]> parameters() {
- return ChunkLayoutTestInfo.chunkLayoutParameters();
+ public static Collection<Object[]> layouts() {
+ return Arrays.asList(new Object[][] {
+ {ChunkLayOutVersion.FILE_PER_CHUNK, 0},
+ {ChunkLayOutVersion.FILE_PER_CHUNK, BYTES_PER_CHECKSUM},
+ {ChunkLayOutVersion.FILE_PER_CHUNK, CHUNK_SIZE},
+ {ChunkLayOutVersion.FILE_PER_CHUNK, BLOCK_SIZE},
+ {ChunkLayOutVersion.FILE_PER_BLOCK, 0},
+ {ChunkLayOutVersion.FILE_PER_BLOCK, BYTES_PER_CHECKSUM},
+ {ChunkLayOutVersion.FILE_PER_BLOCK, CHUNK_SIZE},
+ {ChunkLayOutVersion.FILE_PER_BLOCK, BLOCK_SIZE}
Review comment:
This increases integration test run time significantly (almost *an hour* for only `TestChunkInputStream` and `TestKeyInputStream` on my machine). Are all these combinations really necessary? Do all test cases depend on the value of `blockThreshold`?
Nit: manually enumerating the parameter matrix seems too verbose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r657056266
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##########
@@ -109,45 +111,83 @@
private int chunkIndexOfPrevPosition;
private final Function<BlockID, Pipeline> refreshPipelineFunction;
+ private boolean smallBlock = false;
+ private final OzoneClientConfig clientConfig;
+ @SuppressWarnings("parameternumber")
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
XceiverClientFactory xceiverClientFactory,
- Function<BlockID, Pipeline> refreshPipelineFunction) {
+ Function<BlockID, Pipeline> refreshPipelineFunction,
+ OzoneClientConfig clientConfig) {
this.blockID = blockId;
this.length = blockLen;
this.pipeline = pipeline;
this.token = token;
this.verifyChecksum = verifyChecksum;
this.xceiverClientFactory = xceiverClientFactory;
this.refreshPipelineFunction = refreshPipelineFunction;
+ if (clientConfig != null) {
+ this.smallBlock = (length <= clientConfig.getSmallBlockThreshold());
+ }
+ this.clientConfig = clientConfig;
+ }
+
+ public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory,
+ Function<BlockID, Pipeline> refreshPipelineFunction) {
+ this(blockId, blockLen, pipeline, token, verifyChecksum,
+ xceiverClientFactory, refreshPipelineFunction, null);
}
public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline,
- Token<OzoneBlockTokenIdentifier> token,
- boolean verifyChecksum,
- XceiverClientFactory xceiverClientFactory
- ) {
+ Token<OzoneBlockTokenIdentifier> token, boolean verifyChecksum,
+ XceiverClientFactory xceiverClientFactory) {
this(blockId, blockLen, pipeline, token, verifyChecksum,
- xceiverClientFactory, null);
+ xceiverClientFactory, null, null);
}
/**
* Initialize the BlockInputStream. Get the BlockData (list of chunks) from
* the Container and create the ChunkInputStreams for each Chunk in the Block.
*/
public synchronized void initialize() throws IOException {
-
- // Pre-check that the stream has not been intialized already
+ // Pre-check that the stream has not been initialized already
if (initialized) {
return;
}
+ // Initialize pipeline and client.
+ // irrespective of the container state, we will always read via Standalone
+ // protocol.
+ if (pipeline.getType() != HddsProtos.ReplicationType.STAND_ALONE) {
+ pipeline = Pipeline.newBuilder(pipeline)
+ .setReplicationConfig(new StandaloneReplicationConfig(
+ ReplicationConfig
+ .getLegacyFactor(pipeline.getReplicationConfig())))
+ .build();
+ }
+ acquireClient();
+
List<ChunkInfo> chunks;
- try {
- chunks = getChunkInfos();
- } catch (ContainerNotFoundException ioEx) {
- refreshPipeline(ioEx);
- chunks = getChunkInfos();
+ if (smallBlock) {
+ ChunkInfo chunkInfo = ChunkInfo.newBuilder()
+ .setChunkName(blockID.getLocalID() + "_chunk_" + (chunkIndex + 1))
+ .setOffset(0L).setLen(length)
+ .setChecksumData(ContainerProtos.ChecksumData.newBuilder()
+ .setBytesPerChecksum(clientConfig.getBytesPerChecksum())
+ .setType(clientConfig.getChecksumType())
+ .build())
+ .build();
+ chunks = new ArrayList<>();
+ chunks.add(chunkInfo);
+ } else {
+ try {
+ chunks = getChunkInfos();
+ } catch (ContainerNotFoundException ioEx) {
+ refreshPipeline(ioEx);
Review comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2337: HDDS-5265. Leverage getSmallFile in ChunkInputStream
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2337:
URL: https://github.com/apache/ozone/pull/2337#discussion_r656327745
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
##########
@@ -827,12 +829,25 @@ ContainerCommandResponseProto handleGetSmallFile(
ChunkBuffer data = chunkManager.readChunk(kvContainer, blockID,
chunkInfo, dispatcherContext);
dataBuffers.addAll(data.toByteStringList(byteBufferToByteString));
- chunkInfoProto = chunk;
+ chunk.getChecksumData().getChecksumsList().stream().forEach(
+ (ck -> checksumBuilder.addChecksums(ck)));
Review comment:
Please try to avoid streams on read/write path ([HDDS-3702](https://issues.apache.org/jira/browse/HDDS-3702)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org