You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "Flaugh24 (via GitHub)" <gi...@apache.org> on 2023/08/01 10:43:12 UTC
[GitHub] [ignite-3] Flaugh24 opened a new pull request, #2390: IGNITE-19009 WIP
Flaugh24 opened a new pull request, #2390:
URL: https://github.com/apache/ignite-3/pull/2390
https://issues.apache.org/jira/browse/IGNITE-19009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281702018
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringService.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+
+/**
+ * File transferring service.
+ */
+public interface FileTransferringService extends IgniteComponent {
+ /**
+ * Adds a file provider for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param provider Provider.
+ */
+ <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ );
+
+ /**
+ * Adds a file handler for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param handler Handler.
+ */
+ <M extends TransferMetadata> void addFileHandler(Class<M> metadata, FileHandler<M> handler);
Review Comment:
I don't think so. From my point of view, we should register all providers and handlers on the Ignite startup, and use them during the whole runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296016199
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -488,10 +488,7 @@ void uploadFilesWhenDoNotHaveAccessToWrite() {
);
// Check that files transfer failed.
- assertThat(
- uploaded,
- willThrow(FileTransferException.class, "Failed to upload files:")
- );
+ assertThat(uploaded, willThrow(FileTransferException.class));
Review Comment:
Got it back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286171140
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
+ private final String fileName;
+ private final RandomAccessFile raf;
+ private final int chunkSize;
+
+ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize) {
+ this.fileName = fileName;
+ this.raf = raf;
+ this.chunkSize = chunkSize;
+ }
+
+ static ChunkedFileReader open(File file, int chunkSize) throws IOException {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ return new ChunkedFileReader(file.getName(), raf, chunkSize);
+ }
+
+ byte[] readNextChunk() throws IOException {
+ int toRead = (int) Math.min(chunkSize, raf.length() - raf.getFilePointer());
Review Comment:
What do you mean? `java.io.RandomAccessFile#getFilePointer` returns offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1290205594
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/MessagesUtils.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toList;
+
+import java.nio.file.Path;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+
+/**
+ * Utility class for file transfer messages.
+ */
+final class MessagesUtils {
+ private MessagesUtils() {
+ // No-op.
+ }
+
+ /**
+ * Extracts file headers from files.
+ *
+ * @param factory File transfer factory.
+ * @param paths List of paths.
+ * @return List of file headers.
+ */
+ static List<FileHeader> getHeaders(FileTransferFactory factory, List<Path> paths) {
Review Comment:
It is used in two classes. I don't know what is better - create two small copies or have this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] rpuch commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "rpuch (via GitHub)" <gi...@apache.org>.
rpuch commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286667487
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
Review Comment:
Why is this queue needed? Can the chunks be written out-of-order?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
Review Comment:
This is called from a network thread. No blocking/IO operations should be run on this thread
##########
modules/network/src/main/java/org/apache/ignite/internal/network/configuration/NetworkConfigurationSchema.java:
##########
@@ -78,4 +78,8 @@ public class NetworkConfigurationSchema {
@ConfigValue
@SslConfigurationValidator
public SslConfigurationSchema ssl;
+
+ /** File transferring configuration. */
+ @ConfigValue
+ public FileTransferConfigurationSchema fileTransferring;
Review Comment:
```suggestion
public FileTransferConfigurationSchema fileTransfer;
```
##########
modules/file-transfer/src/testFixtures/java/org/apache/ignite/internal/network/file/FileUtils.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * File utils.
+ */
+public class FileUtils {
+ public static List<File> sortByNames(File... files) {
+ return sortByNames(List.of(files));
+ }
+
+ /**
+ * Sorts files by names.
+ *
+ * @param files Files.
+ * @return Sorted files.
+ */
+ public static List<File> sortByNames(List<File> files) {
+ return files.stream()
+ .sorted(Comparator.comparing(File::getName))
+ .collect(Collectors.toList());
Review Comment:
Let's import `toList()` statically, this would make the line read as an English phrase
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
It looks like there are not tests for this class. Let's add them.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ error.compareAndSet(null, e);
+ }
+ } finally {
+ rateLimiter.release();
+ }
+ });
+ }
+ }
+
+ if (error.get() != null) {
Review Comment:
Do we wait for the sends to complete somewhere? If not, we'll probably miss an error if it happens
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
Review Comment:
Let's use another variant of `IgniteLogger.error()`: the one that takes a Throwable as a second argument. Then the exception does not need to be mentioned explicitly in the log message.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
Review Comment:
`PriorityQueue` is not thread-safe. If there a guarantee that the queue is always accessed from the same thread?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
Review Comment:
This is called from a network thread. No blocking/IO operations should be run on this thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1289775544
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
+ */
+public interface FileConsumer<M extends Metadata> {
+ /**
+ * Handles the uploaded files.
+ *
+ * @param metadata The metadata of the uploaded file.
+ * @param uploadedFiles The temporary files that were uploaded. These files will be deleted after the method returns.
Review Comment:
Is that a good idea to use temp dir? It may not have enough space, we should always use explicitly configured directories to write any data, like "workDir", for example. You can't make such assumption about user's environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1289776938
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
Review Comment:
What I see right now is two channels with matching type, it's a bug, that's why I asked. If it will be replaced in the future - the type will also be changed in the futures accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1291064723
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream().map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ transfers.forEach(this::processTransfer);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransfer(FileTransfer transfer) {
+ synchronized (rateLimiter) {
Review Comment:
I suppose there is a chance that one thread which has a permit can check the queue and see it's empty. At the same time, the other thread can't get a permit from the limiter and places a transfer in the queue. As a result, there will be a transfer in the queue, and no one will process it until the Sender gets the next transfer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] PakhomovAlexander merged pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander merged PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1289782628
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
Review Comment:
I don't see how this clarifies my question. You're doing "messagingService.respond(...).thenCompose(...)", how is it related to executors in FileSender/FileReceiver? Do we guarantee that "handler.result()" will never be completed in that case, so that "handleUpload" is in its proper pool? I'd love to see that in a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288156410
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
Review Comment:
See org.apache.ignite.internal.network.file.FileTransferMessagesHandler#handleFileChunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1291026316
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream().map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ transfers.forEach(this::processTransfer);
Review Comment:
This code could be much simpler if "processTransfer" returned the "result" future. That way you wouldn't need additional "transfers" collection. _I think_
It's actually pretty difficult to understand the entire thing)
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream()
+ .peek(this::processTransferAsync)
+ .map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransferAsync(FileTransfer transfer) {
+ synchronized (rateLimiter) {
+ if (rateLimiter.tryAcquire()) {
+ processTransferWithNextAsync(transfer);
+ } else {
+ requests.add(transfer);
+ }
+ }
+ }
+
+ /**
+ * Process the given transfer and then process the next transfer if the rate limiter is available and there is a next transfer.
+ *
+ * @return A future that will be completed when the request is processed.
+ */
+ private CompletableFuture<Void> processTransferWithNextAsync(FileTransfer transfer) {
+ return sendTransfer(transfer)
+ .thenComposeAsync(v -> {
+ synchronized (rateLimiter) {
+ FileTransfer nextTransfer = requests.poll();
+
+ // If there is a next transfer, process it.
+ // Otherwise, release the rate limiter.
+ if (nextTransfer != null) {
+ return processTransferWithNextAsync(nextTransfer);
+ } else {
+ rateLimiter.release();
+ return completedFuture(null);
+ }
+ }
+ }, executorService);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param transfer The transfer to send.
+ * @return A future that will be completed when the file is sent. The future will be completed successfully always, even if the file is
+ * not sent.
+ */
+ private CompletableFuture<Void> sendTransfer(FileTransfer transfer) {
+ return sendFile(transfer.receiverConsistentId, transfer.transferId, transfer.path, transfer.shouldBeCancelled)
+ .handle((v, e) -> {
+ if (e == null) {
+ transfer.result.complete(null);
+ } else {
+ transfer.result.completeExceptionally(e);
+ }
+ return null;
+ });
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
+ } else {
+ return supplyAsync(() -> {
+ try {
+ return FileChunkMessagesStream.fromPath(chunkSize, id, path);
+ } catch (IOException e) {
+ throw new FileTransferException("Failed to create a file transfer stream", e);
+ }
+ }, executorService)
+ .thenCompose(stream -> {
Review Comment:
What's the deal with "thenCompose"? You could put this code right into "supplyAsync"?
Please explain, don't just close the conversation.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream().map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ transfers.forEach(this::processTransfer);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransfer(FileTransfer transfer) {
+ synchronized (rateLimiter) {
Review Comment:
Please document, how and why you use this synchronization. If it protects the "requests" queue, then why do you use rate limiter as a monitor? Not very intuitive to me
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
Review Comment:
So, what exactly are we limiting? The number of files that we send concurrently?
Can we send a single file concurrently (sending N chunks in parallel)?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream()
+ .peek(this::processTransferAsync)
+ .map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransferAsync(FileTransfer transfer) {
+ synchronized (rateLimiter) {
+ if (rateLimiter.tryAcquire()) {
+ processTransferWithNextAsync(transfer);
+ } else {
+ requests.add(transfer);
+ }
+ }
+ }
+
+ /**
+ * Process the given transfer and then process the next transfer if the rate limiter is available and there is a next transfer.
+ *
+ * @return A future that will be completed when the request is processed.
+ */
+ private CompletableFuture<Void> processTransferWithNextAsync(FileTransfer transfer) {
+ return sendTransfer(transfer)
+ .thenComposeAsync(v -> {
+ synchronized (rateLimiter) {
+ FileTransfer nextTransfer = requests.poll();
+
+ // If there is a next transfer, process it.
+ // Otherwise, release the rate limiter.
+ if (nextTransfer != null) {
+ return processTransferWithNextAsync(nextTransfer);
+ } else {
+ rateLimiter.release();
+ return completedFuture(null);
+ }
+ }
+ }, executorService);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param transfer The transfer to send.
+ * @return A future that will be completed when the file is sent. The future will be completed successfully always, even if the file is
+ * not sent.
+ */
+ private CompletableFuture<Void> sendTransfer(FileTransfer transfer) {
+ return sendFile(transfer.receiverConsistentId, transfer.transferId, transfer.path, transfer.shouldBeCancelled)
+ .handle((v, e) -> {
+ if (e == null) {
+ transfer.result.complete(null);
+ } else {
+ transfer.result.completeExceptionally(e);
+ }
+ return null;
+ });
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
Review Comment:
Why don't we send empty files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1291057739
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream()
+ .peek(this::processTransferAsync)
+ .map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransferAsync(FileTransfer transfer) {
+ synchronized (rateLimiter) {
+ if (rateLimiter.tryAcquire()) {
+ processTransferWithNextAsync(transfer);
+ } else {
+ requests.add(transfer);
+ }
+ }
+ }
+
+ /**
+ * Process the given transfer and then process the next transfer if the rate limiter is available and there is a next transfer.
+ *
+ * @return A future that will be completed when the request is processed.
+ */
+ private CompletableFuture<Void> processTransferWithNextAsync(FileTransfer transfer) {
+ return sendTransfer(transfer)
+ .thenComposeAsync(v -> {
+ synchronized (rateLimiter) {
+ FileTransfer nextTransfer = requests.poll();
+
+ // If there is a next transfer, process it.
+ // Otherwise, release the rate limiter.
+ if (nextTransfer != null) {
+ return processTransferWithNextAsync(nextTransfer);
+ } else {
+ rateLimiter.release();
+ return completedFuture(null);
+ }
+ }
+ }, executorService);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param transfer The transfer to send.
+ * @return A future that will be completed when the file is sent. The future will be completed successfully always, even if the file is
+ * not sent.
+ */
+ private CompletableFuture<Void> sendTransfer(FileTransfer transfer) {
+ return sendFile(transfer.receiverConsistentId, transfer.transferId, transfer.path, transfer.shouldBeCancelled)
+ .handle((v, e) -> {
+ if (e == null) {
+ transfer.result.complete(null);
+ } else {
+ transfer.result.completeExceptionally(e);
+ }
+ return null;
+ });
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
Review Comment:
The original idea was not to acquire the rate limiter, when the file is empty. But now it doesn't make sense.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream()
+ .peek(this::processTransferAsync)
+ .map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransferAsync(FileTransfer transfer) {
+ synchronized (rateLimiter) {
+ if (rateLimiter.tryAcquire()) {
+ processTransferWithNextAsync(transfer);
+ } else {
+ requests.add(transfer);
+ }
+ }
+ }
+
+ /**
+ * Process the given transfer and then process the next transfer if the rate limiter is available and there is a next transfer.
+ *
+ * @return A future that will be completed when the request is processed.
+ */
+ private CompletableFuture<Void> processTransferWithNextAsync(FileTransfer transfer) {
+ return sendTransfer(transfer)
+ .thenComposeAsync(v -> {
+ synchronized (rateLimiter) {
+ FileTransfer nextTransfer = requests.poll();
+
+ // If there is a next transfer, process it.
+ // Otherwise, release the rate limiter.
+ if (nextTransfer != null) {
+ return processTransferWithNextAsync(nextTransfer);
+ } else {
+ rateLimiter.release();
+ return completedFuture(null);
+ }
+ }
+ }, executorService);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param transfer The transfer to send.
+ * @return A future that will be completed when the file is sent. The future will be completed successfully always, even if the file is
+ * not sent.
+ */
+ private CompletableFuture<Void> sendTransfer(FileTransfer transfer) {
+ return sendFile(transfer.receiverConsistentId, transfer.transferId, transfer.path, transfer.shouldBeCancelled)
+ .handle((v, e) -> {
+ if (e == null) {
+ transfer.result.complete(null);
+ } else {
+ transfer.result.completeExceptionally(e);
+ }
+ return null;
+ });
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
Review Comment:
The original idea was not to acquire the rate limiter when the file is empty. But now it doesn't make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1291056487
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final ExecutorService executorService;
+
+ private final Queue<FileTransfer> requests = new LinkedList<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService,
+ ExecutorService executorService
+ ) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Creates {@link FileTransfer} objects for each file and places them in the queue. The files will be sent when the rate limiter is
+ * available.
+ *
+ * @param targetNodeConsistentId The consistent ID of the node to send the files to.
+ * @param transferId The ID of the transfer.
+ * @param paths The paths of the files to send.
+ * @return A future that will be completed when the transfer is complete.
+ */
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ List<FileTransfer> transfers = paths.stream()
+ .map(path -> new FileTransfer(targetNodeConsistentId, transferId, path, shouldBeCancelled))
+ .collect(Collectors.toList());
+
+ CompletableFuture[] results = transfers.stream()
+ .peek(this::processTransferAsync)
+ .map(it -> it.result)
+ .map(it -> it.whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return allOf(results);
+ }
+
+ /**
+ * Processes the given transfer. If the rate limiter is not available, the transfer will be processed later.
+ *
+ * @param transfer The transfer to process.
+ */
+ private void processTransferAsync(FileTransfer transfer) {
+ synchronized (rateLimiter) {
+ if (rateLimiter.tryAcquire()) {
+ processTransferWithNextAsync(transfer);
+ } else {
+ requests.add(transfer);
+ }
+ }
+ }
+
+ /**
+ * Process the given transfer and then process the next transfer if the rate limiter is available and there is a next transfer.
+ *
+ * @return A future that will be completed when the request is processed.
+ */
+ private CompletableFuture<Void> processTransferWithNextAsync(FileTransfer transfer) {
+ return sendTransfer(transfer)
+ .thenComposeAsync(v -> {
+ synchronized (rateLimiter) {
+ FileTransfer nextTransfer = requests.poll();
+
+ // If there is a next transfer, process it.
+ // Otherwise, release the rate limiter.
+ if (nextTransfer != null) {
+ return processTransferWithNextAsync(nextTransfer);
+ } else {
+ rateLimiter.release();
+ return completedFuture(null);
+ }
+ }
+ }, executorService);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param transfer The transfer to send.
+ * @return A future that will be completed when the file is sent. The future will be completed successfully always, even if the file is
+ * not sent.
+ */
+ private CompletableFuture<Void> sendTransfer(FileTransfer transfer) {
+ return sendFile(transfer.receiverConsistentId, transfer.transferId, transfer.path, transfer.shouldBeCancelled)
+ .handle((v, e) -> {
+ if (e == null) {
+ transfer.result.complete(null);
+ } else {
+ transfer.result.completeExceptionally(e);
+ }
+ return null;
+ });
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
+ } else {
+ return supplyAsync(() -> {
+ try {
+ return FileChunkMessagesStream.fromPath(chunkSize, id, path);
+ } catch (IOException e) {
+ throw new FileTransferException("Failed to create a file transfer stream", e);
+ }
+ }, executorService)
+ .thenCompose(stream -> {
Review Comment:
The current implementation limits the number of files that we send concurrently, so we have to wait for the finish of the transfer of one file to start another one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281705602
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
Review Comment:
because org.apache.ignite.network.annotations.Transferable#value is short
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] rpuch commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "rpuch (via GitHub)" <gi...@apache.org>.
rpuch commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286743617
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
Review Comment:
My bad
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
Review Comment:
My bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1290256800
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
+
+ private final MessagingService messagingService;
+
+ private final Queue<FileTransferRequest> requests = new ConcurrentLinkedQueue<>();
+
+ private final Map<FileTransferRequest, CompletableFuture<Void>> results = new ConcurrentHashMap<>();
+
+ FileSender(
+ int chunkSize,
+ Semaphore rateLimiter,
+ MessagingService messagingService) {
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.messagingService = messagingService;
+ }
+
+ CompletableFuture<Void> send(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ if (rateLimiter.tryAcquire()) {
+ return send0(targetNodeConsistentId, transferId, paths)
+ .whenComplete((v, e) -> {
+ rateLimiter.release();
+ processQueue();
+ });
+ } else {
+ // If the rate limiter is full, we should place the request in a queue and wait for the rate limiter to be released.
+ return submitRequestToQueue(targetNodeConsistentId, transferId, paths);
+ }
+ }
+
+ private CompletableFuture<Void> send0(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ // It doesn't make sense to continue file transfer if there is a failure in one of the files.
+ AtomicBoolean shouldBeCancelled = new AtomicBoolean(false);
+
+ CompletableFuture<?>[] futures = paths.stream()
+ .map(file -> sendFile(targetNodeConsistentId, transferId, file, shouldBeCancelled)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ shouldBeCancelled.set(true);
+ }
+ }))
+ .toArray(CompletableFuture[]::new);
+
+ return CompletableFuture.allOf(futures);
+ }
+
+ /**
+ * Sends the file to the node with the given consistent id.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the file to.
+ * @param id The id of the file transfer.
+ * @param path The path of the file to send.
+ * @return A future that will be completed when the file is sent.
+ */
+ private CompletableFuture<Void> sendFile(String receiverConsistentId, UUID id, Path path, AtomicBoolean shouldBeCancelled) {
+ if (path.toFile().length() == 0) {
+ return completedFuture(null);
+ } else {
+ return supplyAsync(() -> {
+ try {
+ return FileChunkMessagesStream.fromPath(chunkSize, id, path);
+ } catch (IOException e) {
+ throw new FileTransferException("Failed to create a file transfer stream", e);
+ }
+ }).thenCompose(stream -> {
+ return sendMessagesFromStream(receiverConsistentId, stream, shouldBeCancelled).thenApply(ignored -> stream);
+ }).whenComplete((stream, e) -> {
+ try {
+ stream.close();
+ } catch (IOException ex) {
+ throw new FileTransferException("Failed to close the file transfer stream", ex);
+ }
+ }).thenApply(stream -> null);
+ }
+ }
+
+ /**
+ * Sends the next message in the stream. If there are no more messages, the future will be completed.
+ *
+ * @param receiverConsistentId The consistent id of the node to send the files to.
+ * @param stream The stream of messages to send.
+ * @return A future that will be completed when the next message is sent.
+ */
+ private CompletableFuture<Void> sendMessagesFromStream(
+ String receiverConsistentId,
+ FileChunkMessagesStream stream,
+ AtomicBoolean shouldBeCancelled
+ ) {
+ try {
+ if (stream.hasNextMessage() && !shouldBeCancelled.get()) {
+ return messagingService.send(receiverConsistentId, FILE_TRANSFER_CHANNEL, stream.nextMessage())
+ .thenCompose(ignored -> sendMessagesFromStream(receiverConsistentId, stream, shouldBeCancelled));
Review Comment:
What thread will execute this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296111151
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -390,75 +414,64 @@ public CompletableFuture<List<Path>> download(String sourceNodeConsistentId, Ide
.identifier(identifier)
.build();
- try {
- Path directory = createTransferDirectory(transferId);
-
- CompletableFuture<List<Path>> downloadedFiles = fileReceiver.registerTransfer(
- sourceNodeConsistentId,
- transferId,
- directory
- )
- .thenApply(ignored -> {
- try {
- IgniteUtils.deleteIfExists(targetDir);
-
- Files.move(directory, targetDir, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
-
- try (Stream<Path> stream = Files.list(targetDir)) {
- return stream.collect(Collectors.toList());
- }
- } catch (IOException e) {
- throw new FileTransferException("Failed to move downloaded files to target directory", e);
- }
- })
- .whenComplete((v, e) -> {
- if (e != null) {
- LOG.error("Failed to download files. Identifier: {}", identifier, e);
- }
-
- IgniteUtils.deleteIfExists(directory);
- });
-
- return messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
- .thenApply(FileDownloadResponse.class::cast)
- .whenCompleteAsync((response, e) -> {
- if (e != null) {
- fileReceiver.cancelTransfer(transferId, e);
- } else if (response.error() != null) {
- fileReceiver.cancelTransfer(transferId, buildException(response.error()));
- } else {
- fileReceiver.receiveFileHeaders(transferId, response.headers());
- }
- }, executorService)
- .thenCompose(v -> downloadedFiles);
- } catch (Exception e) {
- return failedFuture(e);
- }
+ CompletableFuture<List<Path>> downloadedFiles = new CompletableFuture<List<Path>>()
+ .whenComplete((v, e) -> transferIdToConsumer.remove(transferId));
+
+ transferIdToConsumer.put(transferId, new DownloadRequestConsumer(downloadedFiles, targetDir));
+
+ messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
+ .thenApply(FileDownloadResponse.class::cast)
+ .whenComplete((response, e) -> {
+ if (e != null) {
+ downloadedFiles.completeExceptionally(e);
+ } else if (response.error() != null) {
+ downloadedFiles.completeExceptionally(toException(response.error()));
+ }
+ });
+
+ return downloadedFiles;
}
@Override
public CompletableFuture<Void> upload(String targetNodeConsistentId, Identifier identifier) {
return getFileProvider(identifier).files(identifier)
- .thenCompose(files -> {
- FileUploadRequest message = messageFactory.fileUploadRequest()
- .identifier(identifier)
- .headers(fromPaths(messageFactory, files))
- .build();
+ .thenCompose(files -> transferFilesToNode(targetNodeConsistentId, UUID.randomUUID(), identifier, files));
+ }
+
+ private CompletableFuture<Void> transferFilesToNode(
+ String targetNodeConsistentId,
+ UUID transferId,
+ Identifier identifier,
+ List<Path> paths
+ ) {
+ if (paths.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
Review Comment:
I find it puzzling when someone uploads or downloads an empty list of files. What could this mean? That the files weren't found, or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1298177556
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -17,99 +17,87 @@
package org.apache.ignite.internal.network.file;
+import java.io.BufferedOutputStream;
+import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.file.Path;
-import java.util.PriorityQueue;
-import java.util.Queue;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
import org.apache.ignite.internal.network.file.exception.FileValidationException;
import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
/**
- * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
- * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
- * chunk is not found, the file is not written to. The writer is not thread-safe.
+ * Chunked file writer. Writes chunks to a file. Checks that the file size and chunk numbers are valid.
*/
class ChunkedFileWriter implements AutoCloseable {
- private static final int UNKNOWN_FILE_SIZE = -1;
+ private final BufferedOutputStream stream;
- private final RandomAccessFile raf;
+ private final long expectedFileLength;
- private long fileSize;
+ private long bytesWritten = 0;
- private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
-
- private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
- if (fileSize < UNKNOWN_FILE_SIZE) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
-
- this.raf = raf;
- this.fileSize = fileSize;
- }
+ private int expectedNextChunkNumber = 0;
/**
- * Opens a file with unknown size for writing.
- *
- * @param path File path.
- * @return Chunked file writer.
- * @throws FileNotFoundException If the file is not found.
+ * Lock to synchronize access to the file. We don't write to the file simultaneously, but the next chunk number and bytes written can be
+ * updated from different threads.
*/
- static ChunkedFileWriter open(Path path) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), UNKNOWN_FILE_SIZE);
+ private final Lock lock = new ReentrantLock();
+
+ private ChunkedFileWriter(BufferedOutputStream stream, long expectedFileLength) {
+ this.stream = stream;
+ this.expectedFileLength = expectedFileLength;
}
/**
* Opens a file with known size for writing.
*
- * @param path File path.
- * @param fileSize File size. If the file size is unknown, pass {@link #UNKNOWN_FILE_SIZE}.
+ * @param file File path.
+ * @param expectedFileLength Expected file size.
* @return Chunked file writer.
* @throws FileNotFoundException If the file is not found.
*/
- static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ static ChunkedFileWriter open(File file, long expectedFileLength) throws IOException {
+ return new ChunkedFileWriter(new BufferedOutputStream(new FileOutputStream(file)), expectedFileLength);
}
/**
- * Writes a chunk to the file.
+ * Writes a chunk to the file. The chunk number must be equal to the next expected chunk number. Closes the file if the last chunk is
+ * written. Returns true if the last chunk is written.
*
- * @param chunk Chunk.
+ * @param chunk Chunk to write.
+ * @return True if the last chunk is written. False otherwise.
* @throws IOException If an I/O error occurs.
+ * @throws FileValidationException If the chunk number or file size is invalid or expected file size is exceeded.
*/
- void write(FileChunkMessage chunk) throws IOException {
- chunks.add(chunk);
- while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
- raf.write(chunks.poll().data());
+ boolean write(FileChunkMessage chunk) throws IOException {
+ if (chunk.number() != expectedNextChunkNumber) {
+ throw new FileValidationException("Chunk number mismatch: expected " + expectedNextChunkNumber + ", actual " + chunk.number());
}
- if (fileSize != UNKNOWN_FILE_SIZE && raf.getFilePointer() > fileSize) {
- throw new FileValidationException("File size exceeded: expected " + fileSize + ", actual " + raf.getFilePointer());
- }
- }
+ lock.lock();
+ try {
+ stream.write(chunk.data());
Review Comment:
Ok, my bad, I mistaken this method for another, I apologize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1290202551
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/messages/Identifier.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import org.apache.ignite.internal.network.file.FileConsumer;
+import org.apache.ignite.internal.network.file.FileProvider;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Files identifier. This interface is used to mark all identifier messages. Identifier messages are used to retrieve files from
+ * {@link FileProvider} and handle them on the receiving side by {@link FileConsumer}.
+ */
+@Transferable(FileTransferMessageType.FILE_IDENTIFIER)
+public interface Identifier extends NetworkMessage {
Review Comment:
It is used in the integration tests since we don't have other identifiers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286703455
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
Review Comment:
We reach it by locking in `org.apache.ignite.internal.network.file.FileTransferMessagesHandler`. See `fileNameToLock`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286172678
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
Review Comment:
I have gotten the opposite comments in the previous PRs, so I would prefer to keep it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296077733
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -112,7 +113,12 @@ public class FileTransferServiceImpl implements FileTransferService {
/**
* Map of file handlers.
*/
- private final Map<Short, FileConsumer<Identifier>> metadataToHandler = new ConcurrentHashMap<>();
+ private final Map<Short, FileConsumer<Identifier>> metadataToConsumer = new ConcurrentHashMap<>();
+
+ /**
+ * Map of download requests consumers.
+ */
+ private final Map<UUID, DownloadRequestConsumer> transferIdToConsumer = new ConcurrentHashMap<>();
Review Comment:
There is another handler: org.apache.ignite.internal.network.file.FileTransferMessagesHandler. I want to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sk0x50 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "sk0x50 (via GitHub)" <gi...@apache.org>.
sk0x50 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1283200601
##########
modules/core/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -400,4 +400,15 @@ public static class Compute {
/** Class loader error. */
public static final int CLASS_LOADER_ERR = COMPUTE_ERR_GROUP.registerErrorCode((short) 2);
}
+
+ /**
+ * File transfer error group.
+ */
+ public static class FileTransfer {
Review Comment:
Do we really need a dedicated error group for this? Perhaps, it should be a part of the network error group. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281721882
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}. Exception: {}", recipientConsistentId, e);
+ // todo send error response
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfo info) {
+ FileReceiver receiver = transferIdToReceiver.get(info.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file transfer info: {}", info);
+ } else {
+ receiver.receive(info);
+ }
+ }
+
+ private void processFileHeader(FileHeader header) {
+ FileReceiver receiver = transferIdToReceiver.get(header.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file header: {}", header);
+ } else {
+ receiver.receive(header);
+ }
+ }
+
+ private void processChunkedFile(ChunkedFile chunkedFile) {
+ FileReceiver receiver = transferIdToReceiver.get(chunkedFile.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for chunked file: {}", chunkedFile);
+ } else {
+ receiver.receive(chunkedFile);
+ }
+ }
+
+ private CompletableFuture<FileReceiver> createFileReceiver(UUID transferId) {
+ try {
+ Path directory = Files.createDirectory(tempDirectory.resolve(transferId.toString()));
+ FileReceiver receiver = new FileReceiver(directory);
+ transferIdToReceiver.put(transferId, receiver);
+ return completedFuture(receiver);
+ } catch (IOException e) {
+ return failedFuture(e);
+ }
+ }
+
+ private CompletableFuture<Path> handleFileTransferring(UUID transferId, FileReceiver fileReceiver) {
+ return fileReceiver.result()
+ .whenComplete((v, e) -> {
+ transferIdToReceiver.remove(transferId);
+ try {
+ fileReceiver.close();
+ } catch (Exception ex) {
+ LOG.error("Failed to close file receiver: {}. Exception: {}", fileReceiver, ex);
+ }
+ if (e != null) {
+ LOG.error("Failed to receive file. Id: {}. Exception: {}", transferId, e);
+ deleteDirectoryIfExists(fileReceiver.dir());
+ }
+ });
+ }
+
+ private static void deleteDirectoryIfExists(Path directory) {
+ try {
+ FilesUtils.deleteDirectoryIfExists(directory);
+ } catch (IOException e) {
+ LOG.error("Failed to delete directory: {}. Exception: {}", directory, e);
+ }
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(Stream.concat(
+ transferIdToReceiver.values().stream(),
+ Stream.of(fileSender)
+ ));
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileProvider<TransferMetadata>) provider
+ );
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileHandler(
+ Class<M> metadata,
+ FileHandler<M> handler
+ ) {
+ metadataToHandler.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileHandler<TransferMetadata>) handler
+ );
+ }
+
+ @Override
+ public CompletableFuture<Path> download(String node, TransferMetadata transferMetadata) {
+ UUID transferId = UUID.randomUUID();
+ FileDownloadRequest message = FileDownloadRequestImpl.builder()
+ .transferId(transferId)
+ .metadata(transferMetadata)
+ .build();
+
+ return createFileReceiver(message.transferId())
+ .thenCompose(fileReceiver -> messagingService.invoke(node, FILE_TRANSFERRING_CHANNEL, message, Long.MAX_VALUE)
Review Comment:
To be honest, I don't have any idea how much it should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286694392
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ error.compareAndSet(null, e);
+ }
+ } finally {
+ rateLimiter.release();
+ }
+ });
+ }
+ }
+
+ if (error.get() != null) {
Review Comment:
Yep, we are waiting for the result there `org.apache.ignite.internal.network.file.FileTransferServiceImpl#sendFiles`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286216855
##########
modules/network-api/src/main/java/org/apache/ignite/network/MessagingService.java:
##########
@@ -76,6 +76,26 @@ default CompletableFuture<Void> send(ClusterNode recipient, NetworkMessage msg)
*/
CompletableFuture<Void> send(ClusterNode recipient, ChannelType channelType, NetworkMessage msg);
+ /**
+ * Tries to send the given message via specified channel asynchronously to the specific cluster member.
+ *
+ * <p>Guarantees:
Review Comment:
I have created a ticket to update the doc
https://issues.apache.org/jira/browse/IGNITE-20175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286696683
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
Review Comment:
No, it's called from org.apache.ignite.internal.network.file.FileReceiver#executorService
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288155339
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
Review Comment:
It does. `ChunkedFileWriter` is not thread-safe and synchronization is external. I have updated the documentation of the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288259549
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
Review Comment:
We have it. It's the `DeploymentUnits` channel. It's going to be replaced by this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288235393
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
Review Comment:
See org.apache.ignite.internal.network.file.FileTransferServiceImpl#start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288485834
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
`org.apache.ignite.internal.util.IgniteUtils#deleteIfExists` works as expected. It doesn't throw anything if the file doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281698343
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final long fileSize;
+ private final RandomAccessFile raf;
+ private final Queue<ChunkedFile> chunks = new PriorityQueue<>(ChunkedFile.COMPARATOR);
+
+ ChunkedFileWriter(Path path, long fileSize) throws FileNotFoundException {
+ this.raf = new RandomAccessFile(path.toFile(), "rw");
Review Comment:
There is no "w" mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] rpuch commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "rpuch (via GitHub)" <gi...@apache.org>.
rpuch commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286746929
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
Review Comment:
So there is synchronization that is external to the class which method uses a non-thread-safe data structure. It should probably be mentioned in a comment as otherwise it's easy to miss (and break in the future)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] rpuch commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "rpuch (via GitHub)" <gi...@apache.org>.
rpuch commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286751295
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/exception/FileHandlerNotFoundException.java:
##########
@@ -15,30 +15,22 @@
* limitations under the License.
*/
-package org.apache.ignite.internal.network.file;
+package org.apache.ignite.internal.network.file.exception;
-import java.io.File;
-import java.util.Comparator;
-import java.util.List;
-import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.lang.ErrorGroups.Network;
+import org.apache.ignite.lang.IgniteException;
/**
- * File utils.
+ * Exception thrown when file handler is not found for a metadata message.
*/
-public class FileUtils {
- public static List<File> sortByNames(File... files) {
- return sortByNames(List.of(files));
- }
-
+public class FileHandlerNotFoundException extends IgniteException {
Review Comment:
Isn't it `FileConsumer` now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286837941
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
No reason. Just I hadn't known about those methods.
What do you mean by fixing `deleteIfExistsThrowable `? I don't see anything to fix there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] PakhomovAlexander commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286009821
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferringConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration
+ private FileTransferringConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node node0 = cluster.members.get(0);
Review Comment:
Could you name `node0` like `sourceNode` and `node1` -> `targetNode`.
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/FileContentMatcher.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import static org.hamcrest.CoreMatchers.is;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+
+/**
+ * {@link TypeSafeMatcher} for matching file content.
+ */
+public class FileContentMatcher extends TypeSafeMatcher<File> {
+
+ /** Matcher to forward the result of the completable future. */
+ private final Matcher<byte[]> matcher;
+
+ private FileContentMatcher(Matcher<byte[]> matcher) {
+ this.matcher = matcher;
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param file File to match.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(File file) {
+ try {
+ return new FileContentMatcher(is(Files.readAllBytes(file.toPath())));
Review Comment:
You can use the static method here instead of the constructor. `return hasContent(is(Files....))`
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferringConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration
+ private FileTransferringConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node node0 = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = node0.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ node0.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node node1 = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFilesFuture = node1.fileTransferringService().download(
+ node0.nodeName(),
+ MetadataImpl.builder().build()
+ );
+
+ assertThat(
+ downloadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(Set.of(file1, file2, file3, file4), new HashSet<>(files));
Review Comment:
Why do you use sets in the assertion? This might hide duplicates.
##########
modules/file-transfer/build.gradle:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply from: "$rootDir/buildscripts/java-core.gradle"
+apply from: "$rootDir/buildscripts/publishing.gradle"
+apply from: "$rootDir/buildscripts/java-junit5.gradle"
+apply from: "$rootDir/buildscripts/java-integration-test.gradle"
+apply from: "$rootDir/buildscripts/java-test-fixtures.gradle"
+
+dependencies {
Review Comment:
This is a huge module and the README would be usefull.
##########
modules/sql-engine/src/test/java/org/apache/ignite/internal/sql/engine/framework/ClusterServiceFactory.java:
##########
@@ -184,6 +184,11 @@ public CompletableFuture<Void> send(ClusterNode recipient, ChannelType channelTy
return CompletableFuture.completedFuture(null);
}
+ @Override
+ public CompletableFuture<Void> send(String recipientConsistentId, ChannelType channelType, NetworkMessage msg) {
+ throw new AssertionError("Not implemented yet");
Review Comment:
Do we have a ticket for the implementation?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/RateLimiter.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+class RateLimiter {
+ private final int maxConcurrentRequests;
+ private final AtomicInteger counter = new AtomicInteger(0);
+
+ RateLimiter(int maxConcurrentRequests) {
+ this.maxConcurrentRequests = maxConcurrentRequests;
+ }
+
+ boolean tryAcquire() {
+ int count = counter.incrementAndGet();
+ if (count <= maxConcurrentRequests) {
Review Comment:
I think using explicit lock is abetter choice here.
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferringConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Members of the cluster. */
+ final List<Node> members;
+
+ /** Latch that is locked until all members are visible in the topology. */
+ private final CountDownLatch startupLatch;
+
+ /** Node finder. */
+ private final NodeFinder nodeFinder;
+
+ private final FileTransferringConfiguration configuration;
+
+ /**
+ * Creates a test cluster with the given amount of members.
+ *
+ * @param numOfNodes Amount of cluster members.
+ * @param testInfo Test info.
+ */
+ TestCluster(int numOfNodes, FileTransferringConfiguration configuration, Path workDir, TestInfo testInfo) {
+ this.startupLatch = new CountDownLatch(numOfNodes - 1);
+
+ int initialPort = 3344;
+
+ List<NetworkAddress> addresses = findLocalAddresses(initialPort, initialPort + numOfNodes);
+
+ this.nodeFinder = new StaticNodeFinder(addresses);
+ this.configuration = configuration;
+
+ var isInitial = new AtomicBoolean(true);
+
+ this.members = addresses.stream()
+ .map(addr -> startNode(workDir, testInfo, addr, isInitial.getAndSet(false)))
+ .collect(toUnmodifiableList());
+ }
+
+ /**
+ * Start cluster node.
+ *
+ * @param testInfo Test info.
+ * @param addr Node address.
+ * @param initial Whether this node is the first one.
+ * @return Started cluster node.
+ */
+ private Node startNode(
+ Path workDir, TestInfo testInfo, NetworkAddress addr, boolean initial
+ ) {
+ ClusterService clusterSvc = clusterService(testInfo, addr.port(), nodeFinder);
+
+ if (initial) {
+ clusterSvc.topologyService().addEventHandler(new TopologyEventHandler() {
+ /** {@inheritDoc} */
+ @Override
+ public void onAppeared(ClusterNode member) {
+ startupLatch.countDown();
+ }
+ });
+ }
+
+ try {
+ Path nodeDir = Files.createDirectory(workDir.resolve("node-" + clusterSvc.nodeName()));
+ FileTransferServiceImpl fileTransferringService = new FileTransferServiceImpl(
+ clusterSvc.nodeName(),
+ clusterSvc.messagingService(),
+ configuration,
+ nodeDir
+ );
+ return new Node(nodeDir, clusterSvc, fileTransferringService);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Starts and waits for the cluster to come up.
+ *
+ * @throws InterruptedException If failed.
+ * @throws AssertionError If the cluster was unable to start in 3 seconds.
+ */
+ void startAwait() throws InterruptedException {
+ members.forEach(Node::start);
+
+ if (!startupLatch.await(3, TimeUnit.SECONDS)) {
Review Comment:
Why is `3` hardcoded?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
+ private final String fileName;
+ private final RandomAccessFile raf;
+ private final int chunkSize;
+
+ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize) {
+ this.fileName = fileName;
+ this.raf = raf;
+ this.chunkSize = chunkSize;
+ }
+
+ static ChunkedFileReader open(File file, int chunkSize) throws IOException {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ return new ChunkedFileReader(file.getName(), raf, chunkSize);
+ }
+
+ byte[] readNextChunk() throws IOException {
+ int toRead = (int) Math.min(chunkSize, raf.length() - raf.getFilePointer());
Review Comment:
I think you can use methods `length()` and `offset()`.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
+ */
+public interface FileHandler<M extends Metadata> {
+ /**
+ * Handles the uploaded files.
+ *
+ * @param uploadedFiles The temporary files that were uploaded. These files will be deleted after the method returns.
Review Comment:
Metadata parameter description is missed.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+
+/**
+ * File receiver.
+ */
+class FileReceiver {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ FileReceiver(ExecutorService executorService) {
+ this.executorService = executorService;
+ }
+
+ FileTransferMessagesHandler registerTransfer(UUID transferId, Path handlerDir) {
+ FileTransferMessagesHandler handler = new FileTransferMessagesHandler(handlerDir);
+ transferIdToHandler.put(transferId, handler);
+ handler.result().whenComplete((files, throwable) -> transferIdToHandler.remove(transferId));
Review Comment:
Why not use the `cancelTransfer` method?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ int chunkSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send,
+ ExecutorService executorService) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = executorService;
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicBoolean interrupted = new AtomicBoolean(false);
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && !interrupted.get()) {
+ if (rateLimiter.tryAcquire()) {
+ send.apply(receiverConsistentId, stream.nextMessage())
+ .whenComplete((res, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ interrupted.set(true);
+ }
+ rateLimiter.release();
Review Comment:
what if `send.apply` throws an exception? I think we have to use `try-final`
##########
modules/core/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -400,4 +400,15 @@ public static class Compute {
/** Class loader error. */
public static final int CLASS_LOADER_ERR = COMPUTE_ERR_GROUP.registerErrorCode((short) 2);
}
+
+ /**
+ * File transfer error group.
+ */
+ public static class FileTransfer {
Review Comment:
Agreed, move the error to the Nework group, please.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
Review Comment:
we tend to use empty lines when defining class fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286694392
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ error.compareAndSet(null, e);
+ }
+ } finally {
+ rateLimiter.release();
+ }
+ });
+ }
+ }
+
+ if (error.get() != null) {
Review Comment:
Yep, we are waiting for the result there org.apache.ignite.internal.network.file.FileTransferServiceImpl#sendFiles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288261407
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
Review Comment:
`org.apache.ignite.internal.network.file.FileSender` and `org.apache.ignite.internal.network.file.FileReceiver` have their own executors where they do all IO operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1289793714
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import static java.util.stream.Collectors.toList;
+
+import java.io.File;
+import java.util.Comparator;
+import java.util.List;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
Is this class only used in tests?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/Channel.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import org.apache.ignite.network.ChannelType;
+
+/**
+ * Channel types used by the file transfer protocol.
+ */
+final class Channel {
+ /**
+ * File transfer channel.
+ */
+ static final ChannelType FILE_TRANSFER_CHANNEL = ChannelType.register((short) 1, "FileTransfer");
Review Comment:
It's still not changed, you simply closed my previous comment. How are we going to have two channels with the same ID in main branch?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+import static org.apache.ignite.internal.network.file.MessagesUtils.getHeaders;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileTransferError;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Identifier;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ /**
+ * Response timeout.
+ */
+ private final int reponseTimeout;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Identifier>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Identifier>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory messageFactory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param nodeName Node name.
+ * @param topologyService Topology service.
+ * @param messagingService Messaging service.
+ * @param configuration File transfer configuration.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ configuration.value().responseTimeout(),
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().chunkSize(),
+ configuration.value().senderThreadPoolSize(),
+ new RateLimiterImpl(configuration.value().maxConcurrentRequests()),
+ messagingService
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param responseTimeout Response timeout.
+ * @param topologyService Topology service.
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ * @param fileSender File sender.
+ * @param fileReceiver File receiver.
+ */
+ FileTransferServiceImpl(
+ int responseTimeout,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.reponseTimeout = responseTimeout;
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
Review Comment:
Shouldn't you use the `member.name()` here?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/RateLimiter.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+/**
+ * Rate limiter interface.
+ */
+public interface RateLimiter {
Review Comment:
I don't think that this interface is necessary at all
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/messages/FileHeader.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import java.io.File;
+import java.nio.file.Path;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * File header.
+ */
+@Transferable(FileTransferMessageType.FILE_HEADER)
+public interface FileHeader extends NetworkMessage {
+ /**
+ * Returns the name of the file.
+ *
+ * @return File name.
+ */
+ String name();
+
+ /**
+ * Returns the size of the file in bytes.
+ *
+ * @return File size.
+ */
+ long length();
+
+ /**
+ * Creates a new {@link FileHeader} instance from the given {@link File}.
+ */
+ static FileHeader fromPath(FileTransferFactory factory, int id, Path path) {
Review Comment:
Wait a minute, you don't even use this "id". Please remove it then
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+import static org.apache.ignite.internal.network.file.MessagesUtils.getHeaders;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileTransferError;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Identifier;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ /**
+ * Response timeout.
+ */
+ private final int reponseTimeout;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Identifier>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Identifier>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory messageFactory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param nodeName Node name.
+ * @param topologyService Topology service.
+ * @param messagingService Messaging service.
+ * @param configuration File transfer configuration.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ configuration.value().responseTimeout(),
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().chunkSize(),
+ configuration.value().senderThreadPoolSize(),
+ new RateLimiterImpl(configuration.value().maxConcurrentRequests()),
+ messagingService
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param responseTimeout Response timeout.
+ * @param topologyService Topology service.
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ * @param fileSender File sender.
+ * @param fileReceiver File receiver.
+ */
+ FileTransferServiceImpl(
+ int responseTimeout,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.reponseTimeout = responseTimeout;
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ fileReceiver.receiveFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ fileReceiver.receiveFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ UUID transferId = UUID.randomUUID();
+
+ Path directory = createTransferDirectory(transferId);
+
+ CompletableFuture<List<Path>> uploadedFiles = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+
+ fileReceiver.receiveFileHeaders(transferId, message.headers());
+
+ FileUploadResponse response = messageFactory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+
+ Identifier identifier = message.identifier();
+
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return uploadedFiles.thenCompose(files -> getFileConsumer(identifier).consume(identifier, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}",
+ e,
+ transferId,
+ identifier
+ );
+ }
+
+ IgniteUtils.deleteIfExists(directory);
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ getFileProvider(message.identifier()).files(message.identifier())
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.identifier(), e);
+
+ FileDownloadResponse response = messageFactory.fileDownloadResponse()
+ .error(buildError(e))
+ .build();
+
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.identifier());
+
+ FileDownloadResponse response = messageFactory.fileDownloadResponse()
+ .error(buildError(new FileTransferException("No files to download")))
+ .build();
+
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = messageFactory.fileDownloadResponse()
+ .headers(getHeaders(messageFactory, files))
+ .build();
+
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String targetNodeConsistentId, UUID transferId, List<Path> paths) {
+ return fileSender.send(targetNodeConsistentId, transferId, paths)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}",
+ e,
+ targetNodeConsistentId,
+ transferId
+ );
+
+ FileTransferErrorMessage message = messageFactory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(buildError(e))
+ .build();
+
+ messagingService.send(targetNodeConsistentId, FILE_TRANSFER_CHANNEL, message);
+ }
+ });
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Identifier> void addFileProvider(
+ Class<M> identifier,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(identifier),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + identifier.getName() + " already exists");
+ } else {
+ return (FileProvider<Identifier>) provider;
+ }
+ }
+ );
+ }
+
+ @Override
+ public <M extends Identifier> void addFileConsumer(
+ Class<M> identifier,
+ FileConsumer<M> consumer
+ ) {
+ metadataToHandler.compute(
+ getMessageType(identifier),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + identifier.getName() + " already exists");
+ } else {
+ return (FileConsumer<Identifier>) consumer;
+ }
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<Path>> download(String sourceNodeConsistentId, Identifier identifier, Path targetDir) {
+ UUID transferId = UUID.randomUUID();
+ FileDownloadRequest downloadRequest = messageFactory.fileDownloadRequest()
+ .transferId(transferId)
+ .identifier(identifier)
+ .build();
+
+ try {
+ Path directory = createTransferDirectory(transferId);
+
+ // Wrap in the completed future to avoid blocking.
Review Comment:
Can you please elaborate?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.network.file.exception.FileValidationException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+/**
+ * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
+ * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
+ * chunk is not found, the file is not written to. The writer is not thread-safe
Review Comment:
```suggestion
* chunk is not found, the file is not written to. The writer is not thread-safe.
```
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/PathMatcher.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import static org.hamcrest.Matchers.both;
+import static org.hamcrest.Matchers.equalTo;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.hamcrest.FeatureMatcher;
+import org.hamcrest.Matcher;
+
+/**
+ * Matchers for {@link Path}.
+ */
+public final class PathMatcher {
+
+ private PathMatcher() {
+ }
+
+ /**
+ * Creates a matcher for matching file content and name.
+ *
+ * @param expectedFile Expected file.
+ * @return Matcher that checks if file has the same content and name as expected file.
+ */
+ public static Matcher<Path> hasSameContentAndName(Path expectedFile) {
+ return both(hasSameContent(expectedFile)).and(hasSameName(expectedFile));
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param expectedFile Expected file.
+ * @return Matcher for matching file content.
+ */
+ public static Matcher<Path> hasSameContent(Path expectedFile) {
+ return hasContent(equalTo(readFile(expectedFile)));
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param contentMatcher Matcher for matching file content.
+ * @return Matcher for matching file content.
+ */
+ public static Matcher<Path> hasContent(Matcher<byte[]> contentMatcher) {
+ return new FeatureMatcher<>(contentMatcher, "A file with content", "content") {
+ @Override
+ protected byte[] featureValueOf(Path actual) {
+ return readFile(actual);
+ }
+ };
+ }
+
+ /**
+ * Creates a matcher for matching file name.
+ *
+ * @param expectedFile Expected file.
+ * @return Matcher for matching file name.
+ */
+ public static Matcher<Path> hasSameName(Path expectedFile) {
+ return hasName(equalTo(expectedFile.getFileName().toString()));
+ }
+
+ /**
+ * Creates a matcher for matching file name.
+ *
+ * @param nameMatcher Matcher for matching file name.
+ * @return Matcher for matching file name.
+ */
+ public static Matcher<Path> hasName(Matcher<String> nameMatcher) {
+ return new FeatureMatcher<>(nameMatcher, "A file with name", "name") {
+ @Override
+ protected String featureValueOf(Path actual) {
+ return actual.getFileName().toString();
+ }
+ };
+ }
+
+ private static byte[] readFile(Path path) {
+ try {
+ return Files.readAllBytes(path);
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+
+}
Review Comment:
```suggestion
}
}
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesStream.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Path;
+import java.util.Iterator;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * A stream of messages to send file.
+ */
+public class FileTransferMessagesStream implements Iterable<FileChunkMessage>, AutoCloseable {
+ private final UUID transferId;
+
+ private final ChunkedFileReader reader;
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ private final FileTransferFactory messageFactory = new FileTransferFactory();
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param reader the reader of the file to send.
+ */
+ private FileTransferMessagesStream(
+ UUID transferId,
+ ChunkedFileReader reader
+ ) {
+ this.transferId = transferId;
+ this.reader = reader;
+ }
+
+ /**
+ * Creates a new stream of messages to send file.
+ *
+ * @param chunkSize the size of the chunks to send.
+ * @param transferId the id of the transfer.
+ * @param path the path of the file to send.
+ * @return a new stream of messages to send files.
+ * @throws IOException if an I/O error occurs.
+ */
+ public static FileTransferMessagesStream fromPath(
+ int chunkSize,
+ UUID transferId,
+ Path path
+ ) throws IOException {
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive");
+ }
+
+ return new FileTransferMessagesStream(transferId, ChunkedFileReader.open(path.toFile(), chunkSize));
+ }
+
+ /**
+ * Returns true if there are more messages to send.
+ *
+ * @return true if there are more messages to send.
+ */
+ boolean hasNextMessage() throws IOException {
+ // Check that the stream is not closed and the reader has more chunks.
+ return !closed.get() && reader.hasNextChunk();
+ }
+
+ /**
+ * Returns the next message to send.
+ *
+ * @return the next message to send.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if there are no more messages to send.
+ */
+ FileChunkMessage nextMessage() throws IOException {
+ if (hasNextMessage()) {
+ return nextChunk();
+ } else {
+ throw new IllegalStateException("There are no more messages to send");
+ }
+ }
+
+ /**
+ * Returns the next chunk of the current file. Throws an exception if the current file is finished.
+ *
+ * @return the next chunk of the current file.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if the current file is finished.
+ */
+ private FileChunkMessage nextChunk() throws IOException {
+ return messageFactory.fileChunkMessage()
+ .transferId(transferId)
+ .fileName(reader.fileName())
+ .offset(reader.offset())
+ .data(reader.readNextChunk())
+ .build();
+ }
+
+ /**
+ * Closes the stream.
+ */
+ @Override
+ public void close() throws IOException {
+ if (closed.compareAndSet(false, true)) {
+ reader.close();
+ }
+ }
+
+ @NotNull
Review Comment:
We don't use NotNull annotation, please remove it
##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -327,6 +327,12 @@ public static class Network {
/** Port is in use. */
public static final int PORT_IN_USE_ERR = NETWORK_ERR_GROUP.registerErrorCode((short) 2);
+
+ /** File transfer error. */
+ public static final int FILE_TRANSFER_ERR = NETWORK_ERR_GROUP.registerErrorCode((short) 3);
+
+ /** File validation error. */
+ public static final int FILE_VALIDATION_ERR = NETWORK_ERR_GROUP.registerErrorCode((short) 4);
Review Comment:
Exceptions are still in internal packages. What are the circumstances, in which a user would get the exception with one of these error codes? I don't see any, so please move these error codes into a proper place.
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,485 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.PathAssertions.namesAndContentEquals;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.emptyIterable;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.Identifier;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+ private static final String DOWNLOADS_DIR = "downloads";
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration("mock.chunkSize=1024")
+ private FileTransferConfiguration configuration;
+
+ private TestCluster cluster;
+
+ private Path sourceDir;
+
+ private final FileTransferFactory messageFactory = new FileTransferFactory();
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException, IOException {
+ sourceDir = Files.createDirectories(workDir.resolve("source"));
+
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void downloadToEmptyDir() {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, chunkSize);
+ Path file2 = randomFile(sourceDir, chunkSize + 1);
+ Path file3 = randomFile(sourceDir, chunkSize * 2);
+ Path file4 = randomFile(sourceDir, chunkSize * 2);
+
+ // Add file provider to the source node.
+ List<Path> files = List.of(file1, file2, file3, file4);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(files)
+ );
+
+ // Download files on the target node from the source node.
+ Node targetNode = cluster.members.get(1);
+ CompletableFuture<List<Path>> downloadedFilesFuture = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNode.workDir().resolve(DOWNLOADS_DIR)
+ );
+
+ // Check that files were downloaded.
+ assertThat(
+ downloadedFilesFuture,
+ willBe(namesAndContentEquals(files))
+ );
+
+ // Check that files were downloaded to the correct directory.
+ assertDownloadsDirContainsFiles(targetNode, files);
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+ }
+
+ @Test
+ void downloadToNonEmptyDir() throws IOException {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, chunkSize);
+ Path file2 = randomFile(sourceDir, chunkSize + 1);
+ Path file3 = randomFile(sourceDir, chunkSize * 2);
+ Path file4 = randomFile(sourceDir, chunkSize * 2);
+
+ // Add file provider to the source node.
+ List<Path> files = List.of(file1, file2, file3, file4);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(files)
+ );
+
+ Node targetNode = cluster.members.get(1);
+
+ // Create downloads directory.
+ Path targetNodeDownloadsDir = targetNode.workDir().resolve(DOWNLOADS_DIR);
+ Files.createDirectories(targetNodeDownloadsDir);
+
+ // Create some files in the downloads directory.
+ Path file5 = randomFile(targetNodeDownloadsDir, chunkSize);
+ Path file6 = randomFile(targetNodeDownloadsDir, chunkSize + 1);
+
+ // Download files on the target node from the source node.
+ CompletableFuture<List<Path>> downloadedFilesFuture = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNodeDownloadsDir
+ );
+
+ // Check that files were downloaded.
+ assertThat(
+ downloadedFilesFuture,
+ willBe(namesAndContentEquals(files))
+ );
+
+ // Check that files were downloaded to the correct directory.
+ assertDownloadsDirContainsFiles(targetNode, files);
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+
+ // Check that previous files in the downloads directory were deleted.
+ try (Stream<Path> stream = Files.list(targetNodeDownloadsDir)) {
+ List<String> filesInDownloadsDir = stream.map(Path::getFileName)
+ .map(Path::toString)
+ .collect(Collectors.toList());
+ assertThat(filesInDownloadsDir, not(containsInAnyOrder(file5.getFileName().toString(), file6.getFileName().toString())));
+ }
+ }
+
+ @Test
+ void downloadNonReadableFiles() {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, chunkSize - 1);
+ Path file2 = randomFile(sourceDir, chunkSize + 1);
+ Path file3 = randomFile(sourceDir, chunkSize * 2);
+
+ // Make file2 non-readable.
+ assertTrue(file2.toFile().setReadable(false));
+
+ // Add file provider to the source node.
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ // Download files on the target node from the source node.
+ Node targetNode = cluster.members.get(1);
+ CompletableFuture<List<Path>> download = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNode.workDir().resolve(DOWNLOADS_DIR)
+ );
+
+ // Check that files transfer failed.
+ assertThat(download, willThrow(FileTransferException.class, "Failed to create a file transfer stream"));
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+ }
+
+ @Test
+ void downloadFilesWhenProviderThrowsException() {
+ // Add file provider to the source node that throws an exception.
+ Node sourceNode = cluster.members.get(0);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> failedFuture(new RuntimeException("Test exception"))
+ );
+
+ // Download files on the target node from the source node.
+ Node targetNode = cluster.members.get(1);
+ CompletableFuture<List<Path>> download = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNode.workDir().resolve(DOWNLOADS_DIR)
+ );
+
+ // Check that files transfer failed.
+ assertThat(download, willThrowWithCauseOrSuppressed(FileTransferException.class));
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+ }
+
+ @Test
+ void downloadFilesWhenDoNotHaveAccessToWrite() {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, 0);
+ Path file2 = randomFile(sourceDir, chunkSize + 1);
+ Path file3 = randomFile(sourceDir, chunkSize * 2);
+ Path file4 = randomFile(sourceDir, chunkSize * 2);
+
+ // Add file provider to the source node.
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ // Make work directory not writable on the target node.
+ Node targetNode = cluster.members.get(1);
+ assertTrue(targetNode.workDir().toFile().setWritable(false));
+
+ // Download files on the target node from the source node.
+ CompletableFuture<List<Path>> downloadedFilesFuture = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNode.workDir().resolve(DOWNLOADS_DIR)
+ );
+
+ // Check that files transfer failed.
+ assertThat(
+ downloadedFilesFuture,
+ willThrowWithCauseOrSuppressed(AccessDeniedException.class)
+ );
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+ }
+
+ @Test
+ void downloadEmptyFileList() {
+ // Set empty file list provider on the source node.
+ Node sourceNode = cluster.members.get(0);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(List.of())
+ );
+
+ // Download files on the target node from the source node.
+ Node targetNode = cluster.members.get(1);
+ CompletableFuture<List<Path>> downloadedFiles = targetNode.fileTransferringService().download(
+ sourceNode.nodeName(),
+ messageFactory.identifier().build(),
+ targetNode.workDir().resolve(DOWNLOADS_DIR)
+
+ );
+
+ // Check that files transfer failed.
+ assertThat(downloadedFiles, willThrow(FileTransferException.class, "No files to download"));
+
+ // Check temporary files were deleted.
+ assertTemporaryFilesWereDeleted(targetNode);
+ }
+
+ @Test
+ void upload() {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, chunkSize);
+ Path file2 = randomFile(sourceDir, chunkSize - 1);
+ Path file3 = randomFile(sourceDir, chunkSize + 1);
+ Path file4 = randomFile(sourceDir, chunkSize * 2);
+
+ // Add file provider to the source node.
+ List<Path> files = List.of(file1, file2, file3, file4);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(files)
+ );
+
+ Node targetNode = cluster.members.get(1);
+
+ // The future that will be completed when the consumer is called.
+ CompletableFuture<Void> uploadedFilesFuture = new CompletableFuture<>();
+ targetNode.fileTransferringService().addFileConsumer(Identifier.class, ((metadata, uploadedFiles) -> {
+ return uploadedFilesFuture.completeAsync(() -> {
+ assertThat(uploadedFiles, namesAndContentEquals(files));
+ return null;
+ });
+ }));
+
+ // Upload files on the target node from the source node.
+ CompletableFuture<Void> upload = sourceNode.fileTransferringService()
+ .upload(targetNode.nodeName(), messageFactory.identifier().build());
+
+ // Check that transfer was successful.
+ assertThat(
+ upload,
+ CompletableFutureMatcher.willCompleteSuccessfully()
+ );
+
+ // Check the consumer was called.
+ assertThat(
+ uploadedFilesFuture,
+ CompletableFutureMatcher.willCompleteSuccessfully()
+ );
+
+ // Check temporary files were deleted.
+ await().until(() -> targetNode.workDir().toFile().listFiles().length == 0);
+ }
+
+ @Test
+ void uploadEmptyFileList() {
+ // Set empty file list provider on the source node.
+ Node sourceNode = cluster.members.get(0);
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(List.of())
+ );
+
+ // Upload files on the target node from the source node.
+ Node targetNode = cluster.members.get(1);
+ CompletableFuture<Void> uploaded = sourceNode.fileTransferringService()
+ .upload(targetNode.nodeName(), messageFactory.identifier().build());
+
+ // Check that files transfer failed.
+ assertThat(
+ uploaded,
+ willThrow(FileTransferException.class, "No files to upload")
+ );
+
+ // Check temporary files were deleted.
+ await().until(() -> targetNode.workDir().toFile().listFiles().length == 0);
+ }
+
+ @Test
+ void uploadNonReadableFiles() {
+ // Generate files on the source node.
+ Node sourceNode = cluster.members.get(0);
+
+ int chunkSize = configuration.value().chunkSize();
+ Path file1 = randomFile(sourceDir, chunkSize - 1);
+ Path file2 = randomFile(sourceDir, chunkSize + 1);
+ Path file3 = randomFile(sourceDir, chunkSize * 2);
+
+ // Make file2 non-readable.
+ assertTrue(file2.toFile().setReadable(false));
+
+ // Add file provider to the source node.
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ // Set file consumer on the target node.
+ Node targetNode = cluster.members.get(1);
+
+ // The future will be completed when the file consumer is called.
+ CompletableFuture<List<Path>> uploadedFilesFuture = new CompletableFuture<>();
+ targetNode.fileTransferringService().addFileConsumer(Identifier.class, ((metadata, uploadedFiles) -> {
+ uploadedFilesFuture.complete(uploadedFiles);
+ return completedFuture(null);
+ }));
+
+ // Upload files on the target node from the source node.
+ Identifier identifier = messageFactory.identifier().build();
+ CompletableFuture<Void> upload = sourceNode.fileTransferringService().upload(targetNode.nodeName(), identifier);
+
+ // Check that files transfer failed.
+ assertThat(
+ upload,
+ willThrow(FileTransferException.class, "Failed to create a file transfer stream")
+ );
+
+ // Check the consumer was not called.
+ assertFalse(uploadedFilesFuture.isDone());
+
+ // Check temporary files were deleted.
+ await().until(() -> targetNode.workDir().toFile().listFiles().length == 0);
+ }
+
+ @Test
+ void uploadFilesWhenProviderThrowsException() {
+ // Set file provider on the source node that throws an exception.
+ Node sourceNode = cluster.members.get(0);
+
+ sourceNode.fileTransferringService().addFileProvider(
+ Identifier.class,
+ req -> failedFuture(new RuntimeException("Test exception"))
+ );
+
+ // Set file consumer on the target node.
+ Node targetNode = cluster.members.get(1);
+
+ // The future will be completed when the file consumer is called.
+ CompletableFuture<List<Path>> uploadedFilesFuture = new CompletableFuture<>();
+ targetNode.fileTransferringService().addFileConsumer(Identifier.class, ((metadata, uploadedFiles) -> {
+ uploadedFilesFuture.complete(uploadedFiles);
+ return completedFuture(null);
+ }));
+
+ // Upload files on the target node from the source node.
+ CompletableFuture<Void> uploaded = sourceNode.fileTransferringService()
+ .upload(targetNode.nodeName(), messageFactory.identifier().build());
+
+ // Check that files transfer failed.
+ assertThat(
+ uploaded,
+ willThrow(RuntimeException.class, "Test exception")
+ );
+
+ // Check the consumer was not called.
+ assertFalse(uploadedFilesFuture.isDone());
+
+ // Check temporary files were deleted.
+ await().until(() -> targetNode.workDir().toFile().listFiles().length == 0);
+ }
+
+
Review Comment:
Do we allow several consecutive empty lines? I thought it won't pass the checkstyle, interesting..
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/MessagesUtils.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toList;
+
+import java.nio.file.Path;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+
+/**
+ * Utility class for file transfer messages.
+ */
+final class MessagesUtils {
+ private MessagesUtils() {
+ // No-op.
+ }
+
+ /**
+ * Extracts file headers from files.
+ *
+ * @param factory File transfer factory.
+ * @param paths List of paths.
+ * @return List of file headers.
+ */
+ static List<FileHeader> getHeaders(FileTransferFactory factory, List<Path> paths) {
Review Comment:
Is this reasonable to extract such small method into utilities?
And why do we store the index of an element inside of the element? What's its purpose, it can be derived from the list
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/messages/Identifier.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import org.apache.ignite.internal.network.file.FileConsumer;
+import org.apache.ignite.internal.network.file.FileProvider;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Files identifier. This interface is used to mark all identifier messages. Identifier messages are used to retrieve files from
+ * {@link FileProvider} and handle them on the receiving side by {@link FileConsumer}.
+ */
+@Transferable(FileTransferMessageType.FILE_IDENTIFIER)
+public interface Identifier extends NetworkMessage {
Review Comment:
Marker interface doesn't need to be transferable itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296114167
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -390,75 +414,64 @@ public CompletableFuture<List<Path>> download(String sourceNodeConsistentId, Ide
.identifier(identifier)
.build();
- try {
- Path directory = createTransferDirectory(transferId);
-
- CompletableFuture<List<Path>> downloadedFiles = fileReceiver.registerTransfer(
- sourceNodeConsistentId,
- transferId,
- directory
- )
- .thenApply(ignored -> {
- try {
- IgniteUtils.deleteIfExists(targetDir);
-
- Files.move(directory, targetDir, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
-
- try (Stream<Path> stream = Files.list(targetDir)) {
- return stream.collect(Collectors.toList());
- }
- } catch (IOException e) {
- throw new FileTransferException("Failed to move downloaded files to target directory", e);
- }
- })
- .whenComplete((v, e) -> {
- if (e != null) {
- LOG.error("Failed to download files. Identifier: {}", identifier, e);
- }
-
- IgniteUtils.deleteIfExists(directory);
- });
-
- return messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
- .thenApply(FileDownloadResponse.class::cast)
- .whenCompleteAsync((response, e) -> {
- if (e != null) {
- fileReceiver.cancelTransfer(transferId, e);
- } else if (response.error() != null) {
- fileReceiver.cancelTransfer(transferId, buildException(response.error()));
- } else {
- fileReceiver.receiveFileHeaders(transferId, response.headers());
- }
- }, executorService)
- .thenCompose(v -> downloadedFiles);
- } catch (Exception e) {
- return failedFuture(e);
- }
+ CompletableFuture<List<Path>> downloadedFiles = new CompletableFuture<List<Path>>()
+ .whenComplete((v, e) -> transferIdToConsumer.remove(transferId));
+
+ transferIdToConsumer.put(transferId, new DownloadRequestConsumer(downloadedFiles, targetDir));
+
+ messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
+ .thenApply(FileDownloadResponse.class::cast)
+ .whenComplete((response, e) -> {
+ if (e != null) {
+ downloadedFiles.completeExceptionally(e);
+ } else if (response.error() != null) {
+ downloadedFiles.completeExceptionally(toException(response.error()));
+ }
+ });
+
+ return downloadedFiles;
}
@Override
public CompletableFuture<Void> upload(String targetNodeConsistentId, Identifier identifier) {
return getFileProvider(identifier).files(identifier)
- .thenCompose(files -> {
- FileUploadRequest message = messageFactory.fileUploadRequest()
- .identifier(identifier)
- .headers(fromPaths(messageFactory, files))
- .build();
+ .thenCompose(files -> transferFilesToNode(targetNodeConsistentId, UUID.randomUUID(), identifier, files));
+ }
+
+ private CompletableFuture<Void> transferFilesToNode(
+ String targetNodeConsistentId,
+ UUID transferId,
+ Identifier identifier,
+ List<Path> paths
+ ) {
+ if (paths.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
Review Comment:
I would prefer to keep empty lists restricted to avoid confusion until we really need support for this. Right now we are going to transfer only deployment units. It's impossible to have a deployment unit without any files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1291060082
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
Review Comment:
The number of files. In the current implementation, we can't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1298395817
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -88,18 +81,12 @@ byte[] readNextChunk() throws IOException {
throw new IOException("No more chunks to read");
}
- int toRead = (int) Math.min(chunkSize, length - offset);
- byte[] data = new byte[toRead];
- int read = stream.read(data);
-
- if (read != toRead) {
- throw new IOException("Failed to read chunk data from file: expected " + toRead + ", actual " + read + "]");
- }
+ byte[] data = stream.readNBytes(chunkSize);
- offset += toRead;
nextChunkNumber++;
+ hasNextChunk = stream.available() > 0;
- if (offset == length) {
+ if (stream.available() == 0) {
Review Comment:
I wonder why it's not a `!hasNextChunk`, but whatever.
I've been looking at this code for too long. If there are more problems, I probably won't notice them at this point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296039052
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -17,99 +17,87 @@
package org.apache.ignite.internal.network.file;
+import java.io.BufferedOutputStream;
+import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.file.Path;
-import java.util.PriorityQueue;
-import java.util.Queue;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
import org.apache.ignite.internal.network.file.exception.FileValidationException;
import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
/**
- * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
- * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
- * chunk is not found, the file is not written to. The writer is not thread-safe.
+ * Chunked file writer. Writes chunks to a file. Checks that the file size and chunk numbers are valid.
*/
class ChunkedFileWriter implements AutoCloseable {
- private static final int UNKNOWN_FILE_SIZE = -1;
+ private final BufferedOutputStream stream;
- private final RandomAccessFile raf;
+ private final long expectedFileLength;
- private long fileSize;
+ private long bytesWritten = 0;
- private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
-
- private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
- if (fileSize < UNKNOWN_FILE_SIZE) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
-
- this.raf = raf;
- this.fileSize = fileSize;
- }
+ private int expectedNextChunkNumber = 0;
/**
- * Opens a file with unknown size for writing.
- *
- * @param path File path.
- * @return Chunked file writer.
- * @throws FileNotFoundException If the file is not found.
+ * Lock to synchronize access to the file. We don't write to the file simultaneously, but the next chunk number and bytes written can be
+ * updated from different threads.
*/
- static ChunkedFileWriter open(Path path) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), UNKNOWN_FILE_SIZE);
+ private final Lock lock = new ReentrantLock();
+
+ private ChunkedFileWriter(BufferedOutputStream stream, long expectedFileLength) {
+ this.stream = stream;
+ this.expectedFileLength = expectedFileLength;
}
/**
* Opens a file with known size for writing.
*
- * @param path File path.
- * @param fileSize File size. If the file size is unknown, pass {@link #UNKNOWN_FILE_SIZE}.
+ * @param file File path.
+ * @param expectedFileLength Expected file size.
* @return Chunked file writer.
* @throws FileNotFoundException If the file is not found.
*/
- static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ static ChunkedFileWriter open(File file, long expectedFileLength) throws IOException {
+ return new ChunkedFileWriter(new BufferedOutputStream(new FileOutputStream(file)), expectedFileLength);
}
/**
- * Writes a chunk to the file.
+ * Writes a chunk to the file. The chunk number must be equal to the next expected chunk number. Closes the file if the last chunk is
+ * written. Returns true if the last chunk is written.
*
- * @param chunk Chunk.
+ * @param chunk Chunk to write.
+ * @return True if the last chunk is written. False otherwise.
* @throws IOException If an I/O error occurs.
+ * @throws FileValidationException If the chunk number or file size is invalid or expected file size is exceeded.
*/
- void write(FileChunkMessage chunk) throws IOException {
- chunks.add(chunk);
- while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
- raf.write(chunks.poll().data());
+ boolean write(FileChunkMessage chunk) throws IOException {
+ if (chunk.number() != expectedNextChunkNumber) {
+ throw new FileValidationException("Chunk number mismatch: expected " + expectedNextChunkNumber + ", actual " + chunk.number());
}
- if (fileSize != UNKNOWN_FILE_SIZE && raf.getFilePointer() > fileSize) {
- throw new FileValidationException("File size exceeded: expected " + fileSize + ", actual " + raf.getFilePointer());
- }
- }
+ lock.lock();
+ try {
+ stream.write(chunk.data());
Review Comment:
java.io.FilterOutputStream#write(byte[]) returns void, so I suppose there is a guarantee ))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288141231
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Startup timeout in seconds. */
+ private static final int STARTUP_TIMEOUT = 3;
Review Comment:
I have partially copied the code from `org.apache.ignite.network.scalecube.ItScaleCubeNetworkMessagingTest.Cluster` and 3 seconds is enough there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286696683
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
Review Comment:
No, it's called from `org.apache.ignite.internal.network.file.FileReceiver#executorService`
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
Review Comment:
No, it's called from `org.apache.ignite.internal.network.file.FileReceiver#executorService`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286703455
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
Review Comment:
We reach it by locking in`org.apache.ignite.internal.network.file.FileTransferMessagesHandler`. See `fileNameToLock`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288490248
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
+ */
+public interface FileConsumer<M extends Metadata> {
+ /**
+ * Handles the uploaded files.
+ *
+ * @param metadata The metadata of the uploaded file.
+ * @param uploadedFiles The temporary files that were uploaded. These files will be deleted after the method returns.
Review Comment:
Exactly. If a consumer wants to keep files, it should move them to another location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286699022
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
Review Comment:
Since we write chunks from different threads, we can get them in out-of-order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288120432
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
it throws "file not found" exception instead of ignoring missing files, as it should according to its name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1289862537
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
+ */
+public interface FileConsumer<M extends Metadata> {
+ /**
+ * Handles the uploaded files.
+ *
+ * @param metadata The metadata of the uploaded file.
+ * @param uploadedFiles The temporary files that were uploaded. These files will be deleted after the method returns.
Review Comment:
The service doesn't use system temporary directories. FileTransferServiceImpl receives the transfer directory in the constructor, where all files will be temporarily saved during transfers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1290008959
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/Channel.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import org.apache.ignite.network.ChannelType;
+
+/**
+ * Channel types used by the file transfer protocol.
+ */
+final class Channel {
+ /**
+ * File transfer channel.
+ */
+ static final ChannelType FILE_TRANSFER_CHANNEL = ChannelType.register((short) 1, "FileTransfer");
Review Comment:
I have fixed it locally. Will push it with other fixes little bit later
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/Channel.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import org.apache.ignite.network.ChannelType;
+
+/**
+ * Channel types used by the file transfer protocol.
+ */
+final class Channel {
+ /**
+ * File transfer channel.
+ */
+ static final ChannelType FILE_TRANSFER_CHANNEL = ChannelType.register((short) 1, "FileTransfer");
Review Comment:
I have fixed it locally. Will push it with other fixes a little bit later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288160458
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
Review Comment:
Updated the documentation of the class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1295639315
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -488,10 +488,7 @@ void uploadFilesWhenDoNotHaveAccessToWrite() {
);
// Check that files transfer failed.
- assertThat(
- uploaded,
- willThrow(FileTransferException.class, "Failed to upload files:")
- );
+ assertThat(uploaded, willThrow(FileTransferException.class));
Review Comment:
Why did you remove the error message check? Looked nice
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -51,63 +61,59 @@ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize)
* @param file File.
* @param chunkSize Chunk size.
* @return Chunked file reader.
- * @throws IOException If an I/O error occurs.
+ * @throws FileNotFoundException If the file does not exist.
*/
- static ChunkedFileReader open(File file, int chunkSize) throws IOException {
- return new ChunkedFileReader(file.getName(), new RandomAccessFile(file, "r"), chunkSize);
+ static ChunkedFileReader open(File file, int chunkSize) throws FileNotFoundException {
+ return new ChunkedFileReader(file.length(), new BufferedInputStream(new FileInputStream(file)), chunkSize);
}
/**
* Returns {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*
* @return {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*/
- boolean hasNextChunk() throws IOException {
- return raf.getFilePointer() != raf.length();
+ boolean hasNextChunk() {
+ return offset < length;
}
/**
- * Reads the next chunk.
+ * Reads the next chunk. If there are no more chunks to read, throws an exception. If the last chunk is read successfully, closes the
+ * file.
*
* @return Chunk data.
+ * @throws IllegalStateException If there are no more chunks to read.
* @throws IOException If an I/O error occurs.
*/
byte[] readNextChunk() throws IOException {
if (!hasNextChunk()) {
- throw new IOException("No more chunks to read");
+ throw new IllegalStateException("No more chunks to read");
}
- int toRead = (int) Math.min(chunkSize, length() - offset());
+ int toRead = (int) Math.min(chunkSize, length - offset);
byte[] data = new byte[toRead];
- raf.read(data);
- return data;
- }
+ stream.read(data);
+ offset += toRead;
+ nextChunkNumber++;
- /**
- * Returns the file name.
- *
- * @return File name.
- */
- String fileName() {
- return fileName;
- }
+ if (offset == length) {
+ stream.close();
+ }
- /**
- * Returns the current chunk offset.
- *
- * @return Chunk size.
- */
- long offset() throws IOException {
- return raf.getFilePointer();
+ return data;
}
/**
- * Returns the file length.
+ * Returns the number of the next chunk to read.
*
- * @return File length.
+ * @return The number of the next chunk to read.
+ * @throws IllegalStateException If there are no more chunks to read.
*/
- long length() throws IOException {
- return raf.length();
+ int nextChunkNumber() {
+ if (!hasNextChunk()) {
+ throw new IllegalStateException("No more chunks to read");
Review Comment:
Same here, this is one of the most widely misused exceptions out there.
Why do we even need the exception in this method in the first place?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -242,87 +249,104 @@ private void processUploadRequest(FileUploadRequest message, String senderConsis
CompletableFuture<List<Path>> uploadedFiles = directoryFuture.thenCompose(directory -> fileReceiver.registerTransfer(
senderConsistentId,
transferId,
+ message.headers(),
directory
)).whenComplete((v, e) -> {
if (e != null) {
LOG.error("Failed to register file transfer. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
}
});
- runAsync(() -> fileReceiver.receiveFileHeaders(transferId, message.headers()), executorService)
- .handle((v, e) -> {
- FileUploadResponse response = messageFactory.fileUploadResponse()
- .transferId(transferId)
- .error(e != null ? buildError(new FileTransferException("Failed to receive file headers", e)) : null)
- .build();
+ FileTransferInitResponse response = messageFactory.fileTransferInitResponse().build();
- return messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
- })
- .thenCompose(Function.identity())
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
.whenComplete((v, e) -> {
if (e != null) {
- LOG.error("Failed to send file upload response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
+ LOG.error("Failed to send file transfer response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
fileReceiver.cancelTransfer(transferId, e);
}
})
- .thenCompose(ignored -> uploadedFiles)
- .thenCompose(files -> getFileConsumer(identifier).consume(identifier, files))
- .whenComplete((v, e) -> {
+ .thenComposeAsync(ignored -> uploadedFiles, executorService)
+ .handleAsync((files, e) -> {
+ if (e != null && transferIdToConsumer.containsKey(transferId)) {
+ transferIdToConsumer.get(transferId).onError(e);
+ return failedFuture(e);
+ } else {
+ FileConsumer<Identifier> consumer = transferIdToConsumer.containsKey(transferId)
+ ? transferIdToConsumer.get(transferId)
+ : getFileConsumer(identifier);
+ return consumer.consume(identifier, files);
+ }
+ }, executorService)
+ .thenComposeAsync(it -> it, executorService)
Review Comment:
You could use `identity()`
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -242,87 +249,104 @@ private void processUploadRequest(FileUploadRequest message, String senderConsis
CompletableFuture<List<Path>> uploadedFiles = directoryFuture.thenCompose(directory -> fileReceiver.registerTransfer(
senderConsistentId,
transferId,
+ message.headers(),
directory
)).whenComplete((v, e) -> {
if (e != null) {
LOG.error("Failed to register file transfer. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
}
});
- runAsync(() -> fileReceiver.receiveFileHeaders(transferId, message.headers()), executorService)
- .handle((v, e) -> {
- FileUploadResponse response = messageFactory.fileUploadResponse()
- .transferId(transferId)
- .error(e != null ? buildError(new FileTransferException("Failed to receive file headers", e)) : null)
- .build();
+ FileTransferInitResponse response = messageFactory.fileTransferInitResponse().build();
- return messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
- })
- .thenCompose(Function.identity())
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
.whenComplete((v, e) -> {
if (e != null) {
- LOG.error("Failed to send file upload response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
+ LOG.error("Failed to send file transfer response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
fileReceiver.cancelTransfer(transferId, e);
}
})
- .thenCompose(ignored -> uploadedFiles)
- .thenCompose(files -> getFileConsumer(identifier).consume(identifier, files))
- .whenComplete((v, e) -> {
+ .thenComposeAsync(ignored -> uploadedFiles, executorService)
+ .handleAsync((files, e) -> {
+ if (e != null && transferIdToConsumer.containsKey(transferId)) {
Review Comment:
Please add some comments. All these procedures are hard to understand.
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileSenderTest.java:
##########
@@ -106,25 +116,61 @@ void sendMultipleFiles() {
}
@Test
- void exceptionIsThrownIfFileTransferFailed() {
+ void exceptionIsThrownWhenInvokeReturnException() {
// Setup messaging service to fail on second file transfer.
AtomicInteger count = new AtomicInteger();
- given(messagingService.send(anyString(), any(), any())).will(invocation -> {
+ doAnswer(invocation -> {
if (count.incrementAndGet() == 2) {
return failedFuture(new RuntimeException("Test exception"));
} else {
- return completedFuture(null);
+ return completedFuture(messageFactory.fileChunkResponse().build());
}
+ })
+ .when(messagingService)
+ .invoke(anyString(), eq(Channel.FILE_TRANSFER_CHANNEL), any(FileChunkMessage.class), eq(RESPONSE_TIMEOUT));
- });
+ // When.
+ Path randomFile = FileGenerator.randomFile(workDir, CHUNK_SIZE * 5);
+ UUID transferId = UUID.randomUUID();
+ FileSender sender = new FileSender(
+ CHUNK_SIZE,
+ new Semaphore(4),
+ RESPONSE_TIMEOUT, messagingService,
+ Executors.newSingleThreadExecutor()
Review Comment:
Where's the code that stops these executors?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -242,87 +249,104 @@ private void processUploadRequest(FileUploadRequest message, String senderConsis
CompletableFuture<List<Path>> uploadedFiles = directoryFuture.thenCompose(directory -> fileReceiver.registerTransfer(
senderConsistentId,
transferId,
+ message.headers(),
directory
)).whenComplete((v, e) -> {
if (e != null) {
LOG.error("Failed to register file transfer. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
}
});
- runAsync(() -> fileReceiver.receiveFileHeaders(transferId, message.headers()), executorService)
- .handle((v, e) -> {
- FileUploadResponse response = messageFactory.fileUploadResponse()
- .transferId(transferId)
- .error(e != null ? buildError(new FileTransferException("Failed to receive file headers", e)) : null)
- .build();
+ FileTransferInitResponse response = messageFactory.fileTransferInitResponse().build();
- return messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
- })
- .thenCompose(Function.identity())
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
.whenComplete((v, e) -> {
if (e != null) {
- LOG.error("Failed to send file upload response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
+ LOG.error("Failed to send file transfer response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
fileReceiver.cancelTransfer(transferId, e);
}
})
- .thenCompose(ignored -> uploadedFiles)
- .thenCompose(files -> getFileConsumer(identifier).consume(identifier, files))
- .whenComplete((v, e) -> {
+ .thenComposeAsync(ignored -> uploadedFiles, executorService)
+ .handleAsync((files, e) -> {
+ if (e != null && transferIdToConsumer.containsKey(transferId)) {
+ transferIdToConsumer.get(transferId).onError(e);
+ return failedFuture(e);
+ } else {
+ FileConsumer<Identifier> consumer = transferIdToConsumer.containsKey(transferId)
+ ? transferIdToConsumer.get(transferId)
+ : getFileConsumer(identifier);
+ return consumer.consume(identifier, files);
+ }
+ }, executorService)
+ .thenComposeAsync(it -> it, executorService)
+ .whenCompleteAsync((v, e) -> {
if (e != null) {
LOG.error(
- "Failed to handle file upload. Transfer ID: {}. Metadata: {}",
+ "Failed to handle file transfer. Transfer ID: {}. Metadata: {}",
e,
transferId,
identifier
);
}
directoryFuture.thenAccept(IgniteUtils::deleteIfExists);
- });
+ }, executorService);
}
private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
supplyAsync(() -> getFileProvider(message.identifier()), executorService)
.thenCompose(provider -> provider.files(message.identifier()))
.whenComplete((files, e) -> {
if (e != null) {
- LOG.error("Failed to get files for download. Metadata: {}", message.identifier(), e);
+ LOG.error("Failed to get files for download. Metadata: {}", e, message.identifier());
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .error(buildError(e))
+ .error(fromThrowable(messageFactory, e))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
} else if (files.isEmpty()) {
LOG.warn("No files to download. Metadata: {}", message.identifier());
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .error(buildError(new FileTransferException("No files to download")))
+ .error(fromThrowable(messageFactory, new FileTransferException("No files to download")))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
} else {
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .headers(fromPaths(messageFactory, files))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
- .thenComposeAsync(v -> sendFiles(senderConsistentId, message.transferId(), files), executorService);
+ .thenComposeAsync(v -> {
+ return transferFilesToNode(senderConsistentId, message.transferId(), message.identifier(), files);
+ }, executorService);
}
});
}
- private void processFileChunkMessage(FileChunkMessage message) {
- runAsync(() -> fileReceiver.receiveFileChunk(message), executorService);
+ private void processFileChunkMessage(FileChunkMessage message, String senderConsistentId, long correlationId) {
+ runAsync(() -> fileReceiver.receiveFileChunk(message), executorService)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to receive file chunk. Transfer ID: {}", e, message.transferId());
Review Comment:
By the way, I don't think that this is how we should format our log messages, usually we have all the extra data in brackets, not as a separate sentence. This comment applied to a lot of your log messages
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -17,99 +17,87 @@
package org.apache.ignite.internal.network.file;
+import java.io.BufferedOutputStream;
+import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.file.Path;
-import java.util.PriorityQueue;
-import java.util.Queue;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
import org.apache.ignite.internal.network.file.exception.FileValidationException;
import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
/**
- * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
- * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
- * chunk is not found, the file is not written to. The writer is not thread-safe.
+ * Chunked file writer. Writes chunks to a file. Checks that the file size and chunk numbers are valid.
*/
class ChunkedFileWriter implements AutoCloseable {
- private static final int UNKNOWN_FILE_SIZE = -1;
+ private final BufferedOutputStream stream;
- private final RandomAccessFile raf;
+ private final long expectedFileLength;
- private long fileSize;
+ private long bytesWritten = 0;
- private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
-
- private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
- if (fileSize < UNKNOWN_FILE_SIZE) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
-
- this.raf = raf;
- this.fileSize = fileSize;
- }
+ private int expectedNextChunkNumber = 0;
/**
- * Opens a file with unknown size for writing.
- *
- * @param path File path.
- * @return Chunked file writer.
- * @throws FileNotFoundException If the file is not found.
+ * Lock to synchronize access to the file. We don't write to the file simultaneously, but the next chunk number and bytes written can be
+ * updated from different threads.
*/
- static ChunkedFileWriter open(Path path) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), UNKNOWN_FILE_SIZE);
+ private final Lock lock = new ReentrantLock();
+
+ private ChunkedFileWriter(BufferedOutputStream stream, long expectedFileLength) {
+ this.stream = stream;
+ this.expectedFileLength = expectedFileLength;
}
/**
* Opens a file with known size for writing.
*
- * @param path File path.
- * @param fileSize File size. If the file size is unknown, pass {@link #UNKNOWN_FILE_SIZE}.
+ * @param file File path.
+ * @param expectedFileLength Expected file size.
* @return Chunked file writer.
* @throws FileNotFoundException If the file is not found.
*/
- static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ static ChunkedFileWriter open(File file, long expectedFileLength) throws IOException {
+ return new ChunkedFileWriter(new BufferedOutputStream(new FileOutputStream(file)), expectedFileLength);
}
/**
- * Writes a chunk to the file.
+ * Writes a chunk to the file. The chunk number must be equal to the next expected chunk number. Closes the file if the last chunk is
+ * written. Returns true if the last chunk is written.
*
- * @param chunk Chunk.
+ * @param chunk Chunk to write.
+ * @return True if the last chunk is written. False otherwise.
* @throws IOException If an I/O error occurs.
+ * @throws FileValidationException If the chunk number or file size is invalid or expected file size is exceeded.
*/
- void write(FileChunkMessage chunk) throws IOException {
- chunks.add(chunk);
- while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
- raf.write(chunks.poll().data());
+ boolean write(FileChunkMessage chunk) throws IOException {
+ if (chunk.number() != expectedNextChunkNumber) {
+ throw new FileValidationException("Chunk number mismatch: expected " + expectedNextChunkNumber + ", actual " + chunk.number());
}
- if (fileSize != UNKNOWN_FILE_SIZE && raf.getFilePointer() > fileSize) {
- throw new FileValidationException("File size exceeded: expected " + fileSize + ", actual " + raf.getFilePointer());
- }
- }
+ lock.lock();
+ try {
+ stream.write(chunk.data());
Review Comment:
Same problem as with "read". As far as I know, there's no guarantee that all bytes will be written
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -51,63 +61,59 @@ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize)
* @param file File.
* @param chunkSize Chunk size.
* @return Chunked file reader.
- * @throws IOException If an I/O error occurs.
+ * @throws FileNotFoundException If the file does not exist.
*/
- static ChunkedFileReader open(File file, int chunkSize) throws IOException {
- return new ChunkedFileReader(file.getName(), new RandomAccessFile(file, "r"), chunkSize);
+ static ChunkedFileReader open(File file, int chunkSize) throws FileNotFoundException {
+ return new ChunkedFileReader(file.length(), new BufferedInputStream(new FileInputStream(file)), chunkSize);
}
/**
* Returns {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*
* @return {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*/
- boolean hasNextChunk() throws IOException {
- return raf.getFilePointer() != raf.length();
+ boolean hasNextChunk() {
+ return offset < length;
}
/**
- * Reads the next chunk.
+ * Reads the next chunk. If there are no more chunks to read, throws an exception. If the last chunk is read successfully, closes the
+ * file.
*
* @return Chunk data.
+ * @throws IllegalStateException If there are no more chunks to read.
* @throws IOException If an I/O error occurs.
*/
byte[] readNextChunk() throws IOException {
if (!hasNextChunk()) {
- throw new IOException("No more chunks to read");
+ throw new IllegalStateException("No more chunks to read");
Review Comment:
My advice is not to use IllegalStateException. It doesn't help, IOException was fine.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -242,87 +249,104 @@ private void processUploadRequest(FileUploadRequest message, String senderConsis
CompletableFuture<List<Path>> uploadedFiles = directoryFuture.thenCompose(directory -> fileReceiver.registerTransfer(
senderConsistentId,
transferId,
+ message.headers(),
directory
)).whenComplete((v, e) -> {
if (e != null) {
LOG.error("Failed to register file transfer. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
}
});
- runAsync(() -> fileReceiver.receiveFileHeaders(transferId, message.headers()), executorService)
- .handle((v, e) -> {
- FileUploadResponse response = messageFactory.fileUploadResponse()
- .transferId(transferId)
- .error(e != null ? buildError(new FileTransferException("Failed to receive file headers", e)) : null)
- .build();
+ FileTransferInitResponse response = messageFactory.fileTransferInitResponse().build();
- return messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
- })
- .thenCompose(Function.identity())
+ messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
.whenComplete((v, e) -> {
if (e != null) {
- LOG.error("Failed to send file upload response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
+ LOG.error("Failed to send file transfer response. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
fileReceiver.cancelTransfer(transferId, e);
}
})
- .thenCompose(ignored -> uploadedFiles)
- .thenCompose(files -> getFileConsumer(identifier).consume(identifier, files))
- .whenComplete((v, e) -> {
+ .thenComposeAsync(ignored -> uploadedFiles, executorService)
+ .handleAsync((files, e) -> {
+ if (e != null && transferIdToConsumer.containsKey(transferId)) {
+ transferIdToConsumer.get(transferId).onError(e);
+ return failedFuture(e);
+ } else {
+ FileConsumer<Identifier> consumer = transferIdToConsumer.containsKey(transferId)
+ ? transferIdToConsumer.get(transferId)
+ : getFileConsumer(identifier);
+ return consumer.consume(identifier, files);
+ }
+ }, executorService)
+ .thenComposeAsync(it -> it, executorService)
+ .whenCompleteAsync((v, e) -> {
if (e != null) {
LOG.error(
- "Failed to handle file upload. Transfer ID: {}. Metadata: {}",
+ "Failed to handle file transfer. Transfer ID: {}. Metadata: {}",
e,
transferId,
identifier
);
}
directoryFuture.thenAccept(IgniteUtils::deleteIfExists);
- });
+ }, executorService);
}
private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
supplyAsync(() -> getFileProvider(message.identifier()), executorService)
.thenCompose(provider -> provider.files(message.identifier()))
.whenComplete((files, e) -> {
if (e != null) {
- LOG.error("Failed to get files for download. Metadata: {}", message.identifier(), e);
+ LOG.error("Failed to get files for download. Metadata: {}", e, message.identifier());
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .error(buildError(e))
+ .error(fromThrowable(messageFactory, e))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
} else if (files.isEmpty()) {
LOG.warn("No files to download. Metadata: {}", message.identifier());
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .error(buildError(new FileTransferException("No files to download")))
+ .error(fromThrowable(messageFactory, new FileTransferException("No files to download")))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId);
} else {
FileDownloadResponse response = messageFactory.fileDownloadResponse()
- .headers(fromPaths(messageFactory, files))
.build();
messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
- .thenComposeAsync(v -> sendFiles(senderConsistentId, message.transferId(), files), executorService);
+ .thenComposeAsync(v -> {
+ return transferFilesToNode(senderConsistentId, message.transferId(), message.identifier(), files);
+ }, executorService);
}
});
}
- private void processFileChunkMessage(FileChunkMessage message) {
- runAsync(() -> fileReceiver.receiveFileChunk(message), executorService);
+ private void processFileChunkMessage(FileChunkMessage message, String senderConsistentId, long correlationId) {
+ runAsync(() -> fileReceiver.receiveFileChunk(message), executorService)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to receive file chunk. Transfer ID: {}", e, message.transferId());
Review Comment:
What does it mean, "failed to receive"? You have received it, but wasn't able to process the data. It's not the same
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -51,63 +61,59 @@ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize)
* @param file File.
* @param chunkSize Chunk size.
* @return Chunked file reader.
- * @throws IOException If an I/O error occurs.
+ * @throws FileNotFoundException If the file does not exist.
*/
- static ChunkedFileReader open(File file, int chunkSize) throws IOException {
- return new ChunkedFileReader(file.getName(), new RandomAccessFile(file, "r"), chunkSize);
+ static ChunkedFileReader open(File file, int chunkSize) throws FileNotFoundException {
+ return new ChunkedFileReader(file.length(), new BufferedInputStream(new FileInputStream(file)), chunkSize);
}
/**
* Returns {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*
* @return {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*/
- boolean hasNextChunk() throws IOException {
- return raf.getFilePointer() != raf.length();
+ boolean hasNextChunk() {
+ return offset < length;
}
/**
- * Reads the next chunk.
+ * Reads the next chunk. If there are no more chunks to read, throws an exception. If the last chunk is read successfully, closes the
+ * file.
*
* @return Chunk data.
+ * @throws IllegalStateException If there are no more chunks to read.
* @throws IOException If an I/O error occurs.
*/
byte[] readNextChunk() throws IOException {
if (!hasNextChunk()) {
- throw new IOException("No more chunks to read");
+ throw new IllegalStateException("No more chunks to read");
}
- int toRead = (int) Math.min(chunkSize, length() - offset());
+ int toRead = (int) Math.min(chunkSize, length - offset);
byte[] data = new byte[toRead];
- raf.read(data);
- return data;
- }
+ stream.read(data);
Review Comment:
Does this call guarantees that if fills the entire array? I don't think so, it should return the real number of bytes that it had read. Too bad I didn't notice it earlier. Always be careful with read/write API, it's deceivingly simple.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -17,99 +17,87 @@
package org.apache.ignite.internal.network.file;
+import java.io.BufferedOutputStream;
+import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.file.Path;
-import java.util.PriorityQueue;
-import java.util.Queue;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
import org.apache.ignite.internal.network.file.exception.FileValidationException;
import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
/**
- * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
- * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
- * chunk is not found, the file is not written to. The writer is not thread-safe.
+ * Chunked file writer. Writes chunks to a file. Checks that the file size and chunk numbers are valid.
*/
class ChunkedFileWriter implements AutoCloseable {
- private static final int UNKNOWN_FILE_SIZE = -1;
+ private final BufferedOutputStream stream;
- private final RandomAccessFile raf;
+ private final long expectedFileLength;
- private long fileSize;
+ private long bytesWritten = 0;
- private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
-
- private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
- if (fileSize < UNKNOWN_FILE_SIZE) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
-
- this.raf = raf;
- this.fileSize = fileSize;
- }
+ private int expectedNextChunkNumber = 0;
/**
- * Opens a file with unknown size for writing.
- *
- * @param path File path.
- * @return Chunked file writer.
- * @throws FileNotFoundException If the file is not found.
+ * Lock to synchronize access to the file. We don't write to the file simultaneously, but the next chunk number and bytes written can be
+ * updated from different threads.
*/
- static ChunkedFileWriter open(Path path) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), UNKNOWN_FILE_SIZE);
+ private final Lock lock = new ReentrantLock();
+
+ private ChunkedFileWriter(BufferedOutputStream stream, long expectedFileLength) {
+ this.stream = stream;
+ this.expectedFileLength = expectedFileLength;
}
/**
* Opens a file with known size for writing.
*
- * @param path File path.
- * @param fileSize File size. If the file size is unknown, pass {@link #UNKNOWN_FILE_SIZE}.
+ * @param file File path.
+ * @param expectedFileLength Expected file size.
* @return Chunked file writer.
* @throws FileNotFoundException If the file is not found.
*/
- static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ static ChunkedFileWriter open(File file, long expectedFileLength) throws IOException {
+ return new ChunkedFileWriter(new BufferedOutputStream(new FileOutputStream(file)), expectedFileLength);
}
/**
- * Writes a chunk to the file.
+ * Writes a chunk to the file. The chunk number must be equal to the next expected chunk number. Closes the file if the last chunk is
+ * written. Returns true if the last chunk is written.
*
- * @param chunk Chunk.
+ * @param chunk Chunk to write.
+ * @return True if the last chunk is written. False otherwise.
* @throws IOException If an I/O error occurs.
+ * @throws FileValidationException If the chunk number or file size is invalid or expected file size is exceeded.
*/
- void write(FileChunkMessage chunk) throws IOException {
- chunks.add(chunk);
- while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
- raf.write(chunks.poll().data());
+ boolean write(FileChunkMessage chunk) throws IOException {
+ if (chunk.number() != expectedNextChunkNumber) {
+ throw new FileValidationException("Chunk number mismatch: expected " + expectedNextChunkNumber + ", actual " + chunk.number());
}
- if (fileSize != UNKNOWN_FILE_SIZE && raf.getFilePointer() > fileSize) {
- throw new FileValidationException("File size exceeded: expected " + fileSize + ", actual " + raf.getFilePointer());
- }
- }
+ lock.lock();
+ try {
+ stream.write(chunk.data());
+ expectedNextChunkNumber++;
+ bytesWritten += chunk.data().length;
- /**
- * Checks if the file is finished.
- *
- * @return {@code True} if the file is finished.
- * @throws IOException If an I/O error occurs.
- */
- boolean isFinished() throws IOException {
- return raf.length() == fileSize;
- }
+ if (bytesWritten > expectedFileLength) {
+ throw new FileValidationException("File size mismatch: expected " + expectedFileLength + ", actual " + bytesWritten);
+ }
- /**
- * Sets the file size.
- *
- * @param fileSize File size.
- */
- void fileSize(long fileSize) {
- if (fileSize < 0) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
+ if (bytesWritten == expectedFileLength) {
+ stream.flush();
Review Comment:
What's this for?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -17,99 +17,87 @@
package org.apache.ignite.internal.network.file;
+import java.io.BufferedOutputStream;
+import java.io.File;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.file.Path;
-import java.util.PriorityQueue;
-import java.util.Queue;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
import org.apache.ignite.internal.network.file.exception.FileValidationException;
import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
/**
- * Chunked file writer. Writes chunks in order. If a chunk is not in order, it is stored in a queue. When the next chunk is written, the
- * queues are checked for the next chunk. If the next chunk is found, it is written to the file and removed from the queue. If the next
- * chunk is not found, the file is not written to. The writer is not thread-safe.
+ * Chunked file writer. Writes chunks to a file. Checks that the file size and chunk numbers are valid.
*/
class ChunkedFileWriter implements AutoCloseable {
- private static final int UNKNOWN_FILE_SIZE = -1;
+ private final BufferedOutputStream stream;
- private final RandomAccessFile raf;
+ private final long expectedFileLength;
- private long fileSize;
+ private long bytesWritten = 0;
- private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
-
- private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
- if (fileSize < UNKNOWN_FILE_SIZE) {
- throw new IllegalArgumentException("File size must be non-negative");
- }
-
- this.raf = raf;
- this.fileSize = fileSize;
- }
+ private int expectedNextChunkNumber = 0;
/**
- * Opens a file with unknown size for writing.
- *
- * @param path File path.
- * @return Chunked file writer.
- * @throws FileNotFoundException If the file is not found.
+ * Lock to synchronize access to the file. We don't write to the file simultaneously, but the next chunk number and bytes written can be
+ * updated from different threads.
*/
- static ChunkedFileWriter open(Path path) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), UNKNOWN_FILE_SIZE);
+ private final Lock lock = new ReentrantLock();
+
+ private ChunkedFileWriter(BufferedOutputStream stream, long expectedFileLength) {
+ this.stream = stream;
+ this.expectedFileLength = expectedFileLength;
}
/**
* Opens a file with known size for writing.
*
- * @param path File path.
- * @param fileSize File size. If the file size is unknown, pass {@link #UNKNOWN_FILE_SIZE}.
+ * @param file File path.
+ * @param expectedFileLength Expected file size.
* @return Chunked file writer.
* @throws FileNotFoundException If the file is not found.
*/
- static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
- return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ static ChunkedFileWriter open(File file, long expectedFileLength) throws IOException {
+ return new ChunkedFileWriter(new BufferedOutputStream(new FileOutputStream(file)), expectedFileLength);
}
/**
- * Writes a chunk to the file.
+ * Writes a chunk to the file. The chunk number must be equal to the next expected chunk number. Closes the file if the last chunk is
+ * written. Returns true if the last chunk is written.
*
- * @param chunk Chunk.
+ * @param chunk Chunk to write.
+ * @return True if the last chunk is written. False otherwise.
* @throws IOException If an I/O error occurs.
+ * @throws FileValidationException If the chunk number or file size is invalid or expected file size is exceeded.
*/
- void write(FileChunkMessage chunk) throws IOException {
- chunks.add(chunk);
- while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
- raf.write(chunks.poll().data());
+ boolean write(FileChunkMessage chunk) throws IOException {
+ if (chunk.number() != expectedNextChunkNumber) {
Review Comment:
Maybe this check should be inside of the try/finally block
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -370,7 +394,7 @@ public <M extends Identifier> void addFileConsumer(
Class<M> identifier,
FileConsumer<M> consumer
) {
- metadataToHandler.compute(
+ metadataToConsumer.compute(
Review Comment:
Why did you renamed "handler" to "consumer"? Consumer is too broad of a term
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -112,7 +113,12 @@ public class FileTransferServiceImpl implements FileTransferService {
/**
* Map of file handlers.
*/
- private final Map<Short, FileConsumer<Identifier>> metadataToHandler = new ConcurrentHashMap<>();
+ private final Map<Short, FileConsumer<Identifier>> metadataToConsumer = new ConcurrentHashMap<>();
+
+ /**
+ * Map of download requests consumers.
+ */
+ private final Map<UUID, DownloadRequestConsumer> transferIdToConsumer = new ConcurrentHashMap<>();
Review Comment:
Why "consumer" instead of "handler"?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.allOf;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+import static org.apache.ignite.internal.network.file.Channel.FILE_TRANSFER_CHANNEL;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.network.MessagingService;
+
+/**
+ * A class that sends files to a node. It uses a rate limiter to limit the bandwidth used. It also uses a thread pool to send the files in
+ * parallel.
+ */
+class FileSender {
+ private final int chunkSize;
+
+ private final Semaphore rateLimiter;
Review Comment:
Can you reflect this in the javadoc?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -390,75 +414,64 @@ public CompletableFuture<List<Path>> download(String sourceNodeConsistentId, Ide
.identifier(identifier)
.build();
- try {
- Path directory = createTransferDirectory(transferId);
-
- CompletableFuture<List<Path>> downloadedFiles = fileReceiver.registerTransfer(
- sourceNodeConsistentId,
- transferId,
- directory
- )
- .thenApply(ignored -> {
- try {
- IgniteUtils.deleteIfExists(targetDir);
-
- Files.move(directory, targetDir, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
-
- try (Stream<Path> stream = Files.list(targetDir)) {
- return stream.collect(Collectors.toList());
- }
- } catch (IOException e) {
- throw new FileTransferException("Failed to move downloaded files to target directory", e);
- }
- })
- .whenComplete((v, e) -> {
- if (e != null) {
- LOG.error("Failed to download files. Identifier: {}", identifier, e);
- }
-
- IgniteUtils.deleteIfExists(directory);
- });
-
- return messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
- .thenApply(FileDownloadResponse.class::cast)
- .whenCompleteAsync((response, e) -> {
- if (e != null) {
- fileReceiver.cancelTransfer(transferId, e);
- } else if (response.error() != null) {
- fileReceiver.cancelTransfer(transferId, buildException(response.error()));
- } else {
- fileReceiver.receiveFileHeaders(transferId, response.headers());
- }
- }, executorService)
- .thenCompose(v -> downloadedFiles);
- } catch (Exception e) {
- return failedFuture(e);
- }
+ CompletableFuture<List<Path>> downloadedFiles = new CompletableFuture<List<Path>>()
+ .whenComplete((v, e) -> transferIdToConsumer.remove(transferId));
+
+ transferIdToConsumer.put(transferId, new DownloadRequestConsumer(downloadedFiles, targetDir));
+
+ messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
+ .thenApply(FileDownloadResponse.class::cast)
+ .whenComplete((response, e) -> {
+ if (e != null) {
+ downloadedFiles.completeExceptionally(e);
+ } else if (response.error() != null) {
+ downloadedFiles.completeExceptionally(toException(response.error()));
+ }
+ });
+
+ return downloadedFiles;
}
@Override
public CompletableFuture<Void> upload(String targetNodeConsistentId, Identifier identifier) {
return getFileProvider(identifier).files(identifier)
- .thenCompose(files -> {
- FileUploadRequest message = messageFactory.fileUploadRequest()
- .identifier(identifier)
- .headers(fromPaths(messageFactory, files))
- .build();
+ .thenCompose(files -> transferFilesToNode(targetNodeConsistentId, UUID.randomUUID(), identifier, files));
+ }
+
+ private CompletableFuture<Void> transferFilesToNode(
+ String targetNodeConsistentId,
+ UUID transferId,
+ Identifier identifier,
+ List<Path> paths
+ ) {
+ if (paths.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
Review Comment:
Why is this a problem? What happens if user tries to do this, will he have an error in his log? It's not what he would expect
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileSenderTest.java:
##########
@@ -170,7 +219,7 @@ void maxConcurrentRequestsLimitIsNotExceeded() {
FileSender sender = new FileSender(
CHUNK_SIZE,
new Semaphore(maxConcurrentRequests),
- messagingService,
+ RESPONSE_TIMEOUT, messagingService,
Review Comment:
Every parameter is on a new line, please fox other such places as well
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -197,9 +209,16 @@ private CompletableFuture<Void> sendMessagesFromStream(
) {
try {
if (stream.hasNextMessage() && !shouldBeCancelled.get()) {
- return messagingService.send(receiverConsistentId, FILE_TRANSFER_CHANNEL, stream.nextMessage())
+ return messagingService.invoke(receiverConsistentId, FILE_TRANSFER_CHANNEL, stream.nextMessage(), responseTimeout)
+ .thenApply(FileChunkResponse.class::cast)
.thenComposeAsync(
- ignored -> sendMessagesFromStream(receiverConsistentId, stream, shouldBeCancelled),
+ ack -> {
+ if (ack.error() != null) {
+ return failedFuture(toException(ack.error()));
Review Comment:
How will receiver know that the operation is cancelled in case of error? Does it have its own timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296075408
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -197,9 +209,16 @@ private CompletableFuture<Void> sendMessagesFromStream(
) {
try {
if (stream.hasNextMessage() && !shouldBeCancelled.get()) {
- return messagingService.send(receiverConsistentId, FILE_TRANSFER_CHANNEL, stream.nextMessage())
+ return messagingService.invoke(receiverConsistentId, FILE_TRANSFER_CHANNEL, stream.nextMessage(), responseTimeout)
+ .thenApply(FileChunkResponse.class::cast)
.thenComposeAsync(
- ignored -> sendMessagesFromStream(receiverConsistentId, stream, shouldBeCancelled),
+ ack -> {
+ if (ack.error() != null) {
+ return failedFuture(toException(ack.error()));
Review Comment:
If org.apache.ignite.internal.network.file.FileSender#send returns a failed CompletableFuture, a FileTransferErrorMessage will be sent to the receiver. See org.apache.ignite.internal.network.file.FileTransferServiceImpl#sendFiles. Additionally, a test exists for this scenario: org.apache.ignite.internal.network.file.FileTransferServiceImplTest#uploadFailsWhenSenderReturnsException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] rpuch commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "rpuch (via GitHub)" <gi...@apache.org>.
rpuch commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281514633
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/FileContentMatcher.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import org.hamcrest.Description;
+import org.hamcrest.TypeSafeMatcher;
+
+/**
+ * {@link TypeSafeMatcher} for matching file content.
+ */
+public class FileContentMatcher extends TypeSafeMatcher<Path> {
+
+ private final byte[] expectedContent;
+
+ private FileContentMatcher(byte[] expectedContent) {
+ this.expectedContent = expectedContent;
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param path Path to the file.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(Path path) {
Review Comment:
How about adding a `Matcher` as a parameter? That would make the matcher more flexible. An overload without the `Matcher` parameter can still be left (it would use `is()` as a matcher)
##########
modules/file-transferring/src/testFixtures/java/org/apache/ignite/internal/network/file/FileAssertions.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static org.apache.ignite.internal.testframework.matchers.FileContentMatcher.hasContent;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * File assertions.
+ */
+public class FileAssertions {
+
+ /**
+ * Asserts that the content of the two directories is the same.
+ *
+ * @param expected Expected directory.
+ * @param actual Actual directory.
+ */
+ public static void assertContentEquals(Path expected, Path actual) {
+ try {
+ Files.walkFileTree(expected, new SimpleFileVisitor<>() {
+ @Override
+ public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
+ Path otherDir = actual.resolve(expected.relativize(dir));
+ if (!Files.exists(otherDir)) {
+ throw new AssertionError("Directory " + dir + " does not exist in " + actual);
+ }
+ List<String> expectedFiles = Arrays.stream(dir.toFile().listFiles())
Review Comment:
`File#listFiles()` is nullable. Probably a possibility for a null should be handled here
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/FileContentMatcher.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import org.hamcrest.Description;
+import org.hamcrest.TypeSafeMatcher;
+
+/**
+ * {@link TypeSafeMatcher} for matching file content.
+ */
+public class FileContentMatcher extends TypeSafeMatcher<Path> {
+
+ private final byte[] expectedContent;
+
+ private FileContentMatcher(byte[] expectedContent) {
+ this.expectedContent = expectedContent;
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param path Path to the file.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(Path path) {
+ try {
+ return new FileContentMatcher(Files.readAllBytes(path));
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+
+ @Override
+ protected boolean matchesSafely(Path path) {
+ try {
+ byte[] actualContent = Files.readAllBytes(path);
+ return Arrays.equals(actualContent, expectedContent);
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description.appendText("matches content: ").appendValue(expectedContent);
+ }
+
+ @Override
+ protected void describeMismatchSafely(Path item, Description mismatchDescription) {
+ try {
+ mismatchDescription.appendText("was ").appendValue(Files.readAllLines(item));
Review Comment:
Why is it `readAllLines()` here, while on line 55 it's `readAllBytes()`?
##########
modules/file-transferring/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferringTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.network.file.messages.TransferMetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferringTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node node0 = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = node0.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ File file1 = randomFile(unitPath, 1024 * 1024 * 10).toFile();
Review Comment:
Should we delete the files after the test?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringService.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+
+/**
+ * File transferring service.
+ */
+public interface FileTransferringService extends IgniteComponent {
+ /**
+ * Adds a file provider for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param provider Provider.
+ */
+ <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ );
+
+ /**
+ * Adds a file handler for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param handler Handler.
+ */
+ <M extends TransferMetadata> void addFileHandler(Class<M> metadata, FileHandler<M> handler);
Review Comment:
Don't we need a way to deregister a file handler (or a file provider)?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
Review Comment:
Why is it `Short`? It doesn't look like we can save a lot of memory this way
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringService.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+
+/**
+ * File transferring service.
+ */
+public interface FileTransferringService extends IgniteComponent {
+ /**
+ * Adds a file provider for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param provider Provider.
+ */
+ <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ );
+
+ /**
+ * Adds a file handler for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param handler Handler.
+ */
+ <M extends TransferMetadata> void addFileHandler(Class<M> metadata, FileHandler<M> handler);
+
+ /**
+ * Downloads files for the given metadata.
+ *
+ * @param node Node.
+ * @param transferMetadata Metadata.
+ * @return Temporary path to the downloaded files. The caller is responsible for deleting the files.
+ */
+ CompletableFuture<Path> download(String node, TransferMetadata transferMetadata);
+
+ /**
+ * Uploads files for the given metadata.
+ *
+ * @param node Node.
Review Comment:
Let's say explicitly that it's a consistent ID of a node
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.Executors.newSingleThreadExecutor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+ private final int chunkSize;
+ private final RateLimiter rateLimiter;
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final Queue<FileTransferringRequest> filesToSend = new ConcurrentLinkedQueue<>();
+
+ private final ExecutorService executorService = newSingleThreadExecutor(
+ new NamedThreadFactory("FileSenderExecutor", LOG)
Review Comment:
Thread prefix must contain node name for easier identification
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
Review Comment:
Le's add a catch-all to make sure we see if anything is missed
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringMessagesStream.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.ChunkedFileImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileHeaderImpl;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoImpl;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferringMessagesStream implements ManuallyCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ private final AtomicReference<FileTransferInfo> fileTransferInfo = new AtomicReference<>();
Review Comment:
Why is atomic used here? This class seems to be used in a single thread (at least, the queue is not thread-safe)
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.Executors.newSingleThreadExecutor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+ private final int chunkSize;
+ private final RateLimiter rateLimiter;
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final Queue<FileTransferringRequest> filesToSend = new ConcurrentLinkedQueue<>();
+
+ private final ExecutorService executorService = newSingleThreadExecutor(
+ new NamedThreadFactory("FileSenderExecutor", LOG)
+ );
+
+ FileSender(
+ int chunkSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send
+ ) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ FileTransferringRequest request = new FileTransferringRequest(receiverConsistentId, id, files, chunkSize);
+ filesToSend.add(request);
+ executorService.submit(this::processQueue);
+ return request.result();
+ }
+
+ /**
+ * Processes the queue of files to be sent.
+ */
+ private void processQueue() {
+ while (!filesToSend.isEmpty()) {
+ FileTransferringRequest request = filesToSend.peek();
+ try {
+ FileTransferringMessagesStream stream = request.messagesStream;
+ while (stream.hasNextMessage() && rateLimiter.tryAcquire()) {
+ send.apply(request.receiverConsistentId, stream.nextMessage())
+ .whenComplete((res, e) -> {
+ if (e != null) {
+ request.complete(e);
+ }
+ rateLimiter.release();
+ });
+ }
+ if (!stream.hasNextMessage()) {
+ request.complete();
+ filesToSend.remove();
+ }
+ } catch (Exception e) {
+ request.complete(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ executorService.shutdown();
+ filesToSend.forEach(it -> it.complete(new InterruptedException("File sender is closed")));
Review Comment:
`InterruptedException` has a very specific meaning and does not seem appropriate here. `NodeStoppingException` (if it's about node shutdown time) or maybe just `IgniteInternalException` seems better
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.util.IgniteUtils;
+
+class FileReceiver implements ManuallyCloseable {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<Path> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileReceiver(Path dir) {
+ this.dir = dir;
+ }
+
+ void receive(FileTransferInfo info) {
Review Comment:
There are three overloads of `receive()` method that do quite different things: they receive a 'whole transfer header' (once per transfer), a 'file header' (once per file, N times per transfer) and a 'file chunk' (M times per file). It seems that it would be better to name them differently (like `receiveTransferInfo()`, `receiveFileHeader()`, `receiveFileChunk()` to avoid confusion.
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringMessagesStream.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.ChunkedFileImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileHeaderImpl;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoImpl;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferringMessagesStream implements ManuallyCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ private final AtomicReference<FileTransferInfo> fileTransferInfo = new AtomicReference<>();
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param filesToSend the files to send. Must not be empty.
+ * @param chunkSize the size of the chunks to send. Must be positive.
+ */
+ FileTransferringMessagesStream(
+ UUID transferId,
+ List<File> filesToSend,
+ int chunkSize
+ ) {
+ if (filesToSend.isEmpty()) {
+ throw new IllegalArgumentException("Files to send cannot be empty.");
+ }
+
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive.");
+ }
+
+ this.transferId = transferId;
+ this.filesToSend = new LinkedList<>(filesToSend);
+ this.chunkSize = chunkSize;
+ this.fileTransferInfo.set(fileTransferInfo());
+ }
+
+ /**
+ * Returns true if there are more messages to send.
+ *
+ * @return true if there are more messages to send.
+ */
+ boolean hasNextMessage() throws IOException {
+ // check that the stream is not closed.
+ if (closed.get()) {
+ return false;
+ } else {
+ // check that there are more messages to send.
+ // 1. there is a file transfer info message to send.
+ // 2. there are files to send.
+ // 3. there is a current file to send.
+ return fileTransferInfo.get() != null || !filesToSend.isEmpty() || (currFile != null && !currFile.isFinished());
+ }
+ }
+
+ /**
+ * Returns the next message to send.
+ *
+ * @return the next message to send.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if there are no more messages to send.
+ */
+ NetworkMessage nextMessage() throws IOException {
+ if (!hasNextMessage()) {
+ throw new IllegalStateException("There are no more messages to send.");
+ }
+
+ FileTransferInfo info = fileTransferInfo.getAndSet(null);
+ if (info != null) {
+ return info;
+ } else {
+ if (currFile == null || currFile.isFinished()) {
+ openNextFile();
+ return header();
+ } else {
+ return nextChunk();
+ }
+ }
+ }
+
+ private FileTransferInfo fileTransferInfo() {
+ return FileTransferInfoImpl.builder()
+ .transferId(transferId)
+ .filesCount(filesToSend.size())
+ .build();
+ }
+
+ /**
+ * Returns the header of the current file to send.
+ */
+ private FileHeader header() throws IOException {
+ assert currFile != null : "Current file is null.";
+
+ return FileHeaderImpl.builder()
+ .transferId(transferId)
+ .fileName(currFile.fileName())
+ .fileSize(currFile.length())
+ .build();
+ }
+
+ /**
+ * Returns the next chunk of the current file. Throws an exception if the current file is finished.
+ *
+ * @return the next chunk of the current file.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if the current file is finished.
+ */
+ private ChunkedFile nextChunk() throws IOException {
+ assert currFile != null : "Current file is null.";
+ assert !currFile.isFinished() : "Current file is finished.";
+
+ return ChunkedFileImpl.builder()
Review Comment:
It looks like it's not a `ChunkedFile`, but a `FileChunk`
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.util.IgniteUtils;
+
+class FileReceiver implements ManuallyCloseable {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<Path> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileReceiver(Path dir) {
+ this.dir = dir;
+ }
+
+ void receive(FileTransferInfo info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+ filesCount.set(info.filesCount());
+ }
+
+ void receive(FileHeader header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> receive0(header));
+ }
+
+ void receive(ChunkedFile chunkedFile) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(chunkedFile.fileName(), () -> receive0(chunkedFile));
+ }
+
+ private void receive0(FileHeader header) {
+ try {
+ Path path = Files.createFile(dir.resolve(header.fileName()));
+ fileNameToWriter.put(header.fileName(), new ChunkedFileWriter(path, header.fileSize()));
+ } catch (IOException e) {
+ result.completeExceptionally(e);
+ }
+ }
+
+ private void receive0(ChunkedFile chunkedFile) {
+ try {
+ ChunkedFileWriter writer = fileNameToWriter.get(chunkedFile.fileName());
+ writer.write(chunkedFile);
+
+ if (writer.isFinished()) {
+ writer.close();
+ fileNameToWriter.remove(chunkedFile.fileName());
+ filesFinished.incrementAndGet();
+ }
+
+ if (filesFinished.get() == filesCount.get()) {
+ result.complete(dir);
+ }
+
+ } catch (IOException e) {
+ result.completeExceptionally(e);
Review Comment:
Should the writers be closed here as well?
##########
modules/network-api/src/main/java/org/apache/ignite/network/MessagingService.java:
##########
@@ -76,6 +76,26 @@ default CompletableFuture<Void> send(ClusterNode recipient, NetworkMessage msg)
*/
CompletableFuture<Void> send(ClusterNode recipient, ChannelType channelType, NetworkMessage msg);
+ /**
+ * Tries to send the given message via specified channel asynchronously to the specific cluster member.
+ *
+ * <p>Guarantees:
Review Comment:
Does `ChannelType` affect these guarantees? I mean, if m1 was sent using an overload without the `ChannelType`, but m2 was sent using an overload with a `ChannelType`, will the guarantees still hold?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringService.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+
+/**
+ * File transferring service.
+ */
+public interface FileTransferringService extends IgniteComponent {
+ /**
+ * Adds a file provider for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param provider Provider.
+ */
+ <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ );
+
+ /**
+ * Adds a file handler for the given metadata.
+ *
+ * @param metadata Metadata.
+ * @param handler Handler.
+ */
+ <M extends TransferMetadata> void addFileHandler(Class<M> metadata, FileHandler<M> handler);
+
+ /**
+ * Downloads files for the given metadata.
+ *
+ * @param node Node.
Review Comment:
Let's say explicitly that it's a consistent ID of a node
##########
modules/file-transferring/build.gradle:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply from: "$rootDir/buildscripts/java-core.gradle"
+apply from: "$rootDir/buildscripts/publishing.gradle"
+apply from: "$rootDir/buildscripts/java-junit5.gradle"
+apply from: "$rootDir/buildscripts/java-integration-test.gradle"
+apply from: "$rootDir/buildscripts/java-test-fixtures.gradle"
+
Review Comment:
Would `file-transfer` be a better name for the module?
##########
modules/file-transferring/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferringTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.network.file.messages.TransferMetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferringTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node node0 = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = node0.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ File file1 = randomFile(unitPath, 1024 * 1024 * 10).toFile();
+ File file2 = randomFile(unitPath, 1024 * 1024 * 10 + 1).toFile();
+ File file3 = randomFile(unitPath, 1024 * 1024 * 10 - 1).toFile();
+
+ node0.fileTransferringService().addFileProvider(
+ TransferMetadata.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ Node node1 = cluster.members.get(1);
+ CompletableFuture<Path> download = node1.fileTransferringService().download(
+ node0.nodeName(),
+ TransferMetadataImpl.builder().build()
+ );
+ assertThat(download.thenAccept(path -> assertContentEquals(unitPath, path)), CompletableFutureMatcher.willCompleteSuccessfully());
+ }
+
+ @Test
+ void upload() throws IOException {
+ Node node0 = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = node0.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ File file1 = randomFile(unitPath, 1024 * 1024 * 10).toFile();
+ File file2 = randomFile(unitPath, 1024 * 1024 * 10 + 1).toFile();
+ File file3 = randomFile(unitPath, 1024 * 1024 * 10 - 1).toFile();
+
+ node0.fileTransferringService().addFileProvider(
+ TransferMetadata.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ Node node1 = cluster.members.get(1);
+
+ CompletableFuture<Path> future = new CompletableFuture<>();
+ node1.fileTransferringService().addFileHandler(TransferMetadata.class, ((metadata, uploadedFile) -> {
+ future.complete(uploadedFile);
Review Comment:
It looks like the future is completed with a directory path, not with a file path. It would be nice if the naming reflected this
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}. Exception: {}", recipientConsistentId, e);
+ // todo send error response
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfo info) {
+ FileReceiver receiver = transferIdToReceiver.get(info.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file transfer info: {}", info);
+ } else {
+ receiver.receive(info);
+ }
+ }
+
+ private void processFileHeader(FileHeader header) {
+ FileReceiver receiver = transferIdToReceiver.get(header.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file header: {}", header);
+ } else {
+ receiver.receive(header);
+ }
+ }
+
+ private void processChunkedFile(ChunkedFile chunkedFile) {
+ FileReceiver receiver = transferIdToReceiver.get(chunkedFile.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for chunked file: {}", chunkedFile);
+ } else {
+ receiver.receive(chunkedFile);
+ }
+ }
+
+ private CompletableFuture<FileReceiver> createFileReceiver(UUID transferId) {
+ try {
Review Comment:
The method is completely synchronous, why does it return a future?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.Executors.newSingleThreadExecutor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+ private final int chunkSize;
+ private final RateLimiter rateLimiter;
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final Queue<FileTransferringRequest> filesToSend = new ConcurrentLinkedQueue<>();
+
+ private final ExecutorService executorService = newSingleThreadExecutor(
+ new NamedThreadFactory("FileSenderExecutor", LOG)
+ );
+
+ FileSender(
+ int chunkSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send
+ ) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ FileTransferringRequest request = new FileTransferringRequest(receiverConsistentId, id, files, chunkSize);
+ filesToSend.add(request);
+ executorService.submit(this::processQueue);
+ return request.result();
+ }
+
+ /**
+ * Processes the queue of files to be sent.
+ */
+ private void processQueue() {
+ while (!filesToSend.isEmpty()) {
+ FileTransferringRequest request = filesToSend.peek();
+ try {
+ FileTransferringMessagesStream stream = request.messagesStream;
+ while (stream.hasNextMessage() && rateLimiter.tryAcquire()) {
+ send.apply(request.receiverConsistentId, stream.nextMessage())
+ .whenComplete((res, e) -> {
+ if (e != null) {
+ request.complete(e);
+ }
+ rateLimiter.release();
+ });
+ }
+ if (!stream.hasNextMessage()) {
+ request.complete();
+ filesToSend.remove();
+ }
+ } catch (Exception e) {
+ request.complete(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ executorService.shutdown();
Review Comment:
Please use `IgniteUtils#shutdownAndAwaitTermination()`
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}. Exception: {}", recipientConsistentId, e);
+ // todo send error response
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfo info) {
+ FileReceiver receiver = transferIdToReceiver.get(info.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file transfer info: {}", info);
+ } else {
+ receiver.receive(info);
+ }
+ }
+
+ private void processFileHeader(FileHeader header) {
+ FileReceiver receiver = transferIdToReceiver.get(header.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file header: {}", header);
+ } else {
+ receiver.receive(header);
+ }
+ }
+
+ private void processChunkedFile(ChunkedFile chunkedFile) {
+ FileReceiver receiver = transferIdToReceiver.get(chunkedFile.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for chunked file: {}", chunkedFile);
+ } else {
+ receiver.receive(chunkedFile);
+ }
+ }
+
+ private CompletableFuture<FileReceiver> createFileReceiver(UUID transferId) {
+ try {
+ Path directory = Files.createDirectory(tempDirectory.resolve(transferId.toString()));
+ FileReceiver receiver = new FileReceiver(directory);
+ transferIdToReceiver.put(transferId, receiver);
+ return completedFuture(receiver);
+ } catch (IOException e) {
+ return failedFuture(e);
+ }
+ }
+
+ private CompletableFuture<Path> handleFileTransferring(UUID transferId, FileReceiver fileReceiver) {
+ return fileReceiver.result()
+ .whenComplete((v, e) -> {
+ transferIdToReceiver.remove(transferId);
+ try {
+ fileReceiver.close();
+ } catch (Exception ex) {
+ LOG.error("Failed to close file receiver: {}. Exception: {}", fileReceiver, ex);
+ }
+ if (e != null) {
+ LOG.error("Failed to receive file. Id: {}. Exception: {}", transferId, e);
+ deleteDirectoryIfExists(fileReceiver.dir());
+ }
+ });
+ }
+
+ private static void deleteDirectoryIfExists(Path directory) {
+ try {
+ FilesUtils.deleteDirectoryIfExists(directory);
+ } catch (IOException e) {
+ LOG.error("Failed to delete directory: {}. Exception: {}", directory, e);
+ }
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(Stream.concat(
+ transferIdToReceiver.values().stream(),
+ Stream.of(fileSender)
+ ));
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileProvider<TransferMetadata>) provider
+ );
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileHandler(
+ Class<M> metadata,
+ FileHandler<M> handler
+ ) {
+ metadataToHandler.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileHandler<TransferMetadata>) handler
+ );
+ }
+
+ @Override
+ public CompletableFuture<Path> download(String node, TransferMetadata transferMetadata) {
+ UUID transferId = UUID.randomUUID();
+ FileDownloadRequest message = FileDownloadRequestImpl.builder()
Review Comment:
We usually use the corresponding factory to instantiate a builder
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}. Exception: {}", recipientConsistentId, e);
+ // todo send error response
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfo info) {
+ FileReceiver receiver = transferIdToReceiver.get(info.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file transfer info: {}", info);
+ } else {
+ receiver.receive(info);
+ }
+ }
+
+ private void processFileHeader(FileHeader header) {
+ FileReceiver receiver = transferIdToReceiver.get(header.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file header: {}", header);
+ } else {
+ receiver.receive(header);
+ }
+ }
+
+ private void processChunkedFile(ChunkedFile chunkedFile) {
+ FileReceiver receiver = transferIdToReceiver.get(chunkedFile.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for chunked file: {}", chunkedFile);
+ } else {
+ receiver.receive(chunkedFile);
+ }
+ }
+
+ private CompletableFuture<FileReceiver> createFileReceiver(UUID transferId) {
+ try {
+ Path directory = Files.createDirectory(tempDirectory.resolve(transferId.toString()));
+ FileReceiver receiver = new FileReceiver(directory);
+ transferIdToReceiver.put(transferId, receiver);
+ return completedFuture(receiver);
+ } catch (IOException e) {
+ return failedFuture(e);
+ }
+ }
+
+ private CompletableFuture<Path> handleFileTransferring(UUID transferId, FileReceiver fileReceiver) {
+ return fileReceiver.result()
+ .whenComplete((v, e) -> {
+ transferIdToReceiver.remove(transferId);
+ try {
+ fileReceiver.close();
+ } catch (Exception ex) {
+ LOG.error("Failed to close file receiver: {}. Exception: {}", fileReceiver, ex);
+ }
+ if (e != null) {
+ LOG.error("Failed to receive file. Id: {}. Exception: {}", transferId, e);
+ deleteDirectoryIfExists(fileReceiver.dir());
+ }
+ });
+ }
+
+ private static void deleteDirectoryIfExists(Path directory) {
+ try {
+ FilesUtils.deleteDirectoryIfExists(directory);
+ } catch (IOException e) {
+ LOG.error("Failed to delete directory: {}. Exception: {}", directory, e);
+ }
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(Stream.concat(
+ transferIdToReceiver.values().stream(),
+ Stream.of(fileSender)
+ ));
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileProvider<TransferMetadata>) provider
+ );
+ }
+
+ @Override
+ public <M extends TransferMetadata> void addFileHandler(
+ Class<M> metadata,
+ FileHandler<M> handler
+ ) {
+ metadataToHandler.put(
+ metadata.getAnnotation(Transferable.class).value(),
+ (FileHandler<TransferMetadata>) handler
+ );
+ }
+
+ @Override
+ public CompletableFuture<Path> download(String node, TransferMetadata transferMetadata) {
+ UUID transferId = UUID.randomUUID();
+ FileDownloadRequest message = FileDownloadRequestImpl.builder()
+ .transferId(transferId)
+ .metadata(transferMetadata)
+ .build();
+
+ return createFileReceiver(message.transferId())
+ .thenCompose(fileReceiver -> messagingService.invoke(node, FILE_TRANSFERRING_CHANNEL, message, Long.MAX_VALUE)
Review Comment:
Why is `MAX_VALUE` used as a timeout? This operation should probably be fast
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final long fileSize;
+ private final RandomAccessFile raf;
+ private final Queue<ChunkedFile> chunks = new PriorityQueue<>(ChunkedFile.COMPARATOR);
+
+ ChunkedFileWriter(Path path, long fileSize) throws FileNotFoundException {
+ this.raf = new RandomAccessFile(path.toFile(), "rw");
Review Comment:
Is 'r' really needed here?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
Review Comment:
I suggest renaming the method to something like `sendFiles()` to make sure there is no confusion with `MessagingService#send()`
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringMessagesStream.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.ChunkedFileImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileHeaderImpl;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoImpl;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferringMessagesStream implements ManuallyCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ private final AtomicReference<FileTransferInfo> fileTransferInfo = new AtomicReference<>();
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param filesToSend the files to send. Must not be empty.
+ * @param chunkSize the size of the chunks to send. Must be positive.
+ */
+ FileTransferringMessagesStream(
+ UUID transferId,
+ List<File> filesToSend,
+ int chunkSize
+ ) {
+ if (filesToSend.isEmpty()) {
+ throw new IllegalArgumentException("Files to send cannot be empty.");
+ }
+
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive.");
+ }
+
+ this.transferId = transferId;
+ this.filesToSend = new LinkedList<>(filesToSend);
+ this.chunkSize = chunkSize;
+ this.fileTransferInfo.set(fileTransferInfo());
+ }
+
+ /**
+ * Returns true if there are more messages to send.
+ *
+ * @return true if there are more messages to send.
+ */
+ boolean hasNextMessage() throws IOException {
+ // check that the stream is not closed.
+ if (closed.get()) {
+ return false;
+ } else {
+ // check that there are more messages to send.
+ // 1. there is a file transfer info message to send.
+ // 2. there are files to send.
+ // 3. there is a current file to send.
+ return fileTransferInfo.get() != null || !filesToSend.isEmpty() || (currFile != null && !currFile.isFinished());
+ }
+ }
+
+ /**
+ * Returns the next message to send.
+ *
+ * @return the next message to send.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if there are no more messages to send.
+ */
+ NetworkMessage nextMessage() throws IOException {
+ if (!hasNextMessage()) {
+ throw new IllegalStateException("There are no more messages to send.");
+ }
+
+ FileTransferInfo info = fileTransferInfo.getAndSet(null);
+ if (info != null) {
+ return info;
+ } else {
+ if (currFile == null || currFile.isFinished()) {
+ openNextFile();
+ return header();
+ } else {
+ return nextChunk();
+ }
+ }
+ }
+
+ private FileTransferInfo fileTransferInfo() {
+ return FileTransferInfoImpl.builder()
+ .transferId(transferId)
+ .filesCount(filesToSend.size())
+ .build();
+ }
+
+ /**
+ * Returns the header of the current file to send.
+ */
+ private FileHeader header() throws IOException {
+ assert currFile != null : "Current file is null.";
+
+ return FileHeaderImpl.builder()
+ .transferId(transferId)
+ .fileName(currFile.fileName())
+ .fileSize(currFile.length())
+ .build();
+ }
+
+ /**
+ * Returns the next chunk of the current file. Throws an exception if the current file is finished.
+ *
+ * @return the next chunk of the current file.
+ * @throws IOException if an I/O error occurs.
+ * @throws IllegalStateException if the current file is finished.
+ */
+ private ChunkedFile nextChunk() throws IOException {
+ assert currFile != null : "Current file is null.";
+ assert !currFile.isFinished() : "Current file is finished.";
+
+ return ChunkedFileImpl.builder()
+ .transferId(transferId)
+ .fileName(currFile.fileName())
+ .offset(currFile.offset())
+ .data(currFile.readNextChunk())
+ .build();
+ }
+
+ private void openNextFile() throws IOException {
Review Comment:
The method promises to just open next file, but it also closes the previous one. It should be renamed (how about `switchToNextFile()`)?
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/messages/ChunkedFile.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import java.util.Comparator;
+import java.util.UUID;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Chunked file.
+ */
+@Transferable(FileTransferringMessageType.CHUNKED_FILE)
+public interface ChunkedFile extends NetworkMessage {
Review Comment:
I suggest renaming it to `FileChunk`
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.util.IgniteUtils;
+
+class FileReceiver implements ManuallyCloseable {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<Path> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileReceiver(Path dir) {
+ this.dir = dir;
+ }
+
+ void receive(FileTransferInfo info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+ filesCount.set(info.filesCount());
+ }
+
+ void receive(FileHeader header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> receive0(header));
+ }
+
+ void receive(ChunkedFile chunkedFile) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(chunkedFile.fileName(), () -> receive0(chunkedFile));
+ }
+
+ private void receive0(FileHeader header) {
+ try {
+ Path path = Files.createFile(dir.resolve(header.fileName()));
Review Comment:
If I'm not mistaken, this method is called from a MessagingService incoming thread (which is one per Ignite instance), but it can do I/O and/or block on a lock. This might cause starvation.
I suggest calling `FileReceiver` methods in an explicit thread pool.
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
Review Comment:
Why is it an error if a provider returned an empty list of files?
##########
modules/network-api/src/main/java/org/apache/ignite/network/MessagingService.java:
##########
@@ -76,6 +76,26 @@ default CompletableFuture<Void> send(ClusterNode recipient, NetworkMessage msg)
*/
CompletableFuture<Void> send(ClusterNode recipient, ChannelType channelType, NetworkMessage msg);
+ /**
+ * Tries to send the given message via specified channel asynchronously to the specific cluster member.
+ *
+ * <p>Guarantees:
+ * <ul>
+ * <li>Messages send to same receiver will be delivered in the same order as they were sent;</li>
+ * <li>If a message N has been successfully delivered to a member implies that all messages to same receiver
+ * preceding N have also been successfully delivered.</li>
+ * </ul>
+ *
+ * <p>Please note that the guarantees only work for same (sender, receiver) pairs. That is, if A sends m1 and m2
+ * to B, then the guarantees are maintained. If, on the other hand, A sends m1 to B and m2 to C, then no guarantees
+ * exist.
+ *
+ * @param recipientConsistentId Consistent ID of the recipient of the message.
+ * @param msg Message which should be delivered.
+ * @return Future of the send operation.
+ */
+ CompletableFuture<Void> send(String recipientConsistentId, ChannelType channelType, NetworkMessage msg);
Review Comment:
Documentation about `ChannelType` is missing.
Also, it's not clear what will happen when using old `send()` without this parameter. Some default channel wlll be used? No channel will be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1281718453
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ FileDownloadResponse response = FileDownloadResponseImpl.builder()
+ .build();
+
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> send(senderConsistentId, message.transferId(), files));
+ })
+ .thenCompose(it -> it);
+ }
+
+ private CompletableFuture<Void> send(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}. Exception: {}", recipientConsistentId, e);
+ // todo send error response
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfo info) {
+ FileReceiver receiver = transferIdToReceiver.get(info.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file transfer info: {}", info);
+ } else {
+ receiver.receive(info);
+ }
+ }
+
+ private void processFileHeader(FileHeader header) {
+ FileReceiver receiver = transferIdToReceiver.get(header.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for file header: {}", header);
+ } else {
+ receiver.receive(header);
+ }
+ }
+
+ private void processChunkedFile(ChunkedFile chunkedFile) {
+ FileReceiver receiver = transferIdToReceiver.get(chunkedFile.transferId());
+ if (receiver == null) {
+ LOG.warn("File receiver is not found for chunked file: {}", chunkedFile);
+ } else {
+ receiver.receive(chunkedFile);
+ }
+ }
+
+ private CompletableFuture<FileReceiver> createFileReceiver(UUID transferId) {
+ try {
Review Comment:
Just to handle exceptions in the "future" style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1283162381
##########
modules/file-transferring/src/main/java/org/apache/ignite/internal/network/file/FileTransferringServiceImpl.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.messages.ChunkedFile;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponseImpl;
+import org.apache.ignite.internal.network.file.messages.FileHeader;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfo;
+import org.apache.ignite.internal.network.file.messages.FileTransferringMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequestImpl;
+import org.apache.ignite.internal.network.file.messages.TransferMetadata;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Implementation of {@link FileTransferringService}.
+ */
+public class FileTransferringServiceImpl implements FileTransferringService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferringServiceImpl.class);
+
+ private static final int CHUNK_SIZE = 1024 * 1024;
+ private static final int CONCURRENT_REQUESTS = 10;
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ private final Path tempDirectory;
+
+ private final FileSender fileSender;
+
+ private final Map<UUID, FileReceiver> transferIdToReceiver = new ConcurrentHashMap<>();
+ private final Map<Short, FileProvider<TransferMetadata>> metadataToProvider = new ConcurrentHashMap<>();
+ private final Map<Short, FileHandler<TransferMetadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferringServiceImpl(MessagingService messagingService, Path tempDirectory) {
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = new FileSender(CHUNK_SIZE, new RateLimiter(CONCURRENT_REQUESTS), (recipientConsistentId, message) -> {
+ return messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ });
+ }
+
+ @Override
+ public void start() {
+ messagingService.addMessageHandler(FileTransferringMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message);
+ } else if (message instanceof FileTransferInfo) {
+ processFileTransferInfo((FileTransferInfo) message);
+ } else if (message instanceof FileHeader) {
+ processFileHeader((FileHeader) message);
+ } else if (message instanceof ChunkedFile) {
+ processChunkedFile((ChunkedFile) message);
+ }
+ });
+ }
+
+ private void processUploadRequest(FileUploadRequest message) {
+ createFileReceiver(message.transferId())
+ .thenCompose(receiver -> handleFileTransferring(message.transferId(), receiver))
+ .thenCompose(path -> {
+ return metadataToHandler.get(message.metadata().messageType()).handleUpload(message.metadata(), path)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to handle file upload. Metadata: {}", message.metadata(), e);
+ }
+
+ deleteDirectoryIfExists(path);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ metadataToProvider.get(message.metadata().messageType()).files(message.metadata())
+ .handle((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ // todo send error response
+ return completedFuture(null);
+ }
+
+ if (files.isEmpty()) {
+ LOG.error("No files found for download. Metadata: {}", message.metadata());
Review Comment:
I can't imagine a situation where a provider can return an empty list and it's a valid case. Anyway, we can change it in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286696060
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
Review Comment:
No, it's called from org.apache.ignite.internal.network.file.FileReceiver#executorService
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1287086418
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ private final Map<String, Set<UUID>> senderConsistentIdToTransferIds = new ConcurrentHashMap<>();
+
+ private final RefCountedObjectPool<String, ReentrantLock> senderConsistentIdToLock = new RefCountedObjectPool<>();
+
+ FileReceiver(String nodeName, int threadPoolSize) {
+ this.executorService = new ThreadPoolExecutor(
Review Comment:
No. All methods of `Executors` create a ThreadPoolExecutor with non-zero `corePoolSize`. I'm not sure that we need to keep threads for file transfers. It doesn't look like a frequent operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288234102
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
Review Comment:
We call it on the receiver when one node uploads files to another node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288268310
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
+ fileReceiver.receiveFileTransferInfo(info);
+ }
+
+ private void processFileHeader(FileHeaderMessage header) {
+ fileReceiver.receiveFileHeader(header);
+ }
+
+ private void processFileChunk(FileChunkMessage fileChunk) {
+ fileReceiver.receiveFileChunk(fileChunk);
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Metadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileProvider<Metadata>) provider;
+ }
+ );
+ }
+
+ @Override
+ public <M extends Metadata> void addFileHandler(
+ Class<M> metadata,
+ FileConsumer<M> handler
+ ) {
+ metadataToHandler.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileConsumer<Metadata>) handler;
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<File>> download(String nodeConsistentId, Metadata metadata) {
+ return completedFuture(UUID.randomUUID())
+ .thenApply(transferId -> factory.fileDownloadRequest()
+ .transferId(transferId)
+ .metadata(metadata)
+ .build())
+ .thenCompose(message -> {
+ Path directory = createTransferDirectory(message.transferId());
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(nodeConsistentId, message.transferId(), directory);
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message,
+ RESPONSE_TIMEOUT)
+ .thenApply(FileDownloadResponse.class::cast)
+ .thenCompose(response -> {
+ if (response.error() != null) {
+ return failedFuture(new FileTransferException(response.error()));
+ } else {
+ return handler.result();
+ }
+ });
+ });
+ }
+
+ @Override
+ public CompletableFuture<Void> upload(String nodeConsistentId, Metadata metadata) {
+ return supplyAsync(() -> getFileProvider(metadata))
+ .thenCompose(provider -> provider.files(metadata))
+ .thenCompose(files -> {
+ FileUploadRequest message = factory.fileUploadRequest()
+ .metadata(metadata)
+ .build();
+
+ if (files.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
+ } else {
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message, RESPONSE_TIMEOUT)
+ .thenApply(FileUploadResponse.class::cast)
+ .thenCompose(response -> sendFiles(nodeConsistentId, response.transferId(), files));
+ }
+ });
+ }
+
+ private static short getMessageType(Class<?> metadata) {
Review Comment:
We do the same thing when registering `NetworkMessageHandler` (`org.apache.ignite.network.AbstractMessagingService#addMessageHandler`). The problem with numeric message types is we can't validate them at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286704342
##########
modules/core/src/main/java/org/apache/ignite/internal/util/FilesUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * Files utilities.
+ */
+public class FilesUtils {
Review Comment:
I think we already have `org.apache.ignite.internal.util.IgniteUtils#deleteIfExists` and `org.apache.ignite.internal.util.IgniteUtils#deleteIfExistsThrowable`. What's the reason for adding a literal copy? Seems very confusing to me.
Just fix the `deleteIfExistsThrowable` and that's it.
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/FileContentMatcher.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import static org.hamcrest.CoreMatchers.is;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+
+/**
+ * {@link TypeSafeMatcher} for matching file content.
+ */
+public class FileContentMatcher extends TypeSafeMatcher<File> {
+
+ /** Matcher to forward the result of the completable future. */
+ private final Matcher<byte[]> matcher;
+
+ private FileContentMatcher(Matcher<byte[]> matcher) {
+ this.matcher = matcher;
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param file File to match.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(File file) {
+ try {
+ return hasContent(is(Files.readAllBytes(file.toPath())));
Review Comment:
`org.hamcrest.Matchers#equalTo` would fit better than `is`
##########
modules/core/src/testFixtures/java/org/apache/ignite/internal/testframework/matchers/FileContentMatcher.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.testframework.matchers;
+
+import static org.hamcrest.CoreMatchers.is;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+
+/**
+ * {@link TypeSafeMatcher} for matching file content.
+ */
+public class FileContentMatcher extends TypeSafeMatcher<File> {
+
+ /** Matcher to forward the result of the completable future. */
+ private final Matcher<byte[]> matcher;
+
+ private FileContentMatcher(Matcher<byte[]> matcher) {
+ this.matcher = matcher;
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param file File to match.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(File file) {
+ try {
+ return hasContent(is(Files.readAllBytes(file.toPath())));
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+
+ /**
+ * Creates a matcher for matching file content.
+ *
+ * @param matcher Matcher for matching file content.
+ * @return Matcher for matching file content.
+ */
+ public static FileContentMatcher hasContent(Matcher<byte[]> matcher) {
+ return new FileContentMatcher(matcher);
+ }
+
+ @Override
+ protected boolean matchesSafely(File file) {
+ try {
+ return matcher.matches(Files.readAllBytes(file.toPath()));
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description.appendText("matches content: ").appendDescriptionOf(matcher);
+ }
+
+ @Override
+ protected void describeMismatchSafely(File item, Description mismatchDescription) {
+ try {
+ mismatchDescription.appendText("was ").appendValue(Files.readAllBytes(item.toPath()));
+ } catch (IOException e) {
+ throw new RuntimeException("Could not read file content", e);
+ }
+ }
+}
+
Review Comment:
Please remove this empty line at the end
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration("mock.chunkSize=1024")
+ private FileTransferConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFilesFuture = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+
+ assertThat(
+ downloadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(sortByNames(file1, file2, file3, file4), sortByNames(files));
Review Comment:
Is there an easier way to do this? Such check should probably be inside of `willBe` matcher, not in the future itself. Maybe some "comtainsInAnyOrder" with another composit matcher (hasContent && name matching) would look better
##########
modules/file-transfer/build.gradle:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply from: "$rootDir/buildscripts/java-core.gradle"
+apply from: "$rootDir/buildscripts/publishing.gradle"
+apply from: "$rootDir/buildscripts/java-junit5.gradle"
+apply from: "$rootDir/buildscripts/java-integration-test.gradle"
+apply from: "$rootDir/buildscripts/java-test-fixtures.gradle"
+
+dependencies {
+ annotationProcessor project(':ignite-network-annotation-processor')
+
+ implementation project(':ignite-core')
+ implementation project(':ignite-network')
+ implementation project(':ignite-configuration-api')
+ implementation libs.jetbrains.annotations
+
+ testImplementation(testFixtures(project))
+ testImplementation(testFixtures(project(':ignite-core')))
+ testImplementation libs.hamcrest.core
+ testImplementation libs.mockito.core
+ testImplementation libs.mockito.junit
+
+ integrationTestImplementation(testFixtures(project))
+ integrationTestImplementation(testFixtures(project(':ignite-core')))
+ integrationTestImplementation(testFixtures(project(':ignite-network')))
+ integrationTestImplementation(testFixtures(project(':ignite-configuration')))
+ integrationTestImplementation libs.hamcrest.core
+ integrationTestImplementation libs.awaitility
+
+ testFixturesImplementation(testFixtures(project(':ignite-core')))
+ testFixturesImplementation libs.hamcrest.core
+}
+
+description = 'ignite-file-io'
Review Comment:
What?
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration("mock.chunkSize=1024")
+ private FileTransferConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFilesFuture = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+
+ assertThat(
+ downloadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(sortByNames(file1, file2, file3, file4), sortByNames(files));
+ }),
+ willCompleteSuccessfully()
+ );
+ }
+
+ @Test
+ void downloadNonReadableFiles() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, chunkSize - 1);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+
+ assertTrue(file2.setReadable(false));
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> download = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+ assertThat(download, willThrow(FileTransferException.class, "Permission denied"));
+ }
+
+ @Test
+ void downloadFilesWhenProviderThrowsException() {
+ Node targetNode = cluster.members.get(0);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> failedFuture(new RuntimeException("Test exception"))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> download = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+ assertThat(download, willThrowWithCauseOrSuppressed(FileTransferException.class));
+ }
+
+ @Test
+ void downloadFilesWhenDoNotHaveAccessToWrite() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ assertTrue(sourceNode.workDir().toFile().setWritable(false));
+
+ CompletableFuture<List<File>> downloadedFilesFuture = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+
+ assertThat(
+ downloadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(sortByNames(file1, file2, file3, file4), sortByNames(files));
+ }),
+ willThrowWithCauseOrSuppressed(AccessDeniedException.class)
+ );
+ }
+
+ @Test
+ void downloadEmptyFileList() {
+ Node targetNode = cluster.members.get(0);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of())
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFiles = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+ assertThat(downloadedFiles, willThrow(FileTransferException.class, "No files to download"));
+ }
+
+ @Test
+ void upload() throws IOException {
+ Node sourceNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = sourceNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize - 1);
+ File file3 = randomFile(unitPath, chunkSize + 1);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ sourceNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node targetNode = cluster.members.get(1);
+
+ CompletableFuture<List<File>> uploadedFilesFuture = new CompletableFuture<>();
+ targetNode.fileTransferringService().addFileHandler(Metadata.class, ((metadata, uploadedFilesDir) -> {
+ uploadedFilesFuture.complete(uploadedFilesDir);
+ return completedFuture(null);
+ }));
+
+ sourceNode.fileTransferringService().upload(targetNode.nodeName(), MetadataImpl.builder().build());
+
+ assertThat(
+ uploadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(sortByNames(file1, file2, file3, file4), sortByNames(files));
+ }),
+ willCompleteSuccessfully()
+ );
+ }
+
+ @Test
+ void uploadEmptyFileList() {
+ Node sourceNode = cluster.members.get(0);
+
+ sourceNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of())
+ );
+
+ Node targetNode = cluster.members.get(1);
+
+ assertThat(
+ sourceNode.fileTransferringService().upload(targetNode.nodeName(), MetadataImpl.builder().build()),
+ willThrow(FileTransferException.class, "No files to upload")
+ );
+ }
+
+ @Test
+ void uploadNonReadableFiles() throws IOException {
+ Node sourceNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = sourceNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, chunkSize - 1);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+
+ assertTrue(file2.setReadable(false));
+
+ sourceNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3))
+ );
+
+ Node targetNode = cluster.members.get(1);
+
+ CompletableFuture<List<File>> uploadedFilesFuture = new CompletableFuture<>();
+ targetNode.fileTransferringService().addFileHandler(Metadata.class, ((metadata, uploadedFilesDir) -> {
+ uploadedFilesFuture.complete(uploadedFilesDir);
+ return completedFuture(null);
+ }));
+
+ assertThat(
+ sourceNode.fileTransferringService().upload(targetNode.nodeName(), MetadataImpl.builder().build()),
+ willThrow(FileTransferException.class, "Failed to send files to node: ")
+ );
+
+ assertThat(
+ uploadedFilesFuture,
+ willTimeoutIn(1, TimeUnit.SECONDS)
Review Comment:
Usually I don't like tests that would spend a second on a simple assertion. Is there a better way to check the same thing?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
+ raf.write(chunks.poll().data());
+ }
+
+ if (fileSize > -1 && raf.getFilePointer() > fileSize) {
+ throw new IOException("File size exceeded");
Review Comment:
Should we have a more specific type of exception?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
Review Comment:
This may be a stupid question, but - do we call this method on the sender or on the receiver? I don't understand it from the documentation.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
Review Comment:
What class is responsible for handling "node left" events and cancelling all operations?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ private final Map<String, Set<UUID>> senderConsistentIdToTransferIds = new ConcurrentHashMap<>();
+
+ private final RefCountedObjectPool<String, ReentrantLock> senderConsistentIdToLock = new RefCountedObjectPool<>();
+
+ FileReceiver(String nodeName, int threadPoolSize) {
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-receiver", LOG)
+ );
+ }
+
+ FileTransferMessagesHandler registerTransfer(String senderConsistentId, UUID transferId, Path handlerDir) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ FileTransferMessagesHandler handler = new FileTransferMessagesHandler(handlerDir);
+ transferIdToHandler.put(transferId, handler);
+ senderConsistentIdToTransferIds.compute(senderConsistentId, (k, v) -> {
+ if (v == null) {
+ v = new HashSet<>();
+ }
+ v.add(transferId);
+ return v;
+ });
+ handler.result().whenComplete((files, throwable) -> deregisterTransfer(senderConsistentId, transferId));
+ return handler;
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ private void deregisterTransfer(String senderConsistentId, UUID transferId) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ transferIdToHandler.remove(transferId);
+ Set<UUID> uuids = senderConsistentIdToTransferIds.get(senderConsistentId);
+ uuids.remove(transferId);
+ if (uuids.isEmpty()) {
+ senderConsistentIdToTransferIds.remove(senderConsistentId);
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ void cancelTransfersFromSender(String senderConsistentId) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ new HashSet<>(senderConsistentIdToTransferIds.get(senderConsistentId))
Review Comment:
I'm sorry, but why do you create a new hash set here? Also, why not using a `for ( : )` instead of streams, what are the benefits?
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration("mock.chunkSize=1024")
+ private FileTransferConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFilesFuture = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
+ );
+
+ assertThat(
+ downloadedFilesFuture.thenAccept(files -> {
+ assertContentEquals(sortByNames(file1, file2, file3, file4), sortByNames(files));
+ }),
+ willCompleteSuccessfully()
+ );
+ }
+
+ @Test
+ void downloadNonReadableFiles() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
Review Comment:
Could these be constants? I see these values being the same in many tests
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
Review Comment:
Is this OK that we block the sending thread with the rate limiter? Doesn't look like a good solution to me
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/ItFileTransferTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrowWithCauseOrSuppressed;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.TestCluster.Node;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.network.file.messages.MetadataImpl;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for file transferring.
+ */
+@ExtendWith(ConfigurationExtension.class)
+@ExtendWith(WorkDirectoryExtension.class)
+public class ItFileTransferTest {
+
+ @WorkDirectory
+ private Path workDir;
+
+ @InjectConfiguration("mock.chunkSize=1024")
+ private FileTransferConfiguration configuration;
+
+ private TestCluster cluster;
+
+ @BeforeEach
+ void setUp(TestInfo testInfo) throws InterruptedException {
+ cluster = new TestCluster(2, configuration, workDir, testInfo);
+ cluster.startAwait();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ cluster.shutdown();
+ }
+
+ @Test
+ void download() throws IOException {
+ Node targetNode = cluster.members.get(0);
+
+ String unit = "unit";
+ String version = "1.0.0";
+ Path unitPath = targetNode.workDir().resolve(unit).resolve(version);
+ Files.createDirectories(unitPath);
+
+ int chunkSize = configuration.value().chunkSize();
+ File file1 = randomFile(unitPath, 0);
+ File file2 = randomFile(unitPath, chunkSize + 1);
+ File file3 = randomFile(unitPath, chunkSize * 2);
+ File file4 = randomFile(unitPath, chunkSize * 2);
+
+ targetNode.fileTransferringService().addFileProvider(
+ Metadata.class,
+ req -> completedFuture(List.of(file1, file2, file3, file4))
+ );
+
+ Node sourceNode = cluster.members.get(1);
+ CompletableFuture<List<File>> downloadedFilesFuture = sourceNode.fileTransferringService().download(
+ targetNode.nodeName(),
+ MetadataImpl.builder().build()
Review Comment:
That's not how you receive a builder, please use a factory. That would be a good habit.
I'm actually surprised that `Impl` classes are not package-private. They should be
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesStream.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferMessagesStream implements Iterable<NetworkMessage>, AutoCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ @Nullable
+ private FileTransferInfoMessage fileTransferInfoMessage;
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param filesToSend the files to send. Must not be empty.
+ * @param chunkSize the size of the chunks to send. Must be positive.
+ */
+ FileTransferMessagesStream(
+ UUID transferId,
+ List<File> filesToSend,
+ int chunkSize
+ ) {
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive.");
+ }
+
+ if (filesToSend.isEmpty()) {
+ throw new IllegalArgumentException("Files to send must not be empty.");
+ }
+
+ this.transferId = transferId;
+ this.filesToSend = new LinkedList<>(filesToSend);
Review Comment:
Very interesting, could you comment the reason for this choice?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
Review Comment:
Don't we already have a channel type 1?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
Review Comment:
What about other 3 parameters?
##########
modules/file-transfer/README.md:
##########
@@ -0,0 +1,23 @@
+# Ignite File Transfer
+
+This module provides a way to transfer files between the host and the target.
+
+## Usage
+
+The module provides a `org.apache.ignite.internal.network.file.FileTransferService` class that can be used to transfer files between the
+host and the target.
+
+### Metadata
+For all files, that age going to be transferred, the metadata should be provided.
Review Comment:
```suggestion
For all files, that are going to be transferred, the metadata should be provided.
```
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Startup timeout in seconds. */
+ private static final int STARTUP_TIMEOUT = 3;
+
+ /** Members of the cluster. */
+ final List<Node> members;
+
+ /** Latch that is locked until all members are visible in the topology. */
+ private final CountDownLatch startupLatch;
+
+ /** Node finder. */
+ private final NodeFinder nodeFinder;
+
+ private final FileTransferConfiguration configuration;
+
+ /**
+ * Creates a test cluster with the given amount of members.
+ *
+ * @param numOfNodes Amount of cluster members.
+ * @param testInfo Test info.
+ */
+ TestCluster(int numOfNodes, FileTransferConfiguration configuration, Path workDir, TestInfo testInfo) {
+ this.startupLatch = new CountDownLatch(numOfNodes - 1);
+
+ int initialPort = 3344;
+
+ List<NetworkAddress> addresses = findLocalAddresses(initialPort, initialPort + numOfNodes);
+
+ this.nodeFinder = new StaticNodeFinder(addresses);
+ this.configuration = configuration;
+
+ var isInitial = new AtomicBoolean(true);
+
+ this.members = addresses.stream()
+ .map(addr -> startNode(workDir, testInfo, addr, isInitial.getAndSet(false)))
+ .collect(toUnmodifiableList());
+ }
+
+ /**
+ * Start cluster node.
+ *
+ * @param testInfo Test info.
+ * @param addr Node address.
+ * @param initial Whether this node is the first one.
+ * @return Started cluster node.
+ */
+ private Node startNode(
+ Path workDir, TestInfo testInfo, NetworkAddress addr, boolean initial
+ ) {
+ ClusterService clusterSvc = clusterService(testInfo, addr.port(), nodeFinder);
+
+ if (initial) {
+ clusterSvc.topologyService().addEventHandler(new TopologyEventHandler() {
+ /** {@inheritDoc} */
+ @Override
+ public void onAppeared(ClusterNode member) {
+ startupLatch.countDown();
+ }
+ });
+ }
+
+ try {
+ Path nodeDir = Files.createDirectory(workDir.resolve("node-" + clusterSvc.nodeName()));
+ FileTransferServiceImpl fileTransferringService = new FileTransferServiceImpl(
+ clusterSvc.nodeName(),
+ clusterSvc.topologyService(),
+ clusterSvc.messagingService(),
+ configuration,
+ nodeDir
+ );
+ return new Node(nodeDir, clusterSvc, fileTransferringService);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Starts and waits for the cluster to come up.
+ *
+ * @throws InterruptedException If failed.
+ * @throws AssertionError If the cluster was unable to start in 3 seconds.
+ */
+ void startAwait() throws InterruptedException {
+ members.forEach(Node::start);
+
+ if (!startupLatch.await(STARTUP_TIMEOUT, TimeUnit.SECONDS)) {
+ throw new AssertionError();
+ }
+ }
+
+ /**
+ * Stops the cluster.
+ */
+ void shutdown() throws Exception {
+ for (Node member : members) {
+ member.stop();
+ }
Review Comment:
Can we use "closeAll" here, or something similar? There's a risk of having stale node in case of exception
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
+ private final String fileName;
+ private final RandomAccessFile raf;
+ private final int chunkSize;
+
+ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize) {
+ this.fileName = fileName;
+ this.raf = raf;
+ this.chunkSize = chunkSize;
+ }
+
+ static ChunkedFileReader open(File file, int chunkSize) throws IOException {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ return new ChunkedFileReader(file.getName(), raf, chunkSize);
Review Comment:
I noticed that you often don't add empty lines. Why?
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Startup timeout in seconds. */
+ private static final int STARTUP_TIMEOUT = 3;
+
+ /** Members of the cluster. */
+ final List<Node> members;
+
+ /** Latch that is locked until all members are visible in the topology. */
+ private final CountDownLatch startupLatch;
+
+ /** Node finder. */
+ private final NodeFinder nodeFinder;
+
+ private final FileTransferConfiguration configuration;
+
+ /**
+ * Creates a test cluster with the given amount of members.
+ *
+ * @param numOfNodes Amount of cluster members.
+ * @param testInfo Test info.
+ */
+ TestCluster(int numOfNodes, FileTransferConfiguration configuration, Path workDir, TestInfo testInfo) {
+ this.startupLatch = new CountDownLatch(numOfNodes - 1);
+
+ int initialPort = 3344;
+
+ List<NetworkAddress> addresses = findLocalAddresses(initialPort, initialPort + numOfNodes);
+
+ this.nodeFinder = new StaticNodeFinder(addresses);
+ this.configuration = configuration;
+
+ var isInitial = new AtomicBoolean(true);
+
+ this.members = addresses.stream()
+ .map(addr -> startNode(workDir, testInfo, addr, isInitial.getAndSet(false)))
+ .collect(toUnmodifiableList());
+ }
+
+ /**
+ * Start cluster node.
+ *
+ * @param testInfo Test info.
+ * @param addr Node address.
+ * @param initial Whether this node is the first one.
+ * @return Started cluster node.
+ */
+ private Node startNode(
+ Path workDir, TestInfo testInfo, NetworkAddress addr, boolean initial
+ ) {
+ ClusterService clusterSvc = clusterService(testInfo, addr.port(), nodeFinder);
+
+ if (initial) {
+ clusterSvc.topologyService().addEventHandler(new TopologyEventHandler() {
+ /** {@inheritDoc} */
+ @Override
+ public void onAppeared(ClusterNode member) {
+ startupLatch.countDown();
+ }
+ });
+ }
+
+ try {
+ Path nodeDir = Files.createDirectory(workDir.resolve("node-" + clusterSvc.nodeName()));
+ FileTransferServiceImpl fileTransferringService = new FileTransferServiceImpl(
+ clusterSvc.nodeName(),
+ clusterSvc.topologyService(),
+ clusterSvc.messagingService(),
+ configuration,
+ nodeDir
+ );
+ return new Node(nodeDir, clusterSvc, fileTransferringService);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Starts and waits for the cluster to come up.
+ *
+ * @throws InterruptedException If failed.
+ * @throws AssertionError If the cluster was unable to start in 3 seconds.
+ */
+ void startAwait() throws InterruptedException {
+ members.forEach(Node::start);
+
+ if (!startupLatch.await(STARTUP_TIMEOUT, TimeUnit.SECONDS)) {
+ throw new AssertionError();
Review Comment:
Please add a message
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
+ private final String fileName;
+ private final RandomAccessFile raf;
+ private final int chunkSize;
+
+ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize) {
+ this.fileName = fileName;
+ this.raf = raf;
+ this.chunkSize = chunkSize;
+ }
+
+ static ChunkedFileReader open(File file, int chunkSize) throws IOException {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ return new ChunkedFileReader(file.getName(), raf, chunkSize);
+ }
+
+ byte[] readNextChunk() throws IOException {
+ int toRead = (int) Math.min(chunkSize, length() - offset());
+ byte[] data = new byte[toRead];
+ raf.read(data);
+ return data;
+ }
+
+ public String fileName() {
Review Comment:
So, some methods are public, others are not. Class itself is not public. Maybe there should be no public methods in this class at all?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesStream.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferMessagesStream implements Iterable<NetworkMessage>, AutoCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ @Nullable
+ private FileTransferInfoMessage fileTransferInfoMessage;
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param filesToSend the files to send. Must not be empty.
+ * @param chunkSize the size of the chunks to send. Must be positive.
+ */
+ FileTransferMessagesStream(
+ UUID transferId,
+ List<File> filesToSend,
+ int chunkSize
+ ) {
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive.");
+ }
+
+ if (filesToSend.isEmpty()) {
+ throw new IllegalArgumentException("Files to send must not be empty.");
+ }
+
+ this.transferId = transferId;
+ this.filesToSend = new LinkedList<>(filesToSend);
+ this.chunkSize = chunkSize;
+ this.fileTransferInfoMessage = fileTransferInfo();
+ }
+
+ /**
+ * Returns true if there are more messages to send.
+ *
+ * @return true if there are more messages to send.
+ */
+ boolean hasNextMessage() throws IOException {
+ // check that the stream is not closed.
Review Comment:
```suggestion
// Check that the stream is not closed.
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileProvider.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Files provider.
+ */
+public interface FileProvider<M extends Metadata> {
+ /**
+ * Returns files for the given metadata. The provider must return non empty list of files.
Review Comment:
```suggestion
* Returns files for the given metadata. The provider must return non-empty list of files.
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
Review Comment:
Is `getFileHandler(metadata)` the same as `handler` here? It's hard to follow, which one is which :)
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
Review Comment:
Javadoc is missing, code without documentation always looks unfinished
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Startup timeout in seconds. */
+ private static final int STARTUP_TIMEOUT = 3;
Review Comment:
How stable is this time?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
Review Comment:
I will ask you one more time here to use more empty lines in your code. I didn't write this comment every time. Code this dense is difficult to read and it doesn't look like the rest of Ignite code. Please fix it everywhere in your code.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ private final Map<String, Set<UUID>> senderConsistentIdToTransferIds = new ConcurrentHashMap<>();
+
+ private final RefCountedObjectPool<String, ReentrantLock> senderConsistentIdToLock = new RefCountedObjectPool<>();
+
+ FileReceiver(String nodeName, int threadPoolSize) {
+ this.executorService = new ThreadPoolExecutor(
Review Comment:
Is there a matching method in `Executors` class?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ private final Map<String, Set<UUID>> senderConsistentIdToTransferIds = new ConcurrentHashMap<>();
+
+ private final RefCountedObjectPool<String, ReentrantLock> senderConsistentIdToLock = new RefCountedObjectPool<>();
+
+ FileReceiver(String nodeName, int threadPoolSize) {
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-receiver", LOG)
+ );
+ }
+
+ FileTransferMessagesHandler registerTransfer(String senderConsistentId, UUID transferId, Path handlerDir) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ FileTransferMessagesHandler handler = new FileTransferMessagesHandler(handlerDir);
+ transferIdToHandler.put(transferId, handler);
+ senderConsistentIdToTransferIds.compute(senderConsistentId, (k, v) -> {
+ if (v == null) {
+ v = new HashSet<>();
+ }
+ v.add(transferId);
+ return v;
+ });
+ handler.result().whenComplete((files, throwable) -> deregisterTransfer(senderConsistentId, transferId));
+ return handler;
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ private void deregisterTransfer(String senderConsistentId, UUID transferId) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ transferIdToHandler.remove(transferId);
+ Set<UUID> uuids = senderConsistentIdToTransferIds.get(senderConsistentId);
+ uuids.remove(transferId);
+ if (uuids.isEmpty()) {
+ senderConsistentIdToTransferIds.remove(senderConsistentId);
+ }
Review Comment:
In previous method you use `compute`, but here you use get/remove. What's the reason for discrepancy?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
+ fileReceiver.receiveFileTransferInfo(info);
+ }
+
+ private void processFileHeader(FileHeaderMessage header) {
+ fileReceiver.receiveFileHeader(header);
+ }
+
+ private void processFileChunk(FileChunkMessage fileChunk) {
+ fileReceiver.receiveFileChunk(fileChunk);
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Metadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileProvider<Metadata>) provider;
+ }
+ );
+ }
+
+ @Override
+ public <M extends Metadata> void addFileHandler(
+ Class<M> metadata,
+ FileConsumer<M> handler
+ ) {
+ metadataToHandler.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileConsumer<Metadata>) handler;
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<File>> download(String nodeConsistentId, Metadata metadata) {
+ return completedFuture(UUID.randomUUID())
+ .thenApply(transferId -> factory.fileDownloadRequest()
+ .transferId(transferId)
+ .metadata(metadata)
+ .build())
+ .thenCompose(message -> {
+ Path directory = createTransferDirectory(message.transferId());
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(nodeConsistentId, message.transferId(), directory);
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message,
+ RESPONSE_TIMEOUT)
+ .thenApply(FileDownloadResponse.class::cast)
+ .thenCompose(response -> {
+ if (response.error() != null) {
+ return failedFuture(new FileTransferException(response.error()));
+ } else {
+ return handler.result();
+ }
+ });
+ });
+ }
+
+ @Override
+ public CompletableFuture<Void> upload(String nodeConsistentId, Metadata metadata) {
+ return supplyAsync(() -> getFileProvider(metadata))
+ .thenCompose(provider -> provider.files(metadata))
+ .thenCompose(files -> {
+ FileUploadRequest message = factory.fileUploadRequest()
+ .metadata(metadata)
+ .build();
+
+ if (files.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
+ } else {
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message, RESPONSE_TIMEOUT)
+ .thenApply(FileUploadResponse.class::cast)
+ .thenCompose(response -> sendFiles(nodeConsistentId, response.transferId(), files));
+ }
+ });
+ }
+
+ private static short getMessageType(Class<?> metadata) {
+ Transferable annotation = metadata.getAnnotation(Transferable.class);
+ if (annotation == null) {
+ throw new IllegalArgumentException("Class " + metadata.getName() + " is not annotated with @Transferable");
+ }
+ return annotation.value();
+ }
+
+ private <M extends Metadata> FileProvider<M> getFileProvider(M metadata) {
+ FileProvider<Metadata> provider = metadataToProvider.get(metadata.messageType());
+ if (provider == null) {
+ throw new FileProviderNotFoundException(metadata.getClass());
+ } else {
+ return (FileProvider<M>) provider;
+ }
+ }
+
+ private <M extends Metadata> FileConsumer<M> getFileHandler(M metadata) {
+ FileConsumer<Metadata> handler = metadataToHandler.get(metadata.messageType());
+ if (handler == null) {
+ throw new FileHandlerNotFoundException(metadata.getClass());
+ } else {
+ return (FileConsumer<M>) handler;
+ }
+ }
+
+ private Path createTransferDirectory(UUID transferId) {
+ try {
+ return Files.createDirectories(tempDirectory.resolve(transferId.toString()));
Review Comment:
Why do we use temp directories? Can't we ask handler for the destination?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+
+/**
+ * Handler for the uploaded file.
+ */
+public interface FileConsumer<M extends Metadata> {
+ /**
+ * Handles the uploaded files.
+ *
+ * @param metadata The metadata of the uploaded file.
+ * @param uploadedFiles The temporary files that were uploaded. These files will be deleted after the method returns.
Review Comment:
The "temporary files" thing must be clarified. Does the uploader just write the data into arbitrary temp directory?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/exception/FileProviderNotFoundException.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.exception;
+
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.lang.ErrorGroups.Network;
+import org.apache.ignite.lang.IgniteException;
+
+/**
+ * Exception thrown when file provider is not found for a metadata message.
+ */
+public class FileProviderNotFoundException extends IgniteException {
+ /**
+ * Constructor.
+ *
+ * @param message Message.
+ */
+ public FileProviderNotFoundException(Class<? extends Metadata> message) {
+ super(Network.FILE_PROVIDER_NOT_FOUND_ERR, "File provider not found for message: " + message.getName());
Review Comment:
How is this a public exception for the user? Missing handler is a bug in the product, not an API misuse. Please remove these error codes from the "api" module
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
Review Comment:
Interesting, why not just inline these single-liners?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileReceiver.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.RefCountedObjectPool;
+
+/**
+ * File receiver.
+ */
+class FileReceiver implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileReceiver.class);
+
+ private final ExecutorService executorService;
+
+ private final Map<UUID, FileTransferMessagesHandler> transferIdToHandler = new ConcurrentHashMap<>();
+
+ private final Map<String, Set<UUID>> senderConsistentIdToTransferIds = new ConcurrentHashMap<>();
+
+ private final RefCountedObjectPool<String, ReentrantLock> senderConsistentIdToLock = new RefCountedObjectPool<>();
+
+ FileReceiver(String nodeName, int threadPoolSize) {
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-receiver", LOG)
+ );
+ }
+
+ FileTransferMessagesHandler registerTransfer(String senderConsistentId, UUID transferId, Path handlerDir) {
+ ReentrantLock lock = senderConsistentIdToLock.acquire(senderConsistentId, ignored -> new ReentrantLock());
+ lock.lock();
+ try {
+ FileTransferMessagesHandler handler = new FileTransferMessagesHandler(handlerDir);
+ transferIdToHandler.put(transferId, handler);
+ senderConsistentIdToTransferIds.compute(senderConsistentId, (k, v) -> {
+ if (v == null) {
+ v = new HashSet<>();
+ }
+ v.add(transferId);
+ return v;
+ });
+ handler.result().whenComplete((files, throwable) -> deregisterTransfer(senderConsistentId, transferId));
+ return handler;
+ } finally {
+ lock.unlock();
+ }
+ }
Review Comment:
Where are empty lines? This is very hard to read, and it violates code style guidelines. Please format your code properly
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
+ fileReceiver.receiveFileTransferInfo(info);
+ }
+
+ private void processFileHeader(FileHeaderMessage header) {
+ fileReceiver.receiveFileHeader(header);
+ }
+
+ private void processFileChunk(FileChunkMessage fileChunk) {
+ fileReceiver.receiveFileChunk(fileChunk);
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Metadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileProvider<Metadata>) provider;
+ }
+ );
+ }
+
+ @Override
+ public <M extends Metadata> void addFileHandler(
+ Class<M> metadata,
+ FileConsumer<M> handler
+ ) {
+ metadataToHandler.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileConsumer<Metadata>) handler;
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<File>> download(String nodeConsistentId, Metadata metadata) {
+ return completedFuture(UUID.randomUUID())
+ .thenApply(transferId -> factory.fileDownloadRequest()
+ .transferId(transferId)
+ .metadata(metadata)
+ .build())
+ .thenCompose(message -> {
+ Path directory = createTransferDirectory(message.transferId());
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(nodeConsistentId, message.transferId(), directory);
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message,
+ RESPONSE_TIMEOUT)
+ .thenApply(FileDownloadResponse.class::cast)
+ .thenCompose(response -> {
+ if (response.error() != null) {
+ return failedFuture(new FileTransferException(response.error()));
+ } else {
+ return handler.result();
+ }
+ });
+ });
+ }
+
+ @Override
+ public CompletableFuture<Void> upload(String nodeConsistentId, Metadata metadata) {
+ return supplyAsync(() -> getFileProvider(metadata))
+ .thenCompose(provider -> provider.files(metadata))
+ .thenCompose(files -> {
+ FileUploadRequest message = factory.fileUploadRequest()
+ .metadata(metadata)
+ .build();
+
+ if (files.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
+ } else {
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message, RESPONSE_TIMEOUT)
+ .thenApply(FileUploadResponse.class::cast)
+ .thenCompose(response -> sendFiles(nodeConsistentId, response.transferId(), files));
+ }
+ });
+ }
+
+ private static short getMessageType(Class<?> metadata) {
Review Comment:
Maybe we should register handlers by the numeric message types, rather than classes? Cause it looks a bit hacky right now
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/RateLimiter.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+class RateLimiter {
Review Comment:
Why don't you use a semaphore instead?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
Review Comment:
Why not just providing a messaging service here?
##########
modules/file-transfer/src/integrationTest/java/org/apache/ignite/internal/network/file/TestCluster.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.stream.Collectors.toUnmodifiableList;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.utils.ClusterServiceTestUtils.findLocalAddresses;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.ClusterService;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.network.NodeFinder;
+import org.apache.ignite.network.StaticNodeFinder;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.junit.jupiter.api.TestInfo;
+
+/**
+ * Test cluster.
+ */
+public class TestCluster {
+ /** Startup timeout in seconds. */
+ private static final int STARTUP_TIMEOUT = 3;
+
+ /** Members of the cluster. */
+ final List<Node> members;
+
+ /** Latch that is locked until all members are visible in the topology. */
+ private final CountDownLatch startupLatch;
+
+ /** Node finder. */
+ private final NodeFinder nodeFinder;
+
+ private final FileTransferConfiguration configuration;
+
+ /**
+ * Creates a test cluster with the given amount of members.
+ *
+ * @param numOfNodes Amount of cluster members.
+ * @param testInfo Test info.
+ */
+ TestCluster(int numOfNodes, FileTransferConfiguration configuration, Path workDir, TestInfo testInfo) {
+ this.startupLatch = new CountDownLatch(numOfNodes - 1);
+
+ int initialPort = 3344;
+
+ List<NetworkAddress> addresses = findLocalAddresses(initialPort, initialPort + numOfNodes);
+
+ this.nodeFinder = new StaticNodeFinder(addresses);
+ this.configuration = configuration;
+
+ var isInitial = new AtomicBoolean(true);
+
+ this.members = addresses.stream()
+ .map(addr -> startNode(workDir, testInfo, addr, isInitial.getAndSet(false)))
+ .collect(toUnmodifiableList());
+ }
+
+ /**
+ * Start cluster node.
+ *
+ * @param testInfo Test info.
+ * @param addr Node address.
+ * @param initial Whether this node is the first one.
+ * @return Started cluster node.
+ */
+ private Node startNode(
+ Path workDir, TestInfo testInfo, NetworkAddress addr, boolean initial
+ ) {
+ ClusterService clusterSvc = clusterService(testInfo, addr.port(), nodeFinder);
+
+ if (initial) {
+ clusterSvc.topologyService().addEventHandler(new TopologyEventHandler() {
+ /** {@inheritDoc} */
Review Comment:
This comment it not required. I don't mind having it, just wanted to mention that
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
+ fileReceiver.receiveFileTransferInfo(info);
+ }
+
+ private void processFileHeader(FileHeaderMessage header) {
+ fileReceiver.receiveFileHeader(header);
+ }
+
+ private void processFileChunk(FileChunkMessage fileChunk) {
+ fileReceiver.receiveFileChunk(fileChunk);
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Metadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileProvider<Metadata>) provider;
+ }
+ );
+ }
+
+ @Override
+ public <M extends Metadata> void addFileHandler(
+ Class<M> metadata,
+ FileConsumer<M> handler
+ ) {
+ metadataToHandler.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileConsumer<Metadata>) handler;
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<File>> download(String nodeConsistentId, Metadata metadata) {
+ return completedFuture(UUID.randomUUID())
+ .thenApply(transferId -> factory.fileDownloadRequest()
+ .transferId(transferId)
+ .metadata(metadata)
+ .build())
+ .thenCompose(message -> {
+ Path directory = createTransferDirectory(message.transferId());
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(nodeConsistentId, message.transferId(), directory);
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message,
+ RESPONSE_TIMEOUT)
+ .thenApply(FileDownloadResponse.class::cast)
+ .thenCompose(response -> {
+ if (response.error() != null) {
+ return failedFuture(new FileTransferException(response.error()));
+ } else {
+ return handler.result();
+ }
+ });
+ });
+ }
+
+ @Override
+ public CompletableFuture<Void> upload(String nodeConsistentId, Metadata metadata) {
+ return supplyAsync(() -> getFileProvider(metadata))
+ .thenCompose(provider -> provider.files(metadata))
+ .thenCompose(files -> {
+ FileUploadRequest message = factory.fileUploadRequest()
+ .metadata(metadata)
+ .build();
+
+ if (files.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
+ } else {
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message, RESPONSE_TIMEOUT)
+ .thenApply(FileUploadResponse.class::cast)
+ .thenCompose(response -> sendFiles(nodeConsistentId, response.transferId(), files));
+ }
+ });
+ }
+
+ private static short getMessageType(Class<?> metadata) {
+ Transferable annotation = metadata.getAnnotation(Transferable.class);
+ if (annotation == null) {
+ throw new IllegalArgumentException("Class " + metadata.getName() + " is not annotated with @Transferable");
+ }
+ return annotation.value();
+ }
+
+ private <M extends Metadata> FileProvider<M> getFileProvider(M metadata) {
+ FileProvider<Metadata> provider = metadataToProvider.get(metadata.messageType());
+ if (provider == null) {
+ throw new FileProviderNotFoundException(metadata.getClass());
+ } else {
+ return (FileProvider<M>) provider;
+ }
+ }
+
+ private <M extends Metadata> FileConsumer<M> getFileHandler(M metadata) {
+ FileConsumer<Metadata> handler = metadataToHandler.get(metadata.messageType());
+ if (handler == null) {
+ throw new FileHandlerNotFoundException(metadata.getClass());
+ } else {
+ return (FileConsumer<M>) handler;
+ }
+ }
+
+ private Path createTransferDirectory(UUID transferId) {
+ try {
+ return Files.createDirectories(tempDirectory.resolve(transferId.toString()));
+ } catch (IOException e) {
+ throw new FileTransferException("Failed to create transfer directory. Transfer id: {}. Exception: {}" + transferId, e);
+ }
+ }
+
+ private static void deleteDirectoryIfExists(Path directory) {
+ try {
+ FilesUtils.deleteDirectoryIfExists(directory);
+ } catch (IOException e) {
+ LOG.error("Failed to delete directory: {}. Exception: {}", directory, e);
+ }
+ }
+
+ private static Throwable unwrapFileTransferException(Throwable throwable) {
+ Throwable unwrapped = ExceptionUtils.unwrapCause(throwable);
+ if (unwrapped instanceof FileTransferException) {
+ return unwrapped.getCause() == null ? unwrapped : unwrapped.getCause();
Review Comment:
Wow, why so complicated? I would love to have a comment here, or a simpler solution when it comes to exceptions
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
Review Comment:
This approach is very non-intuitive to me. Why did you chose it instead of good old `send.apply(receiverConsistentId, stream.nextMessage())`?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileWriter.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.file.Path;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+
+class ChunkedFileWriter implements ManuallyCloseable {
+ private final RandomAccessFile raf;
+
+ private long fileSize;
+
+ private final Queue<FileChunkMessage> chunks = new PriorityQueue<>(FileChunkMessage.COMPARATOR);
+
+ private ChunkedFileWriter(RandomAccessFile raf, long fileSize) {
+ this.raf = raf;
+ this.fileSize = fileSize;
+ }
+
+ static ChunkedFileWriter open(Path path, long fileSize) throws FileNotFoundException {
+ return new ChunkedFileWriter(new RandomAccessFile(path.toFile(), "rw"), fileSize);
+ }
+
+ void write(FileChunkMessage chunk) throws IOException {
+ chunks.add(chunk);
+
+ while (!chunks.isEmpty() && chunks.peek().offset() == raf.getFilePointer()) {
Review Comment:
Does this mean that this method will be called concurrently with different chunks? Not only the queue is not thread-safe, the algorithm itself is not thread-safe either
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ error.compareAndSet(null, e);
+ }
+ } finally {
+ rateLimiter.release();
+ }
+ });
+ }
+ }
+
+ if (error.get() != null) {
Review Comment:
I think the question was "this error will be set in the future, asynchronously, why do we check it right here?"
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandlerTest.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.network.NetworkMessage;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+@ExtendWith(WorkDirectoryExtension.class)
+class FileTransferMessagesHandlerTest {
+ private static final int CHUNK_SIZE = 1024; // 1 KB
Review Comment:
We know it's 1 KB :)
In every other place there's no such comment, why here?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
+ }
+
+ private void handleFileChunk0(FileChunkMessage fileChunk) {
+ try {
+ ChunkedFileWriter writer = fileNameToWriter.computeIfAbsent(fileChunk.fileName(), this::writer);
+
+ writer.write(fileChunk);
+
+ if (writer.isFinished()) {
+ writer.close();
+ fileNameToWriter.remove(fileChunk.fileName());
+ filesFinished.incrementAndGet();
+
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileTransferError(Throwable error) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer error after result is already done.");
+ }
+ result.completeExceptionally(error);
+ closeAllWriters();
+ }
+
+ private void doInLock(String fileName, Runnable runnable) {
+ Lock lock = fileNameToLock.computeIfAbsent(fileName, k -> new ReentrantLock());
+ lock.lock();
+ try {
+ runnable.run();
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ private void completeIfAllFilesFinished() throws IOException {
+ if (filesFinished.get() == filesCount.get()) {
+ try (Stream<Path> stream = Files.list(dir)) {
+ List<File> files = stream.map(Path::toFile).collect(Collectors.toList());
+ result.complete(files);
+ }
+ }
+ }
+
+ CompletableFuture<List<File>> result() {
+ return result;
+ }
+
+ private void closeAllWriters() {
+ try {
+ IgniteUtils.closeAllManually(fileNameToWriter.values().stream());
+ } catch (Exception e) {
+ throw new IgniteInternalException(e);
+ }
+ }
+
+ private ChunkedFileWriter writer(String fileName) {
+ // set -1 as file size to indicate that file size is unknown and will be set later
Review Comment:
```suggestion
// Set -1 as file size to indicate that file size is unknown and will be set later.
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
+ private final Path dir;
+ private final AtomicInteger filesCount = new AtomicInteger(-1);
+ private final AtomicInteger filesFinished = new AtomicInteger(0);
+ private final CompletableFuture<List<File>> result = new CompletableFuture<>();
+ private final Map<String, ChunkedFileWriter> fileNameToWriter = new ConcurrentHashMap<>();
+ private final Map<String, Lock> fileNameToLock = new ConcurrentHashMap<>();
+
+ FileTransferMessagesHandler(Path dir) {
+ this.dir = dir;
+ }
+
+ void handleFileTransferInfo(FileTransferInfoMessage info) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer info after result is already done.");
+ }
+
+ filesCount.set(info.filesCount());
+
+ try {
+ completeIfAllFilesFinished();
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileHeader(FileHeaderMessage header) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file header after result is already done.");
+ }
+ doInLock(header.fileName(), () -> handleFileHeader0(header));
+ }
+
+ private void handleFileHeader0(FileHeaderMessage header) {
+ ChunkedFileWriter writer = fileNameToWriter.compute(header.fileName(), (k, v) -> {
+ if (v == null) {
+ return writer(header.fileName(), header.fileSize());
+ } else {
+ v.fileSize(header.fileSize());
+ return v;
+ }
+ });
+
+ try {
+ if (writer.isFinished()) {
+ writer.close();
+ filesFinished.incrementAndGet();
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileChunk(FileChunkMessage fileChunk) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received chunked file after result is already done.");
+ }
+ doInLock(fileChunk.fileName(), () -> handleFileChunk0(fileChunk));
+ }
+
+ private void handleFileChunk0(FileChunkMessage fileChunk) {
+ try {
+ ChunkedFileWriter writer = fileNameToWriter.computeIfAbsent(fileChunk.fileName(), this::writer);
+
+ writer.write(fileChunk);
+
+ if (writer.isFinished()) {
+ writer.close();
+ fileNameToWriter.remove(fileChunk.fileName());
+ filesFinished.incrementAndGet();
+
+ completeIfAllFilesFinished();
+ }
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ }
+ }
+
+ void handleFileTransferError(Throwable error) {
+ if (result.isDone()) {
+ throw new IllegalStateException("Received file transfer error after result is already done.");
+ }
+ result.completeExceptionally(error);
+ closeAllWriters();
+ }
+
+ private void doInLock(String fileName, Runnable runnable) {
+ Lock lock = fileNameToLock.computeIfAbsent(fileName, k -> new ReentrantLock());
+ lock.lock();
+ try {
+ runnable.run();
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ private void completeIfAllFilesFinished() throws IOException {
+ if (filesFinished.get() == filesCount.get()) {
+ try (Stream<Path> stream = Files.list(dir)) {
+ List<File> files = stream.map(Path::toFile).collect(Collectors.toList());
+ result.complete(files);
+ }
+ }
+ }
+
+ CompletableFuture<List<File>> result() {
+ return result;
+ }
+
+ private void closeAllWriters() {
+ try {
+ IgniteUtils.closeAllManually(fileNameToWriter.values().stream());
+ } catch (Exception e) {
+ throw new IgniteInternalException(e);
+ }
+ }
+
+ private ChunkedFileWriter writer(String fileName) {
+ // set -1 as file size to indicate that file size is unknown and will be set later
+ return writer(fileName, -1);
+ }
+
+ private ChunkedFileWriter writer(String fileName, long fileSize) {
+ try {
+ Path path = Files.createFile(dir.resolve(fileName));
+ return ChunkedFileWriter.open(path, fileSize);
+ } catch (IOException e) {
+ handleFileTransferError(e);
+ throw new IgniteInternalException(e);
Review Comment:
Please don't do that, throw proper exception instead
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/messages/FileTransferMessageType.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import org.apache.ignite.network.annotations.MessageGroup;
+
+/**
+ * File transfer message types.
+ */
+@MessageGroup(groupType = 200, groupName = "FileTransfer")
+public final class FileTransferMessageType {
+ public static final short FILE_TRANSFER_INFO = 0;
Review Comment:
Please add a link to the class in javadocs for every constant, like we do it in other classes.
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesHandler.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.lang.IgniteInternalException;
+
+class FileTransferMessagesHandler {
Review Comment:
Again, a total lack of comments in package-private classes. Do you assume that the code is trivial and doesn't require any comment at all?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferMessagesStream.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.network.NetworkMessage;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Stream of messages to send files.
+ */
+public class FileTransferMessagesStream implements Iterable<NetworkMessage>, AutoCloseable {
+ private final UUID transferId;
+
+ private final Queue<File> filesToSend;
+
+ private final int chunkSize;
+
+ @Nullable
+ private ChunkedFileReader currFile;
+
+ @Nullable
+ private FileTransferInfoMessage fileTransferInfoMessage;
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Creates a new stream of messages to send files.
+ *
+ * @param transferId the id of the stream.
+ * @param filesToSend the files to send. Must not be empty.
+ * @param chunkSize the size of the chunks to send. Must be positive.
+ */
+ FileTransferMessagesStream(
+ UUID transferId,
+ List<File> filesToSend,
+ int chunkSize
+ ) {
+ if (chunkSize <= 0) {
+ throw new IllegalArgumentException("Chunk size must be positive.");
+ }
+
+ if (filesToSend.isEmpty()) {
+ throw new IllegalArgumentException("Files to send must not be empty.");
+ }
+
+ this.transferId = transferId;
+ this.filesToSend = new LinkedList<>(filesToSend);
+ this.chunkSize = chunkSize;
+ this.fileTransferInfoMessage = fileTransferInfo();
+ }
+
+ /**
+ * Returns true if there are more messages to send.
+ *
+ * @return true if there are more messages to send.
+ */
+ boolean hasNextMessage() throws IOException {
+ // check that the stream is not closed.
Review Comment:
Please check all your comments
##########
modules/file-transfer/src/testFixtures/java/org/apache/ignite/internal/network/file/FileGenerator.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Random;
+
+/**
+ * File generator.
+ */
+public class FileGenerator {
+ /**
+ * Generates a random file with the given content size. The file be deleted on exit.
+ *
+ * @param dir Directory to create the file in.
+ * @param contentSize Size of the file content.
+ * @return Generated file.
+ */
+ public static File randomFile(Path dir, int contentSize) {
+ try {
+ Path tempFile = Files.createTempFile(dir, null, null);
+ byte[] bytes = new byte[contentSize];
+ new Random().nextBytes(bytes);
+ tempFile.toFile().deleteOnExit();
Review Comment:
Once again, what's the reason?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/messages/FileTransferMessageType.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file.messages;
+
+import org.apache.ignite.network.annotations.MessageGroup;
+
+/**
+ * File transfer message types.
+ */
+@MessageGroup(groupType = 200, groupName = "FileTransfer")
+public final class FileTransferMessageType {
+ public static final short FILE_TRANSFER_INFO = 0;
+ public static final short FILE_HEADER = 1;
+ public static final short FILE_CHUNK = 2;
+ public static final short FILE_DOWNLOAD_REQUEST = 3;
+ public static final short FILE_DOWNLOAD_RESPONSE = 4;
+ public static final short FILE_UPLOAD_REQUEST = 5;
+ public static final short FILE_UPLOAD_RESPONSE = 6;
+ public static final short FILE_TRANSFER_ERROR = 7;
+ public static final short METADATA = 8;
+
+ /**
+ * File transferring metadata.
+ */
+ public static final class Metadata {
Review Comment:
What's this class for? I assume you forgot to delete it
##########
modules/file-transfer/src/testFixtures/java/org/apache/ignite/internal/network/file/FileGenerator.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Random;
+
+/**
+ * File generator.
+ */
+public class FileGenerator {
+ /**
+ * Generates a random file with the given content size. The file be deleted on exit.
+ *
+ * @param dir Directory to create the file in.
+ * @param contentSize Size of the file content.
+ * @return Generated file.
+ */
+ public static File randomFile(Path dir, int contentSize) {
Review Comment:
Why do you use both old and new file API in the same signature?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
Review Comment:
Should this be configurable? I guess so
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileReceiverTest.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.network.NetworkMessage;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+@ExtendWith(WorkDirectoryExtension.class)
+class FileReceiverTest {
+ private static final int CHUNK_SIZE = 1024;
+
+ @WorkDirectory
+ private Path workDir;
+
+ @Test
+ void receiveSingleFile() throws IOException {
+ // when a single file is sent
+ List<File> filesToSend = List.of(randomFile(workDir, CHUNK_SIZE));
+ FileReceiver receiver = new FileReceiver("node1", 10);
+ UUID transferId = UUID.randomUUID();
+
+ Path path = Files.createDirectory(workDir.resolve(transferId.toString()));
+ path.toFile().deleteOnExit();
Review Comment:
But why? Work dir should be deleted after tests. If it's not, we should fix the extension itself
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileReceiverTest.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.network.NetworkMessage;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+@ExtendWith(WorkDirectoryExtension.class)
+class FileReceiverTest {
+ private static final int CHUNK_SIZE = 1024;
+
+ @WorkDirectory
+ private Path workDir;
+
+ @Test
+ void receiveSingleFile() throws IOException {
+ // when a single file is sent
Review Comment:
You get the idea
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
Review Comment:
It would be nice to name it "messageFactory", because "factory" itself is too broad of a term
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
Review Comment:
Once again I see this pattern. What's the reason? Writing
```
UUID transferId = UUID.randomUUID();
```
looks absolutely fine. Why would you need to pollute your code with extra paddings? To avoid writing the type explicitly?
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
Review Comment:
```suggestion
/**
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender implements ManuallyCloseable {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
Review Comment:
Is this a spin-loop that will just burn the CPU?
##########
modules/file-transfer/src/test/java/org/apache/ignite/internal/network/file/FileReceiverTest.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static org.apache.ignite.internal.network.file.FileAssertions.assertContentEquals;
+import static org.apache.ignite.internal.network.file.FileGenerator.randomFile;
+import static org.apache.ignite.internal.network.file.FileUtils.sortByNames;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.testframework.WorkDirectory;
+import org.apache.ignite.internal.testframework.WorkDirectoryExtension;
+import org.apache.ignite.network.NetworkMessage;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+@ExtendWith(WorkDirectoryExtension.class)
+class FileReceiverTest {
+ private static final int CHUNK_SIZE = 1024;
+
+ @WorkDirectory
+ private Path workDir;
+
+ @Test
+ void receiveSingleFile() throws IOException {
+ // when a single file is sent
Review Comment:
```suggestion
// When a single file is sent.
```
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
Review Comment:
Threading model here is not clear to me. All "compose" are not "async", how do you guarantee that network thread won't be executing your disk IO operations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288346028
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileSender.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiFunction;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.NetworkMessage;
+
+class FileSender {
+ private static final IgniteLogger LOG = Loggers.forClass(FileSender.class);
+
+ private final int chunkSize;
+
+ private final RateLimiter rateLimiter;
+
+ private final BiFunction<String, NetworkMessage, CompletableFuture<Void>> send;
+
+ private final ExecutorService executorService;
+
+ FileSender(
+ String nodeName,
+ int chunkSize,
+ int threadPoolSize,
+ RateLimiter rateLimiter,
+ BiFunction<String, NetworkMessage, CompletableFuture<Void>> send) {
+ this.send = send;
+ this.chunkSize = chunkSize;
+ this.rateLimiter = rateLimiter;
+ this.executorService = new ThreadPoolExecutor(
+ 0,
+ threadPoolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ NamedThreadFactory.create(nodeName, "file-sender", LOG)
+ );
+ }
+
+ /**
+ * Adds files to the queue to be sent to the receiver.
+ */
+ CompletableFuture<Void> send(String receiverConsistentId, UUID id, List<File> files) {
+ return CompletableFuture.runAsync(() -> send0(receiverConsistentId, id, files), executorService);
+ }
+
+ private void send0(String receiverConsistentId, UUID id, List<File> files) {
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ try (FileTransferMessagesStream stream = new FileTransferMessagesStream(id, files, chunkSize)) {
+ while (stream.hasNextMessage() && error.get() == null && !Thread.currentThread().isInterrupted()) {
+ if (rateLimiter.tryAcquire()) {
+ CompletableFuture.completedFuture(stream.nextMessage())
+ .thenCompose(message -> send.apply(receiverConsistentId, message))
+ .whenComplete((res, e) -> {
+ try {
+ if (e != null) {
+ LOG.error("Failed to send message to node: {}, transfer id: {}. Exception: {}",
+ receiverConsistentId,
+ id,
+ e
+ );
+ error.compareAndSet(null, e);
+ }
+ } finally {
+ rateLimiter.release();
+ }
+ });
+ }
+ }
+
+ if (error.get() != null) {
Review Comment:
Because it doesn't make sense to continue file transfer if there is a failure in one of the files. Currently, the service doesn't provide recovery functionality. If an error happens during a transfer, the transfer must be started from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1286171140
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import org.apache.ignite.internal.close.ManuallyCloseable;
+
+class ChunkedFileReader implements ManuallyCloseable {
+ private final String fileName;
+ private final RandomAccessFile raf;
+ private final int chunkSize;
+
+ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize) {
+ this.fileName = fileName;
+ this.raf = raf;
+ this.chunkSize = chunkSize;
+ }
+
+ static ChunkedFileReader open(File file, int chunkSize) throws IOException {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ return new ChunkedFileReader(file.getName(), raf, chunkSize);
+ }
+
+ byte[] readNextChunk() throws IOException {
+ int toRead = (int) Math.min(chunkSize, raf.length() - raf.getFilePointer());
Review Comment:
What do you mean? `java.io.RandomAccessFile#getFilePointer` returns offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1288495059
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.network.file;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.CompletableFuture.failedFuture;
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.network.configuration.FileTransferConfiguration;
+import org.apache.ignite.internal.network.file.exception.FileHandlerNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileProviderNotFoundException;
+import org.apache.ignite.internal.network.file.exception.FileTransferException;
+import org.apache.ignite.internal.network.file.messages.FileChunkMessage;
+import org.apache.ignite.internal.network.file.messages.FileDownloadRequest;
+import org.apache.ignite.internal.network.file.messages.FileDownloadResponse;
+import org.apache.ignite.internal.network.file.messages.FileHeaderMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferErrorMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferFactory;
+import org.apache.ignite.internal.network.file.messages.FileTransferInfoMessage;
+import org.apache.ignite.internal.network.file.messages.FileTransferMessageType;
+import org.apache.ignite.internal.network.file.messages.FileUploadRequest;
+import org.apache.ignite.internal.network.file.messages.FileUploadResponse;
+import org.apache.ignite.internal.network.file.messages.Metadata;
+import org.apache.ignite.internal.util.ExceptionUtils;
+import org.apache.ignite.internal.util.FilesUtils;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.network.ChannelType;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.MessagingService;
+import org.apache.ignite.network.TopologyEventHandler;
+import org.apache.ignite.network.TopologyService;
+import org.apache.ignite.network.annotations.Transferable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Implementation of {@link FileTransferService}.
+ */
+public class FileTransferServiceImpl implements FileTransferService {
+ private static final IgniteLogger LOG = Loggers.forClass(FileTransferServiceImpl.class);
+
+ private static final ChannelType FILE_TRANSFERRING_CHANNEL = ChannelType.register((short) 1, "FileTransferring");
+
+ private static final long RESPONSE_TIMEOUT = 10_000;
+
+ /**
+ * Topology service.
+ */
+ private final TopologyService topologyService;
+ /**
+ * Cluster service.
+ */
+ private final MessagingService messagingService;
+
+ /**
+ * Temporary directory for saving files.
+ */
+ private final Path tempDirectory;
+
+ /**
+ * File sender.
+ */
+ private final FileSender fileSender;
+
+ /**
+ * File receiver.
+ */
+ private final FileReceiver fileReceiver;
+
+ /**
+ * Map of file providers.
+ */
+ private final Map<Short, FileProvider<Metadata>> metadataToProvider = new ConcurrentHashMap<>();
+
+ /**
+ * Map of file handlers.
+ */
+ private final Map<Short, FileConsumer<Metadata>> metadataToHandler = new ConcurrentHashMap<>();
+
+ /**
+ * File transfer factory.
+ */
+ private final FileTransferFactory factory = new FileTransferFactory();
+
+ /**
+ * Constructor.
+ *
+ * @param messagingService Messaging service.
+ * @param tempDirectory Temporary directory.
+ */
+ public FileTransferServiceImpl(
+ String nodeName,
+ TopologyService topologyService,
+ MessagingService messagingService,
+ FileTransferConfiguration configuration,
+ Path tempDirectory
+ ) {
+ this(
+ topologyService,
+ messagingService,
+ tempDirectory,
+ new FileSender(
+ nodeName,
+ configuration.value().senderThreadPoolSize(),
+ configuration.value().chunkSize(),
+ new RateLimiter(configuration.value().maxConcurrentRequests()),
+ (recipientConsistentId, message) -> messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL,
+ message)
+ ),
+ new FileReceiver(nodeName, configuration.value().receiverThreadPoolSize())
+ );
+ }
+
+ @TestOnly
+ FileTransferServiceImpl(
+ TopologyService topologyService,
+ MessagingService messagingService,
+ Path tempDirectory,
+ FileSender fileSender,
+ FileReceiver fileReceiver
+ ) {
+ this.topologyService = topologyService;
+ this.messagingService = messagingService;
+ this.tempDirectory = tempDirectory;
+ this.fileSender = fileSender;
+ this.fileReceiver = fileReceiver;
+ }
+
+ @Override
+ public void start() {
+ topologyService.addEventHandler(new TopologyEventHandler() {
+ @Override
+ public void onDisappeared(ClusterNode member) {
+ fileReceiver.cancelTransfersFromSender(member.id());
+ }
+ });
+
+ messagingService.addMessageHandler(FileTransferMessageType.class,
+ (message, senderConsistentId, correlationId) -> {
+ if (message instanceof FileDownloadRequest) {
+ processDownloadRequest((FileDownloadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileUploadRequest) {
+ processUploadRequest((FileUploadRequest) message, senderConsistentId, correlationId);
+ } else if (message instanceof FileTransferErrorMessage) {
+ processFileTransferErrorMessage((FileTransferErrorMessage) message);
+ } else if (message instanceof FileTransferInfoMessage) {
+ processFileTransferInfo((FileTransferInfoMessage) message);
+ } else if (message instanceof FileHeaderMessage) {
+ processFileHeader((FileHeaderMessage) message);
+ } else if (message instanceof FileChunkMessage) {
+ processFileChunk((FileChunkMessage) message);
+ } else {
+ LOG.error("Unexpected message received: {}", message);
+ }
+ });
+ }
+
+ private void processFileTransferErrorMessage(FileTransferErrorMessage message) {
+ fileReceiver.receiveFileTransferErrorMessage(message);
+ }
+
+ private void processUploadRequest(FileUploadRequest message, String senderConsistentId, long correlationId) {
+ completedFuture(UUID.randomUUID())
+ .thenCompose(transferId -> {
+ Path directory = createTransferDirectory(transferId);
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(
+ senderConsistentId,
+ transferId,
+ directory
+ );
+ FileUploadResponse response = factory.fileUploadResponse()
+ .transferId(transferId)
+ .build();
+ Metadata metadata = message.metadata();
+ return messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(ignored -> {
+ return handler.result()
+ .thenCompose(files -> getFileHandler(metadata).handleUpload(metadata, files));
+ })
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error(
+ "Failed to handle file upload. Transfer ID: {}. Metadata: {}. Exception: {}",
+ transferId,
+ metadata,
+ e
+ );
+ }
+ deleteDirectoryIfExists(directory);
+ });
+ });
+ }
+
+ private void processDownloadRequest(FileDownloadRequest message, String senderConsistentId, Long correlationId) {
+ supplyAsync(() -> getFileProvider(message.metadata()))
+ .thenCompose(provider -> provider.files(message.metadata()))
+ .whenComplete((files, e) -> {
+ if (e != null) {
+ LOG.error("Failed to get files for download. Metadata: {}", message.metadata(), e);
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else if (files.isEmpty()) {
+ LOG.warn("No files to download. Metadata: {}", message.metadata());
+ FileDownloadResponse response = factory.fileDownloadResponse()
+ .error("No files to download")
+ .build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId);
+ } else {
+ FileDownloadResponse response = factory.fileDownloadResponse().build();
+ messagingService.respond(senderConsistentId, FILE_TRANSFERRING_CHANNEL, response, correlationId)
+ .thenCompose(v -> sendFiles(senderConsistentId, message.transferId(), files));
+ }
+ });
+ }
+
+ private CompletableFuture<Void> sendFiles(String recipientConsistentId, UUID transferId, List<File> files) {
+ return fileSender.send(recipientConsistentId, transferId, files)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ LOG.error("Failed to send files to node: {}, transfer id: {}. Exception: {}",
+ recipientConsistentId,
+ transferId,
+ e
+ );
+ FileTransferErrorMessage message = factory.fileTransferErrorMessage()
+ .transferId(transferId)
+ .error(unwrapFileTransferException(e).getMessage())
+ .build();
+ messagingService.send(recipientConsistentId, FILE_TRANSFERRING_CHANNEL, message);
+ }
+ });
+ }
+
+ private void processFileTransferInfo(FileTransferInfoMessage info) {
+ fileReceiver.receiveFileTransferInfo(info);
+ }
+
+ private void processFileHeader(FileHeaderMessage header) {
+ fileReceiver.receiveFileHeader(header);
+ }
+
+ private void processFileChunk(FileChunkMessage fileChunk) {
+ fileReceiver.receiveFileChunk(fileChunk);
+ }
+
+ @Override
+ public void stop() throws Exception {
+ IgniteUtils.closeAllManually(fileSender, fileReceiver);
+ }
+
+ @Override
+ public <M extends Metadata> void addFileProvider(
+ Class<M> metadata,
+ FileProvider<M> provider
+ ) {
+ metadataToProvider.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File provider for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileProvider<Metadata>) provider;
+ }
+ );
+ }
+
+ @Override
+ public <M extends Metadata> void addFileHandler(
+ Class<M> metadata,
+ FileConsumer<M> handler
+ ) {
+ metadataToHandler.compute(
+ getMessageType(metadata),
+ (k, v) -> {
+ if (v != null) {
+ throw new IllegalArgumentException("File handler for metadata " + metadata.getName() + " already exists");
+ }
+ return (FileConsumer<Metadata>) handler;
+ }
+ );
+ }
+
+ @Override
+ public CompletableFuture<List<File>> download(String nodeConsistentId, Metadata metadata) {
+ return completedFuture(UUID.randomUUID())
+ .thenApply(transferId -> factory.fileDownloadRequest()
+ .transferId(transferId)
+ .metadata(metadata)
+ .build())
+ .thenCompose(message -> {
+ Path directory = createTransferDirectory(message.transferId());
+ FileTransferMessagesHandler handler = fileReceiver.registerTransfer(nodeConsistentId, message.transferId(), directory);
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message,
+ RESPONSE_TIMEOUT)
+ .thenApply(FileDownloadResponse.class::cast)
+ .thenCompose(response -> {
+ if (response.error() != null) {
+ return failedFuture(new FileTransferException(response.error()));
+ } else {
+ return handler.result();
+ }
+ });
+ });
+ }
+
+ @Override
+ public CompletableFuture<Void> upload(String nodeConsistentId, Metadata metadata) {
+ return supplyAsync(() -> getFileProvider(metadata))
+ .thenCompose(provider -> provider.files(metadata))
+ .thenCompose(files -> {
+ FileUploadRequest message = factory.fileUploadRequest()
+ .metadata(metadata)
+ .build();
+
+ if (files.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
+ } else {
+ return messagingService.invoke(nodeConsistentId, FILE_TRANSFERRING_CHANNEL, message, RESPONSE_TIMEOUT)
+ .thenApply(FileUploadResponse.class::cast)
+ .thenCompose(response -> sendFiles(nodeConsistentId, response.transferId(), files));
+ }
+ });
+ }
+
+ private static short getMessageType(Class<?> metadata) {
+ Transferable annotation = metadata.getAnnotation(Transferable.class);
+ if (annotation == null) {
+ throw new IllegalArgumentException("Class " + metadata.getName() + " is not annotated with @Transferable");
+ }
+ return annotation.value();
+ }
+
+ private <M extends Metadata> FileProvider<M> getFileProvider(M metadata) {
+ FileProvider<Metadata> provider = metadataToProvider.get(metadata.messageType());
+ if (provider == null) {
+ throw new FileProviderNotFoundException(metadata.getClass());
+ } else {
+ return (FileProvider<M>) provider;
+ }
+ }
+
+ private <M extends Metadata> FileConsumer<M> getFileHandler(M metadata) {
+ FileConsumer<Metadata> handler = metadataToHandler.get(metadata.messageType());
+ if (handler == null) {
+ throw new FileHandlerNotFoundException(metadata.getClass());
+ } else {
+ return (FileConsumer<M>) handler;
+ }
+ }
+
+ private Path createTransferDirectory(UUID transferId) {
+ try {
+ return Files.createDirectories(tempDirectory.resolve(transferId.toString()));
Review Comment:
FileConsumer consumes files on upload. If I'm not going to use the upload functionality, I would prefer not to create FileConsumer. I have added a new parameter to `org.apache.ignite.internal.network.file.FileTransferService#download`, and it's `targetDir`. During the transfer, files are written to a temporary directory; once the transfer is finished, they are moved to the target one.
I think we shouldn't rewrite the `targetDir` if the transfer fails. So I would prefer to use temp directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1298179454
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -390,75 +414,64 @@ public CompletableFuture<List<Path>> download(String sourceNodeConsistentId, Ide
.identifier(identifier)
.build();
- try {
- Path directory = createTransferDirectory(transferId);
-
- CompletableFuture<List<Path>> downloadedFiles = fileReceiver.registerTransfer(
- sourceNodeConsistentId,
- transferId,
- directory
- )
- .thenApply(ignored -> {
- try {
- IgniteUtils.deleteIfExists(targetDir);
-
- Files.move(directory, targetDir, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
-
- try (Stream<Path> stream = Files.list(targetDir)) {
- return stream.collect(Collectors.toList());
- }
- } catch (IOException e) {
- throw new FileTransferException("Failed to move downloaded files to target directory", e);
- }
- })
- .whenComplete((v, e) -> {
- if (e != null) {
- LOG.error("Failed to download files. Identifier: {}", identifier, e);
- }
-
- IgniteUtils.deleteIfExists(directory);
- });
-
- return messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
- .thenApply(FileDownloadResponse.class::cast)
- .whenCompleteAsync((response, e) -> {
- if (e != null) {
- fileReceiver.cancelTransfer(transferId, e);
- } else if (response.error() != null) {
- fileReceiver.cancelTransfer(transferId, buildException(response.error()));
- } else {
- fileReceiver.receiveFileHeaders(transferId, response.headers());
- }
- }, executorService)
- .thenCompose(v -> downloadedFiles);
- } catch (Exception e) {
- return failedFuture(e);
- }
+ CompletableFuture<List<Path>> downloadedFiles = new CompletableFuture<List<Path>>()
+ .whenComplete((v, e) -> transferIdToConsumer.remove(transferId));
+
+ transferIdToConsumer.put(transferId, new DownloadRequestConsumer(downloadedFiles, targetDir));
+
+ messagingService.invoke(sourceNodeConsistentId, FILE_TRANSFER_CHANNEL, downloadRequest, responseTimeout)
+ .thenApply(FileDownloadResponse.class::cast)
+ .whenComplete((response, e) -> {
+ if (e != null) {
+ downloadedFiles.completeExceptionally(e);
+ } else if (response.error() != null) {
+ downloadedFiles.completeExceptionally(toException(response.error()));
+ }
+ });
+
+ return downloadedFiles;
}
@Override
public CompletableFuture<Void> upload(String targetNodeConsistentId, Identifier identifier) {
return getFileProvider(identifier).files(identifier)
- .thenCompose(files -> {
- FileUploadRequest message = messageFactory.fileUploadRequest()
- .identifier(identifier)
- .headers(fromPaths(messageFactory, files))
- .build();
+ .thenCompose(files -> transferFilesToNode(targetNodeConsistentId, UUID.randomUUID(), identifier, files));
+ }
+
+ private CompletableFuture<Void> transferFilesToNode(
+ String targetNodeConsistentId,
+ UUID transferId,
+ Identifier identifier,
+ List<Path> paths
+ ) {
+ if (paths.isEmpty()) {
+ return failedFuture(new FileTransferException("No files to upload"));
Review Comment:
What error will a user see in this case? If it's an error message in a server log, then it's a problem. If it's an exception message in his application, then I'm fine with it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1298181750
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -81,17 +81,21 @@ boolean hasNextChunk() {
* file.
*
* @return Chunk data.
- * @throws IllegalStateException If there are no more chunks to read.
* @throws IOException If an I/O error occurs.
*/
byte[] readNextChunk() throws IOException {
if (!hasNextChunk()) {
- throw new IllegalStateException("No more chunks to read");
+ throw new IOException("No more chunks to read");
}
int toRead = (int) Math.min(chunkSize, length - offset);
byte[] data = new byte[toRead];
- stream.read(data);
+ int read = stream.read(data);
+
+ if (read != toRead) {
+ throw new IOException("Failed to read chunk data from file: expected " + toRead + ", actual " + read + "]");
+ }
Review Comment:
This is not what I meant, you should read in a loop.
-1 would mean an unexpected end of file, but you don't check for it exclusively. Please use the API properly
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/FileTransferServiceImpl.java:
##########
@@ -244,69 +254,84 @@ private void processFileTransferInitMessage(FileTransferInitMessage message, Str
Identifier identifier = message.identifier();
- CompletableFuture<Path> directoryFuture = supplyAsync(() -> createTransferDirectory(transferId), executorService);
+ CompletableFuture<Path> directoryFuture = supplyAsync(() -> createTransferDirectory(transferId), executorService)
+ .whenComplete((directory, e) -> {
+ if (e != null) {
+ LOG.error("Failed to create transfer directory [transferId={}, identifier={}]", e, transferId, identifier);
+ }
+ });
CompletableFuture<List<Path>> uploadedFiles = directoryFuture.thenCompose(directory -> fileReceiver.registerTransfer(
senderConsistentId,
transferId,
message.headers(),
directory
- )).whenComplete((v, e) -> {
- if (e != null) {
- LOG.error("Failed to register file transfer. Transfer ID: {}. Metadata: {}", e, transferId, identifier);
- }
- });
-
- FileTransferInitResponse response = messageFactory.fileTransferInitResponse().build();
-
- messagingService.respond(senderConsistentId, FILE_TRANSFER_CHANNEL, response, correlationId)
+ ));
+
+ // Check that directory was created successfully and send response to the sender.
+ directoryFuture.handle((directory, e) -> {
Review Comment:
I know you're tired of me at this point, but does it really have to be one huge statement in the code?
Why not split it into several statements, by extracting temporary results into variables?
Why not extract some parts into methods?
Are you comfortable yourself with the code of such complexity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] Flaugh24 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "Flaugh24 (via GitHub)" <gi...@apache.org>.
Flaugh24 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1296035876
##########
modules/file-transfer/src/main/java/org/apache/ignite/internal/network/file/ChunkedFileReader.java:
##########
@@ -51,63 +61,59 @@ private ChunkedFileReader(String fileName, RandomAccessFile raf, int chunkSize)
* @param file File.
* @param chunkSize Chunk size.
* @return Chunked file reader.
- * @throws IOException If an I/O error occurs.
+ * @throws FileNotFoundException If the file does not exist.
*/
- static ChunkedFileReader open(File file, int chunkSize) throws IOException {
- return new ChunkedFileReader(file.getName(), new RandomAccessFile(file, "r"), chunkSize);
+ static ChunkedFileReader open(File file, int chunkSize) throws FileNotFoundException {
+ return new ChunkedFileReader(file.length(), new BufferedInputStream(new FileInputStream(file)), chunkSize);
}
/**
* Returns {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*
* @return {@code false} if there are no more chunks to read. Otherwise, returns {@code true}.
*/
- boolean hasNextChunk() throws IOException {
- return raf.getFilePointer() != raf.length();
+ boolean hasNextChunk() {
+ return offset < length;
}
/**
- * Reads the next chunk.
+ * Reads the next chunk. If there are no more chunks to read, throws an exception. If the last chunk is read successfully, closes the
+ * file.
*
* @return Chunk data.
+ * @throws IllegalStateException If there are no more chunks to read.
* @throws IOException If an I/O error occurs.
*/
byte[] readNextChunk() throws IOException {
if (!hasNextChunk()) {
- throw new IOException("No more chunks to read");
+ throw new IllegalStateException("No more chunks to read");
}
- int toRead = (int) Math.min(chunkSize, length() - offset());
+ int toRead = (int) Math.min(chunkSize, length - offset);
byte[] data = new byte[toRead];
- raf.read(data);
- return data;
- }
+ stream.read(data);
+ offset += toRead;
+ nextChunkNumber++;
- /**
- * Returns the file name.
- *
- * @return File name.
- */
- String fileName() {
- return fileName;
- }
+ if (offset == length) {
+ stream.close();
+ }
- /**
- * Returns the current chunk offset.
- *
- * @return Chunk size.
- */
- long offset() throws IOException {
- return raf.getFilePointer();
+ return data;
}
/**
- * Returns the file length.
+ * Returns the number of the next chunk to read.
*
- * @return File length.
+ * @return The number of the next chunk to read.
+ * @throws IllegalStateException If there are no more chunks to read.
*/
- long length() throws IOException {
- return raf.length();
+ int nextChunkNumber() {
+ if (!hasNextChunk()) {
+ throw new IllegalStateException("No more chunks to read");
Review Comment:
I would expect that there is a next chunk if it has a number. So we have to return a special value (e.g., -1) or throw an exception to indicate that there are no more chunks to read. Actually, I don't see a difference between those approaches because it is expected that a user has checked the absence of the next chunk by calling hasNextChunk().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sk0x50 commented on a diff in pull request #2390: IGNITE-19009 Introduce file transfer support in messaging
Posted by "sk0x50 (via GitHub)" <gi...@apache.org>.
sk0x50 commented on code in PR #2390:
URL: https://github.com/apache/ignite-3/pull/2390#discussion_r1283203408
##########
modules/core/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -400,4 +400,15 @@ public static class Compute {
/** Class loader error. */
public static final int CLASS_LOADER_ERR = COMPUTE_ERR_GROUP.registerErrorCode((short) 2);
}
+
+ /**
+ * File transfer error group.
+ */
+ public static class FileTransfer {
+ /** File transfer error group. */
+ public static final ErrorGroup FILE_TRANSFER_ERR_GROUP = ErrorGroup.newGroup("FILE_TRANSFER", (short) 17);
+
+ /** Generic error. */
+ public static final int GENERIC_ERR = FILE_TRANSFER_ERR_GROUP.registerErrorCode((short) 1);
Review Comment:
Please, avoid using generic error codes, just because it does not make much sense to the end user. This must be used with great care.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org