You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "tungtose (via GitHub)" <gi...@apache.org> on 2025/12/12 09:42:16 UTC

[PR] feat(performance): socket migration across shard [iggy]

tungtose opened a new pull request, #2476:
URL: https://github.com/apache/iggy/pull/2476

   This PR addressing #2386
   
   Enable TCP socket migration across shards. When a client sends messages to the wrong shard, migrate the socket to the correct shard to eliminate cross-shard forwarding overhead
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "tungtose (via GitHub)" <gi...@apache.org>.
tungtose commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2627806220


##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,96 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration
+            && !(session.is_migrated() || balanced_partitioning)
+            && let Some(target_shard) = shard.find_shard(&namespace)
+            && target_shard.id != shard.id
+        {
+            debug!(
+                "TCP wrong shared detected: migrating from_shard {}, to_shard {}",
+                shard.id, target_shard.id
+            );
+
+            if let Some(fd) = sender.take_and_migrate_tcp() {
+                let payload = ShardRequestPayload::SocketTransfer {
+                    fd,
+                    from_shard: shard.id,
+                    client_id: session.client_id,
+                    user_id,
+                    address: session.ip_address,
+                    initial_data: batch,
+                };
+
+                let request = ShardRequest::new(
+                    self.stream_id.clone(),
+                    self.topic_id.clone(),
+                    partition_id,
+                    payload,
+                );
+
+                let socket_transfer_msg = ShardMessage::Request(request);
+
+                shard
+                    .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
+                    .await?;

Review Comment:
   Is this good enough?
   ```
                   if let Err(e) = shard
                       .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
                       .await
                   {
                       error!("tranfer socket to another shard failed, drop connection. {e:?}");
                       return Ok(HandlerResult::Finished);
                   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "hubert-laserdata (via GitHub)" <gi...@apache.org>.
hubert-laserdata commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2634855198


##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,100 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration

Review Comment:
   i think we also want to disable it if PartitioningKind == MessagesKey



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2627560347


##########
core/server/src/streaming/utils/crypto.rs:
##########
@@ -32,6 +32,7 @@ pub fn hash_password(password: &str) -> String {
 }
 
 pub fn verify_password(password: &str, hash: &str) -> bool {
+    // tracing::error!("Checking password: {password}, hash: {hash}");

Review Comment:
   remove this



##########
core/common/src/sender/mod.rs:
##########
@@ -114,6 +117,26 @@ impl SenderKind {
         Self::WebSocketTls(stream)
     }
 
+    pub fn take_and_migrate_tcp(&mut self) -> Option<RawFd> {
+        match self {
+            SenderKind::Tcp(tcp_sender) => {
+                let stream = tcp_sender.stream.take()?;
+                let poll_fd = stream.into_poll_fd().ok()?;
+
+                let raw_fd = poll_fd.as_fd();
+                let Ok(owned_fd) = nix::unistd::dup(raw_fd) else {
+                    // TODO(tungtose): recover tcp stream?
+                    error!("Failed to dup fd");
+                    return None;
+                };
+
+                let raw_fd = owned_fd.into_raw_fd();
+                Some(raw_fd)
+            }
+            _ => None,

Review Comment:
   add TODO: support TCP TLS



##########
core/common/src/sender/mod.rs:
##########
@@ -114,6 +117,26 @@ impl SenderKind {
         Self::WebSocketTls(stream)
     }
 
+    pub fn take_and_migrate_tcp(&mut self) -> Option<RawFd> {
+        match self {
+            SenderKind::Tcp(tcp_sender) => {
+                let stream = tcp_sender.stream.take()?;
+                let poll_fd = stream.into_poll_fd().ok()?;
+
+                let raw_fd = poll_fd.as_fd();
+                let Ok(owned_fd) = nix::unistd::dup(raw_fd) else {
+                    // TODO(tungtose): recover tcp stream?
+                    error!("Failed to dup fd");
+                    return None;
+                };
+
+                let raw_fd = owned_fd.into_raw_fd();

Review Comment:
   i was wondering, can we use OwnedFd instead of into_raw_fd? this way we woudl have auto close if theres error later (raii)



##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,96 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration
+            && !(session.is_migrated() || balanced_partitioning)
+            && let Some(target_shard) = shard.find_shard(&namespace)
+            && target_shard.id != shard.id
+        {
+            debug!(
+                "TCP wrong shared detected: migrating from_shard {}, to_shard {}",
+                shard.id, target_shard.id
+            );
+
+            if let Some(fd) = sender.take_and_migrate_tcp() {
+                let payload = ShardRequestPayload::SocketTransfer {
+                    fd,
+                    from_shard: shard.id,
+                    client_id: session.client_id,
+                    user_id,
+                    address: session.ip_address,
+                    initial_data: batch,
+                };
+
+                let request = ShardRequest::new(
+                    self.stream_id.clone(),
+                    self.topic_id.clone(),
+                    partition_id,
+                    payload,
+                );
+
+                let socket_transfer_msg = ShardMessage::Request(request);
+
+                shard
+                    .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
+                    .await?;

Review Comment:
   if send_request_to_shard_or_recoil fails, the fd is already extracted from the TcpSender (stream is None), but no cleanup occurs,so fd leaks and the connection is in an inconsistent state



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz merged PR #2476:
URL: https://github.com/apache/iggy/pull/2476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #2476:
URL: https://github.com/apache/iggy/pull/2476#issuecomment-3655305346

   > > overall, design in this PR is sound, however i'd like you to consider: all 50+ handlers now return HandlerResult but only SendMessages use Migrated. have you thought about adding IggyError::ConnectionMigrated(to_shard) and catch it in connection handler? this would greatly reduce changed files in the PR
   > > @spetz @numinnex opinions?
   > 
   > I actually did that in the first place, but on second thought, this is not an error, so I have to change all of the handler results. I would be happy to change it if you prefer to treat it as an error
   
   I mean your approach does make sense... Lets wait for other core devs opinion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2618913259


##########
core/server/src/shard/handlers.rs:
##########
@@ -15,6 +15,8 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use std::os::fd::FromRawFd;
+

Review Comment:
   remove empty line between use items, correct in other places too



##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,105 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration
+            && !(session.is_migrated() || balanced_partitioning)
+            && let Some(target_shard) = shard.find_shard(&namespace)
+            && target_shard.id != shard.id
+        {
+            info!(
+                "TCP wrong shared detected: from_shard {}, to_shard {}",
+                shard.id, target_shard.id
+            );
+
+            if let Some(fd) = sender.take_and_migrate_tcp() {
+                let payload = ShardRequestPayload::SocketTransfer {
+                    fd,
+                    from_shard: shard.id,
+                    client_id: session.client_id,
+                    user_id,
+                    address: session.ip_address,
+                    initial_data: batch,
+                };
+
+                let request = ShardRequest::new(
+                    self.stream_id.clone(),
+                    self.topic_id.clone(),
+                    partition_id,
+                    payload,
+                );
+
+                let socket_transfer_msg = ShardMessage::Request(request);
+
+                if let Err(err) = shard
+                    .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
+                    .await
+                {
+                    error!(
+                        "Failed to send socket transfer to shard {}, err: {:?}",
+                        target_shard.id, err
+                    );
+
+                    // TODO(tungtose): restore TcpStream to current shard or drop

Review Comment:
   i'd drop connection



##########
core/server/src/shard/handlers.rs:
##########
@@ -312,10 +319,80 @@ async fn handle_request(
             shard.broadcast_event_to_all_shards(event).await?;
             Ok(ShardResponse::DeleteStreamResponse(stream))
         }
+        ShardRequestPayload::SocketTransfer {
+            fd,
+            from_shard,
+            client_id,
+            user_id,
+            address,
+            initial_data,
+        } => {
+            info!(
+                "Received socket transfer msg, fd: {fd}, from_shard: {from_shard}, address: {address}"
+            );
+
+            // restore TcpStream from fd
+            let tcp_stream = unsafe { TcpStream::from_raw_fd(fd) };

Review Comment:
   before reconstructing TcpStream check if fd is valid, something like 
   `nix::sys::stat::fstat(fd).map_err(|e| IggyError::IoError(format!("Invalid fd: {}", e)))?;`



##########
core/server/src/streaming/session.rs:
##########
@@ -60,6 +62,14 @@ impl Session {
         self.active.set(false);
     }
 
+    pub fn is_migrated(&self) -> bool {

Review Comment:
   whats the intended behavior? once migrated, can this session never migrate again?is that intentional? pls add brief doc comment



##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,105 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration
+            && !(session.is_migrated() || balanced_partitioning)
+            && let Some(target_shard) = shard.find_shard(&namespace)
+            && target_shard.id != shard.id
+        {
+            info!(
+                "TCP wrong shared detected: from_shard {}, to_shard {}",

Review Comment:
   typo, and this print is info. is this intended?



##########
core/configs/server.toml:
##########
@@ -141,6 +141,9 @@ enabled = true
 # For example, "127.0.0.1:8090" listens on localhost only on port 8090.
 address = "127.0.0.1:8090"
 
+# Enable TCP socket migration across shards.

Review Comment:
   add startup log to server when its enabled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2613626772


##########
Cargo.toml:
##########
@@ -142,6 +142,7 @@ integration = { path = "core/integration" }
 keyring = { version = "3.6.3", features = ["sync-secret-service", "vendored"] }
 lazy_static = "1.5.0"
 log = "0.4.29"
+libc = "0.2"

Review Comment:
   please use nix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "tungtose (via GitHub)" <gi...@apache.org>.
tungtose commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2627806220


##########
core/server/src/binary/handlers/messages/send_messages_handler.rs:
##########
@@ -107,19 +110,96 @@ impl ServerCommandHandler for SendMessages {
         );
         batch.validate()?;
 
+        shard.ensure_topic_exists(&self.stream_id, &self.topic_id)?;
+
+        let numeric_stream_id = shard
+            .streams
+            .with_stream_by_id(&self.stream_id, streams::helpers::get_stream_id());
+
+        let numeric_topic_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            topics::helpers::get_topic_id(),
+        );
+
+        // TODO(tungtose): dry this code && get partition_id below have a side effect
+        let partition_id = shard.streams.with_topic_by_id(
+            &self.stream_id,
+            &self.topic_id,
+            |(root, auxilary, ..)| match self.partitioning.kind {
+                PartitioningKind::Balanced => {
+                    let upperbound = root.partitions().len();
+                    let pid = auxilary.get_next_partition_id(upperbound);
+                    Ok(pid)
+                }
+                PartitioningKind::PartitionId => Ok(u32::from_le_bytes(
+                    self.partitioning.value[..self.partitioning.length as usize]
+                        .try_into()
+                        .map_err(|_| IggyError::InvalidNumberEncoding)?,
+                ) as usize),
+                PartitioningKind::MessagesKey => {
+                    let upperbound = root.partitions().len();
+                    Ok(
+                        topics::helpers::calculate_partition_id_by_messages_key_hash(
+                            upperbound,
+                            &self.partitioning.value,
+                        ),
+                    )
+                }
+            },
+        )?;
+
+        let namespace = IggyNamespace::new(numeric_stream_id, numeric_topic_id, partition_id);
         let user_id = session.get_user_id();
+        let balanced_partitioning = matches!(self.partitioning.kind, PartitioningKind::Balanced);
+        let enabled_socket_migration = shard.config.tcp.socket_migration;
+
+        if enabled_socket_migration
+            && !(session.is_migrated() || balanced_partitioning)
+            && let Some(target_shard) = shard.find_shard(&namespace)
+            && target_shard.id != shard.id
+        {
+            debug!(
+                "TCP wrong shared detected: migrating from_shard {}, to_shard {}",
+                shard.id, target_shard.id
+            );
+
+            if let Some(fd) = sender.take_and_migrate_tcp() {
+                let payload = ShardRequestPayload::SocketTransfer {
+                    fd,
+                    from_shard: shard.id,
+                    client_id: session.client_id,
+                    user_id,
+                    address: session.ip_address,
+                    initial_data: batch,
+                };
+
+                let request = ShardRequest::new(
+                    self.stream_id.clone(),
+                    self.topic_id.clone(),
+                    partition_id,
+                    payload,
+                );
+
+                let socket_transfer_msg = ShardMessage::Request(request);
+
+                shard
+                    .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
+                    .await?;

Review Comment:
   Combine with `OwnedFd`. Is this good enough?
   ```
                   if let Err(e) = shard
                       .send_request_to_shard_or_recoil(Some(&namespace), socket_transfer_msg)
                       .await
                   {
                       error!("tranfer socket to another shard failed, drop connection. {e:?}");
                       return Ok(HandlerResult::Finished);
                   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "tungtose (via GitHub)" <gi...@apache.org>.
tungtose commented on PR #2476:
URL: https://github.com/apache/iggy/pull/2476#issuecomment-3655071811

   > overall, design in this PR is sound, however i'd like you to consider: all 50+ handlers now return HandlerResult but only SendMessages use Migrated. have you thought about adding IggyError::ConnectionMigrated(to_shard) and catch it in connection handler? this would greatly reduce changed files in the PR
   > 
   > @spetz @numinnex opinions?
   
   I actually did that in the first place, but a second thought was that this is not an error, so I have to change all of the handler results. I would be happy to change it if the maintainer prefers to treat it as an error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "tungtose (via GitHub)" <gi...@apache.org>.
tungtose commented on PR #2476:
URL: https://github.com/apache/iggy/pull/2476#issuecomment-3660008328

   @hubcio I have resolved all comments and fixed the C# tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(server): socket migration across shard [iggy]

Posted by "tungtose (via GitHub)" <gi...@apache.org>.
tungtose commented on code in PR #2476:
URL: https://github.com/apache/iggy/pull/2476#discussion_r2622860250


##########
core/configs/server.toml:
##########
@@ -141,6 +141,9 @@ enabled = true
 # For example, "127.0.0.1:8090" listens on localhost only on port 8090.
 address = "127.0.0.1:8090"
 
+# Enable TCP socket migration across shards.

Review Comment:
   resolved



##########
core/server/src/streaming/session.rs:
##########
@@ -60,6 +62,14 @@ impl Session {
         self.active.set(false);
     }
 
+    pub fn is_migrated(&self) -> bool {

Review Comment:
   resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org