You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by "yamt (via GitHub)" <gi...@apache.org> on 2024/07/11 13:27:30 UTC

[PR] Reimplement readv to deal with short reads [nuttx]

yamt opened a new pull request, #12674:
URL: https://github.com/apache/nuttx/pull/12674

   ## Summary
   The original implementation seems to have an assumption that the file is a regular file.
   
   This commit re-implements it so that it can deal with tty, sockets, etc.
   
   ## Impact
   
   ## Testing
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1678634841


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > but adding `malloc` and trying to avoid memory leak with nested cancellation point is also conceptually wrong.
   
   it's just inefficient.
   it's far better than the original implementation, which is broken in a user-visible way.
   
   > I do not have a better solution right not and just noticed and `writev` seems also to suffer from the same problem.
   
   right. writev/preadv/pwritev have the same problem.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2591614117

   i'm closing this as https://github.com/apache/nuttx/pull/14898, which fixes the issue for the specific driver i care right now, was merged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2578043905

   should we close this pr without merging? @yamt 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2579393807

   > should we close this pr without merging? @yamt
   
   what this PR does is still valid.
   however, at this point i'm more interested in finishing https://github.com/apache/nuttx/pull/14898.
   i will mark this a draft for now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674237880


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   consider that you readv() on a tty, using a huge buffer, say, 1024 bytes.
   if you got some data, say, 1 byte, readv() should return the 1 byte without waiting for the following 1023 bytes.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675062979


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   no.
   
   * these tricks are not thread-safe.
   * we should preserve packet boundary for some kind of fds like datagram socket.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675469100


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > like this:
   > 
   > ```
   > ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
   > {
   >   return iovcnt ? read(fildes, iov->iov_base, iov->iov_len) : 0;
   > }
   > ```
   
   what is it compatible with?
   to me it seems broken even for regular files.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686724977


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }

Review Comment:
   > I see why iovec-based multiple reads could be a problem (e.g., read data available == size of iovec, then we get stuck in blocked read). Ok, I am fine to merge this PR with malloc, and maybe someone can solve how to remove malloc later.
   
   i guess that tricks involving multiple read() calls for iovcnt>1 are all broken in one way or another.
   implementing readv() on the top of read() is a design mistake. it should be the opposite.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686689065


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  enter_cancellation_point();
+  buffer = malloc(total_size);
+  if (buffer == NULL)
+    {
+      return -1;
+    }
 
-              /* Update pointers and counts in order to handle partial
-               * buffer reads.
-               */
+  nread = read(fildes, buffer, total_size);
+  if (nread == -1)

Review Comment:
   if you want that way, we should return -1 instead of nread i guess.
   anyway, it isn't this PR is about.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354475820

   > > > > > can we make a decision?
   > > > > 
   > > > > 
   > > > > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   > > > 
   > > > 
   > > > it doesn't conform the spec.
   > > 
   > > 
   > > Why?
   > 
   > consider:
   > 
   > * file types which are usually not interruptible. eg. regular files
   
   if spec enforce the implementation must read all available data, please point out the statement.
   
   > * file types which are supposed to preserve data boundaries. eg. udp sockets
   
   since udp already support sendmsg, the right fix is mapping writev to sendmsg by checking the fd type is socket.
   
   > * file types which are supposed to preserve read/write atomicity. eg. regular files
   
   it's caller responsibility to ensure the atomicity if readv/writev just can finish the partial job.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674312172


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   the loop can be changed to:
   ```
   ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
   {
     ssize_t ntotal = 0;
     ssize_t nread;
     int i;
   
     /* Process each entry in the struct iovec array */
   
     for (i = 0; i < iovcnt; i++)
       {
         /* Ignore zero-length reads */
   
         if (iov[i].iov_len > 0)
           {
             nread = read(iov[i].iov_base, iov[i].iov_len);
             if (nread <= 0)
                 {
                   break;
                 }
   
             buffer += nread;
             ntotal += nread;
   
             if (nread != iov[i].iov_len)
                 {
                   break;
                 }
           }
       }
   
     return ntotal ? ntotal : nread;
   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675070090


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   the only sane way w/o malloc i can think of is to make the kernel implement readv natively and implement read on the top of it. but it would be a far more invasive change than this PR. (probably we need to modify every drivers) thus i submitted this PR for now.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686680236


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  buffer = malloc(total_size);

Review Comment:
   it doesn't work. please read the discussions in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354291423

   > can we make a decision?
   
   ping


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354479737

   BTW, it's better to do the different action base on the fd type:
   
   1. forward to sendmsg/recvmsg for socket type
   2. read/write as much as possible for regular file
   3. read/write the first block for all other type
   
   of course, it may need to move readv/writev from libc to fs/vfs to do this type of dispatch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2493510274

   since the better approach is merged(https://github.com/apache/nuttx/pull/13498), let's close this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675462820


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   like this:
   ```
   ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
   {
     return iovcnt ? read(fildes, iov->iov_base, iov->iov_len) : 0;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674202908


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   yes, we can break out by returning the length we really got so far.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "hartmannathan (via GitHub)" <gi...@apache.org>.
hartmannathan commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686706648


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }

Review Comment:
   > @yamt Agreed, but if we can eliminate malloc (see other feedback [here](https://github.com/apache/nuttx/pull/12674#discussion_r1685824633)) then this optimization can be eliminated also.
   
   I see why iovec-based multiple reads could be a problem (e.g., read data available == size of iovec, then we get stuck in blocked read). Ok, I am fine to merge this PR with malloc, and maybe someone can solve how to remove malloc later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674312172


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   the loop can be changed to:
   ```
   ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
   {
     ssize_t ntotal;
     ssize_t nread;
     int i;
   
     /* Process each entry in the struct iovec array */
   
     for (i = 0, ntotal = 0; i < iovcnt; i++)
       {
         /* Ignore zero-length reads */
   
         if (iov[i].iov_len > 0)
           {
             nread = read(iov[i].iov_base, iov[i].iov_len);
             if (nread <= 0)
                 {
                   break;
                 }
   
             buffer  += nread;
             ntotal  += nread;
   
             if (nread != iov[i].iov_len)
                 {
                   break;
                 }
           }
       }
   
     return ntotal ? ntotal : nread;
   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674963627


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   changing to nonblock mode or calling select/poll with 0 timeout could fix this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1676999523


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > @yamt @xiaoxiang781216 how does Linux, QNX, MacOS, etc implement it? Although the specification doesn't require some assumptions, it is important to follow what other OSes are doing, to avoid breaking applications when porting it to NuttX
   
   Linux improve the driver model by changing argument from void * to iovec*.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674240774


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > yes, we can break out by returning the length we really got so far.
   
   i don't think you can do that reliably because data boundary can happen to match iov boundary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "pkarashchenko (via GitHub)" <gi...@apache.org>.
pkarashchenko commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1676874321


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   I agree that there is an issue with `readv` behaviour, but adding `malloc` and trying to avoid memory leak with nested cancellation point is also conceptually wrong. I do not have a better solution right not and just noticed and `writev` seems also to suffer from the same problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2355190176

   > > > > * file types which are usually not interruptible. eg. regular files
   > > > 
   > > > 
   > > > if spec enforce the implementation must read all available data, please point out the statement.
   > > 
   > > 
   > > i don't know where/if it's explicitly specified in the spec. it's the traditional common behavior which many applications rely on.
   > 
   > But the spec explicitly allows return the short length. Caller always need handle the short return correctly.
   
   whatever the standard allows, i don't think it's a good idea to break the semantics which real applications have been relying on for decades.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675458454


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > another compatible implementation is always just read the first element.
   
   i don't understand what you mean. can you explain?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1676999523


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > @yamt @xiaoxiang781216 how does Linux, QNX, MacOS, etc implement it? Although the specification doesn't require some assumptions, it is important to follow what other OSes are doing, to avoid breaking applications when porting it to NuttX
   
   Linux improve the driver model by changing argument from void * to iovec*. The official approach is extending file_operation_s to support iovec like what Linux or NuttX sock_intf_s(https://github.com/apache/nuttx/pull/2959).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675084706


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   if you are so unhappy with malloc, how about doing fstat() and fallback to the old implementation for regular files? i guess it covers ~90% of cases. i personally prefer to avoid such "optimization" for the sake of simplicity though.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2225074402

   > The original implementation seems to have an assumption that the file is a regular file.
   
   well, the original implementation was actually broken for regular files as well wrt read/write atomicity.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 closed pull request #12674: Reimplement readv to deal with short reads
URL: https://github.com/apache/nuttx/pull/12674


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2290816752

   can we make a decision?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354459114

   > > > can we make a decision?
   > > 
   > > 
   > > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   > 
   > it doesn't conform the spec.
   
   Why?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "hartmannathan (via GitHub)" <gi...@apache.org>.
hartmannathan commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686701196


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }

Review Comment:
   @yamt Agreed, but if we can eliminate malloc (see other feedback [here](https://github.com/apache/nuttx/pull/12674#discussion_r1685824633)) then this optimization can be eliminated also.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1686678428


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }

Review Comment:
   the "common logic" below involves malloc. it's better to avoid it when easy.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674312172


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   the loop can be changed to:
   ```
   ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
   {
     ssize_t ntotal = 0;
     ssize_t nread;
     int i;
   
     /* Process each entry in the struct iovec array */
   
     for (i = 0; i < iovcnt; i++)
       {
         /* Ignore zero-length reads */
   
         if (iov[i].iov_len > 0)
           {
             nread = read(iov[i].iov_base, iov[i].iov_len);
             if (nread <= 0)
               {
                 break;
               }
   
             buffer += nread;
             ntotal += nread;
   
             if (nread != iov[i].iov_len)
               {
                 break;
               }
           }
       }
   
     return ntotal ? ntotal : nread;
   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674212698


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   BTW, I don't get the problem from your commit message. could you explain more?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675084706


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   if you are so unhappy with malloc, how about doing stat() and fallback to the old implementation for regular files? i guess it covers ~90% of cases. i personally prefer to avoid such "optimization" for the sake of simplicity though.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675722128


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   compatible with the spec, the spec never requires we return all available data:
   https://pubs.opengroup.org/onlinepubs/009695399/functions/readv.html
   so, the implementation avoids the allocation and conform to the spec.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674092440


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  buffer = malloc(total_size);

Review Comment:
   i can't think of any trivial ways.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "pkarashchenko (via GitHub)" <gi...@apache.org>.
pkarashchenko commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1676876399


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   Sounds like `struct file_operations` read and write methods should actually operate with vectors (the same as `file_read` and `file_write`), but such a change basically would require rewriting thousands lines of the code...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "acassis (via GitHub)" <gi...@apache.org>.
acassis commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675760835


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   @yamt @xiaoxiang781216 how Linux, QNX, MacOS, etc implement it? Although the specification doesn't requires some assumptions, it is important to follow what other OSes are doing, to avoid  breaking applications when porting it to NuttX



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2574540912

   @yamt could you fix the conflict? so we can merge this patch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354468853

   > > > > can we make a decision?
   > > > 
   > > > 
   > > > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   > > 
   > > 
   > > it doesn't conform the spec.
   > 
   > Why?
   
   consider:
   
   * file types which are usually not interruptible. eg. regular files
   * file types which are supposed to preserve data boundaries. eg. udp sockets
   * file types which are supposed to preserve read/write atomicity. eg. regular files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354454226

   > > can we make a decision?
   > 
   > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   
   it doesn't conform the spec.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "acassis (via GitHub)" <gi...@apache.org>.
acassis commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675760835


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   @yamt @xiaoxiang781216 how Linux, QNX, MacOS, etc implement it? Although the specification doesn't require some assumptions, it is important to follow what other OSes are doing, to avoid  breaking applications when porting it to NuttX



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675449969


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   another compatible implementation is always just read the first element.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2223050585

   > @yamt nice! Is there some example that showing the failure? Should it be added to apps/testing/ ?
   
   micropython repl (with https://github.com/micropython/micropython/pull/13676) on toywasm and wamr.
   
   > Another question: did you run ostest and LTS locally to confirm there is not known side effects?
   
   no.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354446764

   > can we make a decision?
   
   I would prefer to just read the first memory block, which conform the spec and don't need malloc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354501134

   > > > > > > can we make a decision?
   > > > > > 
   > > > > > 
   > > > > > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   > > > > 
   > > > > 
   > > > > it doesn't conform the spec.
   > > > 
   > > > 
   > > > Why?
   > > 
   > > 
   > > consider:
   > > 
   > > * file types which are usually not interruptible. eg. regular files
   > 
   > if spec enforce the implementation must read all available data, please point out the statement.
   
   i don't know where/if it's explicitly specified in the spec.
   it's the traditional common behavior which many applications rely on.
   
   > 
   > > * file types which are supposed to preserve data boundaries. eg. udp sockets
   > 
   > since udp already support sendmsg, the right fix is mapping writev to sendmsg by checking the fd type is socket.
   
   udp is just an example.
   eg. there can be such character devices.
   
   > 
   > > * file types which are supposed to preserve read/write atomicity. eg. regular files
   > 
   > it's caller responsibility to ensure the atomicity if readv/writev just can finish the partial job.
   
   no.
   search "atomic" in https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674799609


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > the loop can be changed to:
   
   it seems broken to me.
   
   consider:
   * iovcnt == 2
   * iov[0]->iov_len == 4
   * iov[1]->iov_len == 4
   * 4 byte data is available to read from the fd



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1677000397


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   > no.
   > 
   > * these tricks are not thread-safe.
   
   > * we should preserve packet boundary for some kind of fds like datagram socket.
   
   socket already improve this case by: https://github.com/apache/nuttx/pull/2959. But file_operation_s doesn't have the similar patch, so it's hard to modify readv to utilize this new capability from socket layer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2497751473

   > since the better approach is merged(#13498), let's close this pr.
   
   https://github.com/apache/nuttx/pull/14898 itself doesn't fix the problem i wanted to fix with this PR.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354506947

   > It's better to do the different action base on the fd type before we add readv/writev to file_operation:
   > 
   >     1. forward to sendmsg/recvmsg for socket type
   > 
   >     2. read/write as much as possible for regular file
   > 
   >     3. read/write the first block for all other type
   > 
   > 
   > of course, it may need to move readv/writev from libc to fs/vfs to do this type of dispatch. BTW, to simplify readv/writev dispatch, we can add readv/writev callback to file_operation and provide the default readv/writev implementation if the driver or file system doesn't provide one.
   
   see https://github.com/apache/nuttx/pull/12674#discussion_r1678639259
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2354534509

   i don't think it's a good idea to block an obvious fix like this for months just saying "there can be a better solution" w/o actually providing any better solutions.
   
   if this PR was merged two months ago, i might even have implemented the ideal iov-based solution at least partly in the two months.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2355166165

   > > > > > > > can we make a decision?
   > > > > > > 
   > > > > > > 
   > > > > > > I would prefer to just read the first memory block, which conform the spec and don't need malloc.
   > > > > > 
   > > > > > 
   > > > > > it doesn't conform the spec.
   > > > > 
   > > > > 
   > > > > Why?
   > > > 
   > > > 
   > > > consider:
   > > > 
   > > > * file types which are usually not interruptible. eg. regular files
   > > 
   > > 
   > > if spec enforce the implementation must read all available data, please point out the statement.
   > 
   > i don't know where/if it's explicitly specified in the spec. it's the traditional common behavior which many applications rely on.
   > 
   
   But the spec explicitly allows return the short length. Caller always need handle the short return correctly.
   
   > i don't think it's a good idea to block an obvious fix like this for months just saying "there can be a better solution" w/o actually providing any better solutions.
   > 
   > if this PR was merged two months ago, i might even have implemented the ideal iov-based solution at least partly in the two months.
   
   Since this solution isn't perfect, especially the allocation happens in the read/write path, I hesitate to merge this change, but other maintainers could merge it if they think it's OK.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "hartmannathan (via GitHub)" <gi...@apache.org>.
hartmannathan commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1685817660


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  enter_cancellation_point();
+  buffer = malloc(total_size);
+  if (buffer == NULL)
+    {
+      return -1;
+    }
 
-              /* Update pointers and counts in order to handle partial
-               * buffer reads.
-               */
+  nread = read(fildes, buffer, total_size);
+  if (nread == -1)

Review Comment:
   Would it be better to do "if (nread < 0)" (defensive coding)?



##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +72,66 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }

Review Comment:
   Is this optimization necessary? Would it be better to remove it and use common logic below?



##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  buffer = malloc(total_size);

Review Comment:
   The only way to avoid malloc is to read directly into each iov and avoid the copy. In other words: iterate iov array. For each iov, if iov_len > 0, read() iov_len bytes into that iov. If any read() returns less than iov_len bytes, break from loop. Keep running counter of total bytes read.
   
   I don't know if this is permissible under POSIX but it (1) avoids malloc, and (2) avoids copying. Even if there is added overhead from multiple calls to read(), it might be better than overhead of malloc(), free(), and copying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674089471


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);
+      return 0;
+    }
 
-              /* Check for a read error */
+  if (iovcnt == 1)
+    {
+      return read(fildes, iov->iov_base, iov->iov_len);
+    }
 
-              if (nread < 0)
-                {
-                  return nread;
-                }
+  total_size = 0;
+  for (i = 0; i < iovcnt; i++)
+    {
+      total_size += iov[i].iov_len;
+    }
 
-              /* Check for an end-of-file condition */
+  if (total_size == 0)
+    {
+      return 0;
+    }
 
-              else if (nread == 0)
-                {
-                  return ntotal;
-                }
+  buffer = malloc(total_size);

Review Comment:
   could we avoid malloc



##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   why we can't read the data piece by piece?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1674096743


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   because, in general, you should not keep reading after a short read.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "acassis (via GitHub)" <gi...@apache.org>.
acassis commented on PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#issuecomment-2222978257

   @yamt nice! Is there some example that showing the failure? Should it be added to apps/testing/ ?
   
   Another question: did you run ostest and LTS locally to confirm there is not known side effects?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675758689


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   i'm not sure if such an interpretation of the spec makes much sense.
   as you know, regular files are usually not interruptible on unix-like operating systems and many applications are relying on it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "acassis (via GitHub)" <gi...@apache.org>.
acassis commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1675760835


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   @yamt @xiaoxiang781216 how does Linux, QNX, MacOS, etc implement it? Although the specification doesn't require some assumptions, it is important to follow what other OSes are doing, to avoid  breaking applications when porting it to NuttX



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt commented on code in PR #12674:
URL: https://github.com/apache/nuttx/pull/12674#discussion_r1678639259


##########
libs/libc/uio/lib_readv.c:
##########
@@ -68,56 +70,63 @@
 
 ssize_t readv(int fildes, FAR const struct iovec *iov, int iovcnt)
 {
-  ssize_t ntotal;
-  ssize_t nread;
-  size_t remaining;
   FAR uint8_t *buffer;
+  ssize_t nread;
+  size_t tocopy;
+  size_t total_size;
+  size_t offset;
   int i;
 
-  /* Process each entry in the struct iovec array */
-
-  for (i = 0, ntotal = 0; i < iovcnt; i++)
+  if (iovcnt == 0)
     {
-      /* Ignore zero-length reads */
-
-      if (iov[i].iov_len > 0)
-        {
-          buffer    = iov[i].iov_base;
-          remaining = iov[i].iov_len;
-
-          /* Read repeatedly as necessary to fill buffer */
-
-          do
-            {
-              /* NOTE:  read() is a cancellation point */
-
-              nread = read(fildes, buffer, remaining);

Review Comment:
   if someone plans to do the ideal solution (ie. make ~everything iovec-based) anytime soon, it's great.
   otherwise, i'd suggest to merge this PR for now.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reimplement readv to deal with short reads [nuttx]

Posted by "yamt (via GitHub)" <gi...@apache.org>.
yamt closed pull request #12674: Reimplement readv to deal with short reads
URL: https://github.com/apache/nuttx/pull/12674


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org