You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celix.apache.org by GitBox <gi...@apache.org> on 2021/01/27 17:21:36 UTC

[GitHub] [celix] pnoltes opened a new pull request #315: Adds test and improvements for remote service calls for large payloads

pnoltes opened a new pull request #315:
URL: https://github.com/apache/celix/pull/315


   Updates handling of CURL reply and civetweb write for RSA so that large payloads are supported.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] Oipo commented on a change in pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
Oipo commented on a change in pull request #315:
URL: https://github.com/apache/celix/pull/315#discussion_r566041648



##########
File path: bundles/remote_services/remote_service_admin_dfi/src/remote_service_admin_dfi.c
##########
@@ -505,7 +507,24 @@ static int remoteServiceAdmin_callback(struct mg_connection *conn) {
 
             if (rc == CELIX_SUCCESS && response != NULL) {
                 mg_write(conn, data_response_headers, strlen(data_response_headers));
-                mg_write(conn, response, strlen(response));
+
+                char *bufLoc = response;
+                size_t bytesLeft = strlen(response);
+                if (bytesLeft > INT_MAX) {
+                    //NOTE arcording to civetweb mg_write, there is a limit on mg_write for INT_MAX.
+                    RSA_LOG_WARNING(rsa, "nr of bytes to send for a remote call is > INT_MAX, this can lead to issues\n");
+                }
+                while (bytesLeft > 0) {
+                    int send = mg_write(conn, response, strlen(response));

Review comment:
       Probably because you're testing locally. This then bypasses the kernel's TCP stack. To truly test it, you need to have two separate devices connected to each other.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] Oipo commented on a change in pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
Oipo commented on a change in pull request #315:
URL: https://github.com/apache/celix/pull/315#discussion_r565956202



##########
File path: bundles/remote_services/remote_service_admin_dfi/gtest/src/tst_activator.c
##########
@@ -135,12 +135,30 @@ static bool bndTestRemoteString(void *handle) {
     pthread_mutex_lock(&act->mutex);
     if (act->remoteExample != NULL) {
         //test string Call with taking ownership
-        char *tmp = strndup("test1", 1024);
+
+        //test with a large very large string to verify mg_write limits.
+        int testLength = 1024 * 1024 * 5; //5mb
+        char *buf = NULL;
+        size_t bufLen = 0;
+        FILE* stream = open_memstream(&buf, &bufLen);
+        for (int i =0; i < testLength; i++) {
+            fputc('A', stream);
+        }
+        fputc('\0', stream);
+        fclose(stream);
         char *result = NULL;
-        TIMED_EXPR(act->remoteExample->setName1(act->remoteExample->handle, tmp, &result));
+        TIMED_EXPR(act->remoteExample->setName1(act->remoteExample->handle, buf, &result));
         printf("Call setName1 took %f ms\n", diff);
         //note setName1 should take ownership of tmp, so no free(tmp) needed.
-        ok = strncmp("test1", result, 1024) == 0;
+        ok = strncmp("AAAA", result, 4) == 0;

Review comment:
       This doesn´t check if result has more than 4 characters anymore.

##########
File path: bundles/remote_services/remote_service_admin_dfi/src/remote_service_admin_dfi.c
##########
@@ -505,7 +507,24 @@ static int remoteServiceAdmin_callback(struct mg_connection *conn) {
 
             if (rc == CELIX_SUCCESS && response != NULL) {
                 mg_write(conn, data_response_headers, strlen(data_response_headers));
-                mg_write(conn, response, strlen(response));
+
+                char *bufLoc = response;
+                size_t bytesLeft = strlen(response);
+                if (bytesLeft > INT_MAX) {
+                    //NOTE arcording to civetweb mg_write, there is a limit on mg_write for INT_MAX.
+                    RSA_LOG_WARNING(rsa, "nr of bytes to send for a remote call is > INT_MAX, this can lead to issues\n");
+                }
+                while (bytesLeft > 0) {
+                    int send = mg_write(conn, response, strlen(response));

Review comment:
       This sends the same part of the response (from beginning till `send`) continuously. You probably meant `int send = mg_write(conn, response + bufLoc, bytesLeft-bufLoc)`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] pnoltes commented on a change in pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
pnoltes commented on a change in pull request #315:
URL: https://github.com/apache/celix/pull/315#discussion_r565996840



##########
File path: bundles/remote_services/remote_service_admin_dfi/src/remote_service_admin_dfi.c
##########
@@ -505,7 +507,24 @@ static int remoteServiceAdmin_callback(struct mg_connection *conn) {
 
             if (rc == CELIX_SUCCESS && response != NULL) {
                 mg_write(conn, data_response_headers, strlen(data_response_headers));
-                mg_write(conn, response, strlen(response));
+
+                char *bufLoc = response;
+                size_t bytesLeft = strlen(response);
+                if (bytesLeft > INT_MAX) {
+                    //NOTE arcording to civetweb mg_write, there is a limit on mg_write for INT_MAX.
+                    RSA_LOG_WARNING(rsa, "nr of bytes to send for a remote call is > INT_MAX, this can lead to issues\n");
+                }
+                while (bytesLeft > 0) {
+                    int send = mg_write(conn, response, strlen(response));

Review comment:
       ouch. yes this should be bufLoc instead of response, because this got updated.
   But I could not trigger the need for multiple mg_write (although the document implies that this could be needed.).
   I tried to send 100mb and 1000mb of data and still one mg_write was enough. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] codecov-io edited a comment on pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #315:
URL: https://github.com/apache/celix/pull/315#issuecomment-769134520


   # [Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=h1) Report
   > Merging [#315](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=desc) (1079926) into [master](https://codecov.io/gh/apache/celix/commit/afdb357a6c86773acf51840021ebbac1a2618ebd?el=desc) (afdb357) will **increase** coverage by `0.01%`.
   > The diff coverage is `83.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/celix/pull/315/graphs/tree.svg?width=650&height=150&src=pr&token=JdsiThga8P)](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #315      +/-   ##
   ==========================================
   + Coverage   68.69%   68.70%   +0.01%     
   ==========================================
     Files         149      149              
     Lines       30639    30647       +8     
   ==========================================
   + Hits        21046    21057      +11     
   + Misses       9593     9590       -3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...te\_service\_admin\_dfi/src/import\_registration\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9pbXBvcnRfcmVnaXN0cmF0aW9uX2RmaS5j) | `75.87% <ø> (ø)` | |
   | [...te\_service\_admin\_dfi/src/export\_registration\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9leHBvcnRfcmVnaXN0cmF0aW9uX2RmaS5j) | `81.38% <75.00%> (+0.20%)` | :arrow_up: |
   | [...e\_service\_admin\_dfi/src/remote\_service\_admin\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9yZW1vdGVfc2VydmljZV9hZG1pbl9kZmkuYw==) | `83.49% <85.00%> (+0.19%)` | :arrow_up: |
   | [libs/utils/src/hash\_map.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-bGlicy91dGlscy9zcmMvaGFzaF9tYXAuYw==) | `93.37% <0.00%> (+0.28%)` | :arrow_up: |
   | [...dmin\_websocket/src/pubsub\_websocket\_topic\_sender.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9wdWJzdWIvcHVic3ViX2FkbWluX3dlYnNvY2tldC9zcmMvcHVic3ViX3dlYnNvY2tldF90b3BpY19zZW5kZXIuYw==) | `83.24% <0.00%> (+1.11%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=footer). Last update [afdb357...1079926](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] codecov-io commented on pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #315:
URL: https://github.com/apache/celix/pull/315#issuecomment-769134520


   # [Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=h1) Report
   > Merging [#315](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=desc) (1079926) into [master](https://codecov.io/gh/apache/celix/commit/afdb357a6c86773acf51840021ebbac1a2618ebd?el=desc) (afdb357) will **increase** coverage by `0.01%`.
   > The diff coverage is `83.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/celix/pull/315/graphs/tree.svg?width=650&height=150&src=pr&token=JdsiThga8P)](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #315      +/-   ##
   ==========================================
   + Coverage   68.69%   68.70%   +0.01%     
   ==========================================
     Files         149      149              
     Lines       30639    30647       +8     
   ==========================================
   + Hits        21046    21055       +9     
   + Misses       9593     9592       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...te\_service\_admin\_dfi/src/import\_registration\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9pbXBvcnRfcmVnaXN0cmF0aW9uX2RmaS5j) | `75.87% <ø> (ø)` | |
   | [...te\_service\_admin\_dfi/src/export\_registration\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9leHBvcnRfcmVnaXN0cmF0aW9uX2RmaS5j) | `81.38% <75.00%> (+0.20%)` | :arrow_up: |
   | [...e\_service\_admin\_dfi/src/remote\_service\_admin\_dfi.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-YnVuZGxlcy9yZW1vdGVfc2VydmljZXMvcmVtb3RlX3NlcnZpY2VfYWRtaW5fZGZpL3NyYy9yZW1vdGVfc2VydmljZV9hZG1pbl9kZmkuYw==) | `83.49% <85.00%> (+0.19%)` | :arrow_up: |
   | [libs/utils/src/hash\_map.c](https://codecov.io/gh/apache/celix/pull/315/diff?src=pr&el=tree#diff-bGlicy91dGlscy9zcmMvaGFzaF9tYXAuYw==) | `93.37% <0.00%> (+0.28%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=footer). Last update [afdb357...1079926](https://codecov.io/gh/apache/celix/pull/315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] pnoltes commented on a change in pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
pnoltes commented on a change in pull request #315:
URL: https://github.com/apache/celix/pull/315#discussion_r565995432



##########
File path: bundles/remote_services/remote_service_admin_dfi/gtest/src/tst_activator.c
##########
@@ -135,12 +135,30 @@ static bool bndTestRemoteString(void *handle) {
     pthread_mutex_lock(&act->mutex);
     if (act->remoteExample != NULL) {
         //test string Call with taking ownership
-        char *tmp = strndup("test1", 1024);
+
+        //test with a large very large string to verify mg_write limits.
+        int testLength = 1024 * 1024 * 5; //5mb
+        char *buf = NULL;
+        size_t bufLen = 0;
+        FILE* stream = open_memstream(&buf, &bufLen);
+        for (int i =0; i < testLength; i++) {
+            fputc('A', stream);
+        }
+        fputc('\0', stream);
+        fclose(stream);
         char *result = NULL;
-        TIMED_EXPR(act->remoteExample->setName1(act->remoteExample->handle, tmp, &result));
+        TIMED_EXPR(act->remoteExample->setName1(act->remoteExample->handle, buf, &result));
         printf("Call setName1 took %f ms\n", diff);
         //note setName1 should take ownership of tmp, so no free(tmp) needed.
-        ok = strncmp("test1", result, 1024) == 0;
+        ok = strncmp("AAAA", result, 4) == 0;

Review comment:
       See check below:
   `ok = strlen(result) == testLength;`
   this test if the length is the same




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] pnoltes merged pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
pnoltes merged pull request #315:
URL: https://github.com/apache/celix/pull/315


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [celix] pnoltes commented on a change in pull request #315: Adds test and improvements for remote service calls for large payloads

Posted by GitBox <gi...@apache.org>.
pnoltes commented on a change in pull request #315:
URL: https://github.com/apache/celix/pull/315#discussion_r566160515



##########
File path: bundles/remote_services/remote_service_admin_dfi/src/remote_service_admin_dfi.c
##########
@@ -505,7 +507,24 @@ static int remoteServiceAdmin_callback(struct mg_connection *conn) {
 
             if (rc == CELIX_SUCCESS && response != NULL) {
                 mg_write(conn, data_response_headers, strlen(data_response_headers));
-                mg_write(conn, response, strlen(response));
+
+                char *bufLoc = response;
+                size_t bytesLeft = strlen(response);
+                if (bytesLeft > INT_MAX) {
+                    //NOTE arcording to civetweb mg_write, there is a limit on mg_write for INT_MAX.
+                    RSA_LOG_WARNING(rsa, "nr of bytes to send for a remote call is > INT_MAX, this can lead to issues\n");
+                }
+                while (bytesLeft > 0) {
+                    int send = mg_write(conn, response, strlen(response));

Review comment:
       updated
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org