You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by "nanonyme (via GitHub)" <gi...@apache.org> on 2023/11/01 21:13:22 UTC

[PR] Allow integrating to new LocalCAS keepalive time [buildstream]

nanonyme opened a new pull request, #1879:
URL: https://github.com/apache/buildstream/pull/1879

   This is opt-in by user. Integration for https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/issues/89


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "abderrahim (via GitHub)" <gi...@apache.org>.
abderrahim commented on PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#issuecomment-1974840171

   Landing now that we have a bst-2.1 branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1424509271


##########
src/buildstream/_remotespec.py:
##########
@@ -176,6 +181,13 @@ def credentials(self) -> ChannelCredentials:
             )
         return self._credentials
 
+    # grpc keepalive timeout
+    #
+    @property
+    def keepalive_time(self) -> Optional[int]:
+        if self._connection_config:
+            return self._connection_config.get_int("keepalive-timeout", None)

Review Comment:
   Should this say keepalive time or timeout? I know the problem I currently have is being inconsistent. https://grpc.io/docs/guides/keepalive/ talks of keepalive timeout.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389319346


##########
src/buildstream/_remotespec.py:
##########
@@ -253,6 +266,10 @@ def new_from_node(
         if auth_node:
             server_cert, client_key, client_cert = cls._parse_auth(auth_node, basedir)
 
+        keepalive_config = spec_node.get_mapping("keepalive-config", None)
+        if keepalive_config is not None:
+            keepalive_config.validate_keys(["keepalive-time"])

Review Comment:
   This is anyway reasonably localized thing to change. The reasoning here was that full grpc keepalive spec looks like this https://grpc.io/docs/guides/keepalive/#keepalive-configuration-specification



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389333268


##########
src/buildstream/_protos/build/bazel/remote/asset/v1/remote_asset_pb2.py:
##########
@@ -4,9 +4,8 @@
 """Generated protocol buffer code."""
 from google.protobuf import descriptor as _descriptor
 from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import message as _message

Review Comment:
   Yay. grpcio-tools needed doesn't build anymore with modern g++.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "abderrahim (via GitHub)" <gi...@apache.org>.
abderrahim merged PR #1879:
URL: https://github.com/apache/buildstream/pull/1879


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "juergbi (via GitHub)" <gi...@apache.org>.
juergbi commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1424429574


##########
src/buildstream/_remotespec.py:
##########
@@ -112,6 +113,9 @@ def __init__(
         # The grpc credentials object
         self._credentials: Optional[ChannelCredentials] = None
 
+        # Various connection parameters for grpc connection
+        self._connection_config: Optional[int] = connection_config

Review Comment:
   ```suggestion
           self._connection_config: Optional[MappingNode] = connection_config
   ```



##########
src/buildstream/_remotespec.py:
##########
@@ -176,6 +181,13 @@ def credentials(self) -> ChannelCredentials:
             )
         return self._credentials
 
+    # grpc keepalive timeout
+    #
+    @property
+    def keepalive_time(self) -> Optional[int]:
+        if self._connection_config:
+            return self._connection_config.get_int("keepalive-timeout", None)

Review Comment:
   ```suggestion
               return self._connection_config.get_int("keepalive-time", None)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "abderrahim (via GitHub)" <gi...@apache.org>.
abderrahim commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389025273


##########
src/buildstream/_remotespec.py:
##########
@@ -253,6 +266,10 @@ def new_from_node(
         if auth_node:
             server_cert, client_key, client_cert = cls._parse_auth(auth_node, basedir)
 
+        keepalive_config = spec_node.get_mapping("keepalive-config", None)
+        if keepalive_config is not None:
+            keepalive_config.validate_keys(["keepalive-time"])

Review Comment:
   I think this is too much keepalive. We have only one keepalive setting, so no need to put it in a keepalive-config dictionary. We should either have it directly in the server config or (and I think this would be better) in a generic dictionary for connection settings that allows adding things later.



##########
src/buildstream/_protos/build/bazel/remote/asset/v1/remote_asset_pb2.py:
##########
@@ -4,9 +4,8 @@
 """Generated protocol buffer code."""
 from google.protobuf import descriptor as _descriptor
 from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import message as _message

Review Comment:
   I think we were using a specific version of protobuf compiler to generate these so that they work with both old and new grpc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1425641207


##########
src/buildstream/_remotespec.py:
##########
@@ -112,6 +113,9 @@ def __init__(
         # The grpc credentials object
         self._credentials: Optional[ChannelCredentials] = None
 
+        # Various connection parameters for grpc connection
+        self._connection_config: Optional[int] = connection_config

Review Comment:
   I fixed this wrong, now it's correct.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1425960472


##########
src/buildstream/_remotespec.py:
##########
@@ -227,10 +240,11 @@ def new_from_node(
         server_cert: Optional[str] = None
         client_key: Optional[str] = None
         client_cert: Optional[str] = None
+        keepalive_config: Optional[MappingNode] = None

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389097497


##########
src/buildstream/_protos/build/bazel/remote/asset/v1/remote_asset_pb2.py:
##########
@@ -4,9 +4,8 @@
 """Generated protocol buffer code."""
 from google.protobuf import descriptor as _descriptor
 from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import message as _message

Review Comment:
   Is it documented which version this was?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#issuecomment-1974894399

   Opening up on what @juergbi said above:  it was decided this goes out in 2.2.x since it allows projects to require 2.2 as minimum version of BuildStream (and take this feature into use).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1426479819


##########
src/buildstream/_cas/casremote.py:
##########
@@ -13,6 +13,7 @@
 #
 
 import grpc
+from google.protobuf.duration_pb2 import Duration

Review Comment:
   Now it should be correct suppression no-name-in-module



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#issuecomment-1855547193

       [00:02:11] FAILURE [ffeab612] hello.bst: tar source at hello.bst [line 16 column 2]: Error mirroring http://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [Errno 99] Cannot assign requested address>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389327136


##########
src/buildstream/_protos/build/bazel/remote/asset/v1/remote_asset_pb2.py:
##########
@@ -4,9 +4,8 @@
 """Generated protocol buffer code."""
 from google.protobuf import descriptor as _descriptor
 from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import message as _message

Review Comment:
   Looks like it was 3.20.1. This is btw *really* dangerous, the grpcio-tools that is compatible with this old protobuf is getting increasingly hard to install. It would be more maintainable to drop support for older grpc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389327136


##########
src/buildstream/_protos/build/bazel/remote/asset/v1/remote_asset_pb2.py:
##########
@@ -4,9 +4,8 @@
 """Generated protocol buffer code."""
 from google.protobuf import descriptor as _descriptor
 from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import message as _message

Review Comment:
   Looks like it was 3.20.1. This is btw *really* dangerous, the grpcio-tools that is compatible with this old protobuf is getting increasingly hard to install. It would be better to drop support for older grpc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1424499867


##########
src/buildstream/_remotespec.py:
##########
@@ -112,6 +113,9 @@ def __init__(
         # The grpc credentials object
         self._credentials: Optional[ChannelCredentials] = None
 
+        # Various connection parameters for grpc connection
+        self._connection_config: Optional[int] = connection_config

Review Comment:
   You are right, if this is set in project config, then it clearly needs a version bump. I was assuming (perhaps incorrectly) that the most useful way to use this is through user config where you override project CAS with your own config and set timeout there if you're having connection issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "abderrahim (via GitHub)" <gi...@apache.org>.
abderrahim commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1424546304


##########
src/buildstream/_remotespec.py:
##########
@@ -176,6 +181,13 @@ def credentials(self) -> ChannelCredentials:
             )
         return self._credentials
 
+    # grpc keepalive timeout
+    #
+    @property
+    def keepalive_time(self) -> Optional[int]:
+        if self._connection_config:
+            return self._connection_config.get_int("keepalive-timeout", None)

Review Comment:
   If you look carefully, it talks about both KEEPALIVE_TIME and KEEPALIVE_TIMEOUT (The time between pings vs how long to wait for a response to a ping). 
   
   This is about the keepalive time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "abderrahim (via GitHub)" <gi...@apache.org>.
abderrahim commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1425760907


##########
src/buildstream/_remotespec.py:
##########
@@ -227,10 +240,11 @@ def new_from_node(
         server_cert: Optional[str] = None
         client_key: Optional[str] = None
         client_cert: Optional[str] = None
+        keepalive_config: Optional[MappingNode] = None

Review Comment:
   linter is complaining this is unused.



##########
src/buildstream/_cas/casremote.py:
##########
@@ -13,6 +13,7 @@
 #
 
 import grpc
+from google.protobuf.duration_pb2 import Duration

Review Comment:
   I don't know if it's fatal, but linter is complaining about this. Maybe we need to add some dependency?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1389096388


##########
src/buildstream/_remotespec.py:
##########
@@ -253,6 +266,10 @@ def new_from_node(
         if auth_node:
             server_cert, client_key, client_cert = cls._parse_auth(auth_node, basedir)
 
+        keepalive_config = spec_node.get_mapping("keepalive-config", None)
+        if keepalive_config is not None:
+            keepalive_config.validate_keys(["keepalive-time"])

Review Comment:
   This was suggestion by @juergbi if I understood right. There are about ten keepalive settings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1416247838


##########
src/buildstream/_remotespec.py:
##########
@@ -253,6 +266,10 @@ def new_from_node(
         if auth_node:
             server_cert, client_key, client_cert = cls._parse_auth(auth_node, basedir)
 
+        keepalive_config = spec_node.get_mapping("keepalive-config", None)
+        if keepalive_config is not None:
+            keepalive_config.validate_keys(["keepalive-time"])

Review Comment:
   I converted this to more generic connection-config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "jjardon (via GitHub)" <gi...@apache.org>.
jjardon commented on PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#issuecomment-1937116624

   @juergbi @abderrahim @gtristan would it be possible to get this merged please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#issuecomment-1852714288

   Is there in addition to https://docs.buildstream.build/master/using_config.html#cache-control another place where this should be mentioned?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow integrating to new LocalCAS keepalive time [buildstream]

Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #1879:
URL: https://github.com/apache/buildstream/pull/1879#discussion_r1425960720


##########
src/buildstream/_cas/casremote.py:
##########
@@ -13,6 +13,7 @@
 #
 
 import grpc
+from google.protobuf.duration_pb2 import Duration

Review Comment:
   I tried to suppress this. Remains to be seen if it was successful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org