You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "ruojieranyishen (via GitHub)" <gi...@apache.org> on 2024/12/03 02:47:43 UTC

[PR] Refactor(admin-cli): support close client [incubator-pegasus]

ruojieranyishen opened a new pull request, #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162

   https://github.com/apache/incubator-pegasus/issues/2160
   
   Current admin-cli client does not provide an interface to close replica server TCP connections.
   And `QueryAllNodesDiskInfo` function is difficult to use. `SendQueryDiskInfoRequest` function is private.
   
   ### What problem does this PR solve? <!--add issue link with summary if exists-->
   support close client, disconnect tcp link with replica server.
   
   ##### Tests <!-- At least one of them must be included. -->
   - Manual test (add detailed scripts or steps below)
   The TCP connection to the replica server cannot be closed.
   ![image](https://github.com/user-attachments/assets/138c34af-66d7-4cd2-9362-084e2adc2471)
   
   After support close client,the persistent TCP connection between admincli and the replica server can be disconnected.
   ![image](https://github.com/user-attachments/assets/de495f7e-8eaf-4f96-9917-e143f55fa1b0)
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1877048753


##########
admin-cli/executor/disk_info.go:
##########
@@ -97,6 +99,31 @@ func QueryAllNodesDiskInfo(client *Client, tableName string) (map[string]*radmin
 	return respMap, nil
 }
 
+func QueryAliveNodesDiskInfo(client *Client, tableName string) (map[string]*radmin.QueryDiskInfoResponse, error) {

Review Comment:
   Thinks, I have removed the function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1899937708


##########
admin-cli/executor/client.go:
##########
@@ -65,5 +70,24 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Meta:   meta,
 		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
 		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(client *Client) error {
+	var errorStrings []string
+	err := client.Meta.Close()
+	if err != nil {
+		fmt.Printf("Error: failed to close meta session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())

Review Comment:
   Thinks, I agree and modify code. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1869788786


##########
admin-cli/executor/client.go:
##########
@@ -67,3 +69,46 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Perf:   aggregate.NewPerfClient(metaAddrs),
 	}
 }
+
+// NewClientWithoutExit creates a client for accessing Pegasus cluster for use of admin-cli.
+// This function will not call os.Exit.
+func NewClientWithoutExit(writer io.Writer, metaAddrs []string) (*Client, error) {
+	meta := client.NewRPCBasedMeta(metaAddrs)
+
+	nodes, err := meta.ListNodes()
+	if err != nil {
+		fmt.Fprintf(writer, "fatal: failed to list nodes [%s]\n", err)

Review Comment:
   The "fatal" level log is conventional considered as unacceptable condition, and will be  used before exit the process. It's better to use error or warning level here.



##########
admin-cli/executor/client.go:
##########
@@ -67,3 +69,46 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Perf:   aggregate.NewPerfClient(metaAddrs),
 	}
 }
+
+// NewClientWithoutExit creates a client for accessing Pegasus cluster for use of admin-cli.
+// This function will not call os.Exit.
+func NewClientWithoutExit(writer io.Writer, metaAddrs []string) (*Client, error) {

Review Comment:
   This fucntion is similar to the function above, how about unify them?



##########
admin-cli/executor/disk_info.go:
##########
@@ -97,6 +99,31 @@ func QueryAllNodesDiskInfo(client *Client, tableName string) (map[string]*radmin
 	return respMap, nil
 }
 
+func QueryAliveNodesDiskInfo(client *Client, tableName string) (map[string]*radmin.QueryDiskInfoResponse, error) {

Review Comment:
   Is it related to the subject of this patch?



##########
admin-cli/executor/client.go:
##########
@@ -67,3 +69,46 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Perf:   aggregate.NewPerfClient(metaAddrs),
 	}
 }
+
+// NewClientWithoutExit creates a client for accessing Pegasus cluster for use of admin-cli.
+// This function will not call os.Exit.
+func NewClientWithoutExit(writer io.Writer, metaAddrs []string) (*Client, error) {
+	meta := client.NewRPCBasedMeta(metaAddrs)
+
+	nodes, err := meta.ListNodes()
+	if err != nil {
+		fmt.Fprintf(writer, "fatal: failed to list nodes [%s]\n", err)
+		return nil, fmt.Errorf("fatal: failed to list nodes [%s]", err)
+	}
+
+	var replicaAddrs []string
+	for _, node := range nodes {
+		replicaAddrs = append(replicaAddrs, node.Address.GetAddress())
+	}
+
+	return &Client{
+		Writer: writer,
+		Meta:   meta,
+		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
+		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(writer io.Writer, client *Client) error {

Review Comment:
   Where are the functions used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "empiredan (via GitHub)" <gi...@apache.org>.
empiredan commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1899899250


##########
admin-cli/executor/client.go:
##########
@@ -65,5 +70,24 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Meta:   meta,
 		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
 		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(client *Client) error {
+	var errorStrings []string
+	err := client.Meta.Close()
+	if err != nil {
+		fmt.Printf("Error: failed to close meta session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())

Review Comment:
   What are the possible errors occurring here ? I think they could just be ignored so we would continue to close other components as belows ?



##########
admin-cli/executor/client.go:
##########
@@ -65,5 +70,24 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Meta:   meta,
 		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
 		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(client *Client) error {
+	var errorStrings []string
+	err := client.Meta.Close()
+	if err != nil {
+		fmt.Printf("Error: failed to close meta session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	client.Perf.Close()
+
+	err = client.Nodes.CloseAllNodes()
+	if err != nil {
+		fmt.Printf("Error: failed to close nodes session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())
 	}
+
+	return fmt.Errorf("%s", strings.Join(errorStrings, "\n"))

Review Comment:
   Just return `nil` if there is not any error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1872510306


##########
admin-cli/executor/client.go:
##########
@@ -67,3 +69,46 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Perf:   aggregate.NewPerfClient(metaAddrs),
 	}
 }
+
+// NewClientWithoutExit creates a client for accessing Pegasus cluster for use of admin-cli.
+// This function will not call os.Exit.
+func NewClientWithoutExit(writer io.Writer, metaAddrs []string) (*Client, error) {
+	meta := client.NewRPCBasedMeta(metaAddrs)
+
+	nodes, err := meta.ListNodes()
+	if err != nil {
+		fmt.Fprintf(writer, "fatal: failed to list nodes [%s]\n", err)
+		return nil, fmt.Errorf("fatal: failed to list nodes [%s]", err)
+	}
+
+	var replicaAddrs []string
+	for _, node := range nodes {
+		replicaAddrs = append(replicaAddrs, node.Address.GetAddress())
+	}
+
+	return &Client{
+		Writer: writer,
+		Meta:   meta,
+		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
+		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(writer io.Writer, client *Client) error {

Review Comment:
   The context is that I am estimating the required disk space for a bulk load on the Pegasus gateway using admincli, but currently, admincli does not support disconnect tcp from the replica server.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1872507606


##########
admin-cli/executor/disk_info.go:
##########
@@ -97,6 +99,31 @@ func QueryAllNodesDiskInfo(client *Client, tableName string) (map[string]*radmin
 	return respMap, nil
 }
 
+func QueryAliveNodesDiskInfo(client *Client, tableName string) (map[string]*radmin.QueryDiskInfoResponse, error) {

Review Comment:
   The original QueryNodesDiskInfo was not user-friendly and couldn't handle certain situations, such as missing table partition on nodes or some nodes being unalive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 merged PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "empiredan (via GitHub)" <gi...@apache.org>.
empiredan commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1883241486


##########
admin-cli/executor/client.go:
##########
@@ -45,14 +46,18 @@ type Client struct {
 }
 
 // NewClient creates a client for accessing Pegasus cluster for use of admin-cli.
-func NewClient(writer io.Writer, metaAddrs []string) *Client {
+// When listing nodes fails, willExit == true means call os.Exit().
+func NewClient(writer io.Writer, metaAddrs []string, willExit bool) (*Client, error) {

Review Comment:
   When `willExit` would be set `false` ? Is it necessary to add the param ?
   
   And returned error is also be ignored, is it needed ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1872505247


##########
admin-cli/executor/client.go:
##########
@@ -67,3 +69,46 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Perf:   aggregate.NewPerfClient(metaAddrs),
 	}
 }
+
+// NewClientWithoutExit creates a client for accessing Pegasus cluster for use of admin-cli.
+// This function will not call os.Exit.
+func NewClientWithoutExit(writer io.Writer, metaAddrs []string) (*Client, error) {

Review Comment:
   Thinks, i will unify them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1899937176


##########
admin-cli/executor/client.go:
##########
@@ -65,5 +70,24 @@ func NewClient(writer io.Writer, metaAddrs []string) *Client {
 		Meta:   meta,
 		Nodes:  util.NewPegasusNodeManager(metaAddrs, replicaAddrs),
 		Perf:   aggregate.NewPerfClient(metaAddrs),
+	}, nil
+}
+
+func CloseClient(client *Client) error {
+	var errorStrings []string
+	err := client.Meta.Close()
+	if err != nil {
+		fmt.Printf("Error: failed to close meta session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	client.Perf.Close()
+
+	err = client.Nodes.CloseAllNodes()
+	if err != nil {
+		fmt.Printf("Error: failed to close nodes session [%s]\n", err)
+		errorStrings = append(errorStrings, err.Error())
 	}
+
+	return fmt.Errorf("%s", strings.Join(errorStrings, "\n"))

Review Comment:
   Ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1872507606


##########
admin-cli/executor/disk_info.go:
##########
@@ -97,6 +99,31 @@ func QueryAllNodesDiskInfo(client *Client, tableName string) (map[string]*radmin
 	return respMap, nil
 }
 
+func QueryAliveNodesDiskInfo(client *Client, tableName string) (map[string]*radmin.QueryDiskInfoResponse, error) {

Review Comment:
   The original QueryNodesDiskInfo was not user-friendly and couldn't handle certain situations, such as missing table partition on nodes or some nodes being unalive. Should I submit a new PR?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1876278834


##########
admin-cli/executor/disk_info.go:
##########
@@ -97,6 +99,31 @@ func QueryAllNodesDiskInfo(client *Client, tableName string) (map[string]*radmin
 	return respMap, nil
 }
 
+func QueryAliveNodesDiskInfo(client *Client, tableName string) (map[string]*radmin.QueryDiskInfoResponse, error) {

Review Comment:
   Yes, submit this in another PR would be better. Keep each commit independent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] refactor: support close admin-cli client [incubator-pegasus]

Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on code in PR #2162:
URL: https://github.com/apache/incubator-pegasus/pull/2162#discussion_r1883283925


##########
admin-cli/executor/client.go:
##########
@@ -45,14 +46,18 @@ type Client struct {
 }
 
 // NewClient creates a client for accessing Pegasus cluster for use of admin-cli.
-func NewClient(writer io.Writer, metaAddrs []string) *Client {
+// When listing nodes fails, willExit == true means call os.Exit().
+func NewClient(writer io.Writer, metaAddrs []string, willExit bool) (*Client, error) {

Review Comment:
   The context is that I am estimating the required disk space for a bulk load on the Pegasus gateway using admincli, but currently, admincli does not support disconnect tcp from the replica server. 
   When using admincli to connect to the Pegasus cluster, it should not call the os.Exit function; it should only return an error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org