You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "walkinggo (via GitHub)" <gi...@apache.org> on 2025/02/11 13:46:37 UTC

[PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

walkinggo opened a new pull request, #23959:
URL: https://github.com/apache/pulsar/pull/23959

   <!--
   ### Contribution Checklist
     
     - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*. 
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   -->
   
   <!-- Either this PR fixes an issue, -->
   
   
   <!-- or this PR is one task of an issue -->
   
   Main Issue: #23723
   
   <!-- If the PR belongs to a PIP, please add the PIP link here -->
   
   
   <!-- Details of when a PIP is required and how the PIP process work, please see: https://github.com/apache/pulsar/blob/master/pip/README.md -->
   
   ### Motivation
   
   <!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
   see #23723 . The main work of this PR is to add a method for setting the memory limit of the Pulsar client in a Pulsar Function.
   
   ### Modifications
   
   <!-- Describe the modifications you've done. -->
   1.Memory Limit Support:
   Added MemoryLimit class to define memory constraints for Pulsar clients, which includes both an absolute value and a percentage of the maximum direct memory.
   This memory configuration is applied across several components including FunctionConfig, WorkerConfig, InstanceConfig, and ThreadRuntimeFactory.
   2.Class Updates:
   Introduced new fields in configuration classes such as FunctionConfig, WorkerConfig, and InstanceConfig to hold memory limit settings.
   Updated ThreadRuntimeFactory to handle the memory limit during initialization, ensuring that the client memory limit is respected.
   3.Proto File Changes:
   Updated Function.proto to include a new MemoryLimit field, allowing the configuration of memory limits directly via function details.
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/walkinggo/pulsar/pull/8
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951207891


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;
+import java.util.Optional;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.
+ */
+public class MemoryLimit {
+    Long absoluteValue;
+    Double percentOfMaxDirectMemory;
+    public static Optional<Long> calculateClientMemoryLimit(Optional<MemoryLimit> memoryLimit) {
+        if (memoryLimit.isPresent()) {
+
+            Long absolute = memoryLimit.get().getAbsoluteValue();
+            Double percentOfDirectMem = memoryLimit.get().getPercentOfMaxDirectMemory();
+            if (absolute != null) {
+                checkArgument(absolute > 0, "Absolute memory limit for Pulsar client has to be positive");
+            }
+            if (percentOfDirectMem != null) {
+                checkArgument(percentOfDirectMem > 0 && percentOfDirectMem <= 100,
+                        "Percent of max direct memory limit for Pulsar client must be between 0 and 100");
+            }
+
+            if (absolute != null && percentOfDirectMem != null) {
+                return Optional.of(Math.min(absolute, getBytesPercentDirectMem(percentOfDirectMem)));
+            }
+
+            if (absolute != null) {
+                return Optional.of(absolute);
+            }
+
+            if (percentOfDirectMem != null) {
+                return Optional.of(getBytesPercentDirectMem(percentOfDirectMem));
+            }
+        }
+        return Optional.empty();
+    }
+
+    private static long getBytesPercentDirectMem(double percent) {
+        return (long) (DirectMemoryUtils.jvmMaxDirectMemory() * (percent / 100));
+    }

Review Comment:
   Couldn't this method be moved to `DirectMemoryUtils` with a reasonable name , for example `jvmMaxDirectMemoryPercentage`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952432582


##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -98,8 +98,16 @@ message FunctionDetails {
     bool retainOrdering = 21;
     bool retainKeyOrdering = 22;
     SubscriptionPosition subscriptionPosition = 23;
+    MemoryLimit memoryLimit = 24;
 }
 
+message MemoryLimit {
+    // absolute value in bytes, represents the memory limit in bytes
+    int64 absoluteValue = 1;

Review Comment:
   > why is this called `absoluteValue` if it's about the memory limit in bytes? a better name is `limitInBytes` or picking a name that is also used in existing code.
   
   I think you're right. I have renamed the variable accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#issuecomment-2655630111

   > > > @walkinggo Please first make sure that the CI build passes in your own fork. I see that [walkinggo#8](https://github.com/walkinggo/pulsar/pull/8) referenced in the PR description isn't passing.
   > > 
   > > 
   > > The error in my PR is related to json-smart, but my code doesn't involve modifications to this component. I suspect there might be an issue with Maven Central, and I also noticed that this [PR](https://github.com/apache/pulsar/pull/23966#pullrequestreview-2611658179) is updating the version of json-smart.
   > 
   > @walkinggo PR #23966 has been merged. Please merge origin/master to your PR to get the changes included.
   
   The PR has been successfully merged and passed the CI checks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951205023


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;
+import java.util.Optional;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.
+ */
+public class MemoryLimit {
+    Long absoluteValue;
+    Double percentOfMaxDirectMemory;
+    public static Optional<Long> calculateClientMemoryLimit(Optional<MemoryLimit> memoryLimit) {
+        if (memoryLimit.isPresent()) {
+
+            Long absolute = memoryLimit.get().getAbsoluteValue();
+            Double percentOfDirectMem = memoryLimit.get().getPercentOfMaxDirectMemory();
+            if (absolute != null) {
+                checkArgument(absolute > 0, "Absolute memory limit for Pulsar client has to be positive");
+            }
+            if (percentOfDirectMem != null) {
+                checkArgument(percentOfDirectMem > 0 && percentOfDirectMem <= 100,
+                        "Percent of max direct memory limit for Pulsar client must be between 0 and 100");
+            }
+
+            if (absolute != null && percentOfDirectMem != null) {
+                return Optional.of(Math.min(absolute, getBytesPercentDirectMem(percentOfDirectMem)));
+            }
+
+            if (absolute != null) {
+                return Optional.of(absolute);
+            }
+
+            if (percentOfDirectMem != null) {
+                return Optional.of(getBytesPercentDirectMem(percentOfDirectMem));
+            }
+        }
+        return Optional.empty();
+    }

Review Comment:
   What's the point of this static method? Why not have an instance method directly in the class to do the calculation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951199995


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;

Review Comment:
   this import doesn't make sense. Use `com.google.common.base.Preconditions.checkArgument` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951215801


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;
+import java.util.Optional;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.
+ */
+public class MemoryLimit {
+    Long absoluteValue;
+    Double percentOfMaxDirectMemory;
+    public static Optional<Long> calculateClientMemoryLimit(Optional<MemoryLimit> memoryLimit) {
+        if (memoryLimit.isPresent()) {
+
+            Long absolute = memoryLimit.get().getAbsoluteValue();
+            Double percentOfDirectMem = memoryLimit.get().getPercentOfMaxDirectMemory();
+            if (absolute != null) {
+                checkArgument(absolute > 0, "Absolute memory limit for Pulsar client has to be positive");
+            }
+            if (percentOfDirectMem != null) {
+                checkArgument(percentOfDirectMem > 0 && percentOfDirectMem <= 100,
+                        "Percent of max direct memory limit for Pulsar client must be between 0 and 100");
+            }
+
+            if (absolute != null && percentOfDirectMem != null) {
+                return Optional.of(Math.min(absolute, getBytesPercentDirectMem(percentOfDirectMem)));
+            }
+
+            if (absolute != null) {
+                return Optional.of(absolute);
+            }
+
+            if (percentOfDirectMem != null) {
+                return Optional.of(getBytesPercentDirectMem(percentOfDirectMem));
+            }
+        }
+        return Optional.empty();
+    }

Review Comment:
   If this was an instance method, you could pass the direct memory size as a parameter to the instance method if you'd like to capsulate the logic here. However it seems unnecessary in the first place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952299178


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   Although I suggested making it an instance method as the first thought, it seems that there's no need to have the calculation together with the MemoryLimit class. The calculation could be handled when the value is needed. The MemoryLimit class could be kept as a plain data transfer object (DTO).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951223400


##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -98,8 +98,16 @@ message FunctionDetails {
     bool retainOrdering = 21;
     bool retainKeyOrdering = 22;
     SubscriptionPosition subscriptionPosition = 23;
+    MemoryLimit memoryLimit = 24;
 }
 
+message MemoryLimit {
+    // absolute value in bytes, represents the memory limit in bytes
+    int64 absoluteValue = 1;

Review Comment:
   Oh, I see that this was previously called `absoluteValue`. Let's improve and rename it to a meaningful name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952393249


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   Thank you for your help. They indeed aren't tightly coupled, so I have moved `calculateClientMemoryLimit` back to its original location and also restored `DirectMemoryUtils` to its original position. This way, there's no need to add the Netty dependency to the `pom.xml` file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #23959:
URL: https://github.com/apache/pulsar/pull/23959


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#issuecomment-2653476156

   @walkinggo Please first make sure that the CI build passes in your own fork. I see that https://github.com/walkinggo/pulsar/pull/8 referenced in the PR description isn't passing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1950905709


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   I’m not sure where I should place this utility class that is meant to be used by multiple components.
   
   > Please avoid introducing Netty dependencies to the API module
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#issuecomment-2654066800

   > > @walkinggo Please first make sure that the CI build passes in your own fork. I see that [walkinggo#8](https://github.com/walkinggo/pulsar/pull/8) referenced in the PR description isn't passing.
   > 
   > The error in my PR is related to json-smart, but my code doesn't involve modifications to this component. I suspect there might be an issue with Maven Central, and I also noticed that this [PR](https://github.com/apache/pulsar/pull/23966#pullrequestreview-2611658179) is updating the version of json-smart.
   
   @walkinggo PR #23966 has been merged. Please merge origin/master to your PR to get the changes included.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952311564


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;
+import java.util.Optional;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.
+ */
+public class MemoryLimit {
+    Long absoluteValue;
+    Double percentOfMaxDirectMemory;
+    public static Optional<Long> calculateClientMemoryLimit(Optional<MemoryLimit> memoryLimit) {
+        if (memoryLimit.isPresent()) {
+
+            Long absolute = memoryLimit.get().getAbsoluteValue();
+            Double percentOfDirectMem = memoryLimit.get().getPercentOfMaxDirectMemory();
+            if (absolute != null) {
+                checkArgument(absolute > 0, "Absolute memory limit for Pulsar client has to be positive");
+            }
+            if (percentOfDirectMem != null) {
+                checkArgument(percentOfDirectMem > 0 && percentOfDirectMem <= 100,
+                        "Percent of max direct memory limit for Pulsar client must be between 0 and 100");
+            }
+
+            if (absolute != null && percentOfDirectMem != null) {
+                return Optional.of(Math.min(absolute, getBytesPercentDirectMem(percentOfDirectMem)));
+            }
+
+            if (absolute != null) {
+                return Optional.of(absolute);
+            }
+
+            if (percentOfDirectMem != null) {
+                return Optional.of(getBytesPercentDirectMem(percentOfDirectMem));
+            }
+        }
+        return Optional.empty();
+    }

Review Comment:
   update: https://github.com/apache/pulsar/pull/23959#discussion_r1952299178



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952261021


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   It seems that the MemoryLimit class shouldn't even depend on DirectMemoryUtils so moving it should be reverted. Related comment https://github.com/apache/pulsar/pull/23959#discussion_r1951215801



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952395643


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;

Review Comment:
   The `import` associated with `calculateClientMemoryLimit` has been moved far away, so this import is no longer needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951218865


##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -98,8 +98,16 @@ message FunctionDetails {
     bool retainOrdering = 21;
     bool retainKeyOrdering = 22;
     SubscriptionPosition subscriptionPosition = 23;
+    MemoryLimit memoryLimit = 24;
 }
 
+message MemoryLimit {
+    // absolute value in bytes, represents the memory limit in bytes
+    int64 absoluteValue = 1;

Review Comment:
   why is this called `absoluteValue` if it's about the memory limit in bytes? a better name is `limitInBytes` or picking a name that is also used in existing code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1950892792


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   Please avoid introducing Netty dependencies to the API module



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#issuecomment-2653730243

   > @walkinggo Please first make sure that the CI build passes in your own fork. I see that [walkinggo#8](https://github.com/walkinggo/pulsar/pull/8) referenced in the PR description isn't passing.
   
   The error in my PR is related to json-smart, but my code doesn't involve modifications to this component. I suspect there might be an issue with Maven Central, and I also noticed that this [PR](https://github.com/apache/pulsar/pull/23966#pullrequestreview-2611658179) is updating the version of json-smart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951211226


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/DirectMemoryUtils.java:
##########
@@ -16,7 +16,7 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.apache.pulsar.common.util;
+package org.apache.pulsar.common.functions;

Review Comment:
   Why are you moving this class. Couldn't we keep it in the existing location?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952239929


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   > > its required DirectMemoryUtils
   > 
   > I don't think `MemoryLimit` is coupled with `DirectMemoryUtils`.
   
   `DirectMemoryUtils` was originally located in `pulsar-common`. If I move only `MemoryLimit` without relocating `DirectMemoryUtils`, it would cause a circular dependency issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952498630


##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -98,8 +98,16 @@ message FunctionDetails {
     bool retainOrdering = 21;
     bool retainKeyOrdering = 22;
     SubscriptionPosition subscriptionPosition = 23;
+    MemoryLimit memoryLimit = 24;
 }
 
+message MemoryLimit {
+    // absolute value in bytes, represents the memory limit in bytes
+    int64 absoluteValue = 1;

Review Comment:
   rename field



##########
pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java:
##########
@@ -377,6 +378,11 @@ public class WorkerConfig implements Serializable, PulsarConfiguration {
         doc = "The directory to download functions by runtime manager"
     )
     private String downloadDirectory;
+    @FieldContext(
+            category = CATEGORY_FUNC_RUNTIME_MNG,
+            doc = "Memory limit set for the pulsar client used by all instances"

Review Comment:
   ```suggestion
               doc = "Default memory limit set for the pulsar client used instances"
   ```



##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.

Review Comment:
   referencing old field. Javadoc doesn't yet support markdown, so using backticks in Javadoc isn't useful.



##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/FunctionConfig.java:
##########
@@ -133,7 +133,8 @@ public enum Runtime {
     private Boolean exposePulsarAdminClientEnabled;
     // Whether the consumer should skip to latest position in case of failure recovery
     private Boolean skipToLatest;
-
+    // The momoryLimit of pulsar client

Review Comment:
   typo



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1950901931


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   The main reason I introduced it here is that I moved `MemoryLimit` and its required `DirectMemoryUtils` to this location. However, `DirectMemoryUtils` requires importing `io.netty.util.internal.PlatformDependent`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1951915000


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   >  its required DirectMemoryUtils
   
   I don't think `MemoryLimit` is coupled with `DirectMemoryUtils`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "walkinggo (via GitHub)" <gi...@apache.org>.
walkinggo commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952290227


##########
pulsar-client-admin-api/pom.xml:
##########
@@ -43,6 +43,21 @@
             <artifactId>slf4j-api</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-handler</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-resolver-dns</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-transport-native-epoll</artifactId>
+            <classifier>linux-x86_64</classifier>
+        </dependency>

Review Comment:
   
   
   
   > It seems that the MemoryLimit class shouldn't even depend on DirectMemoryUtils so moving it should be reverted. Related comment [#23959 (comment)](https://github.com/apache/pulsar/pull/23959#discussion_r1951215801)
   
   Sorry, but I'm quite confused. I spent a lot of time thinking about this issue when I was trying to submit this PR. I found that the `calculateClientMemoryLimit` method depends on the `getBytesPercentDirectMem` method, and `getBytesPercentDirectMem` actually needs to call `DirectMemoryUtils.jvmMaxDirectMemory()`. If we move `getBytesPercentDirectMem` to `DirectMemoryUtils`, the dependency from `MemoryLimit` to `DirectMemoryUtils` would still exist. 
   
   So, what you're suggesting is that we shouldn't make `calculateClientMemoryLimit` a static method of `MemoryLimit`, but instead make it an instance method, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#discussion_r1952312069


##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/functions/MemoryLimit.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.functions;
+
+import static org.apache.pulsar.common.policies.data.impl.BookieInfoImpl.BookieInfoImplBuilder.checkArgument;
+import java.util.Optional;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+@Data
+@Accessors(chain = true)
+/**
+ * Memory limit set for the pulsar client used by all instances
+ * If `absoluteValue` and `percentOfMaxDirectMemory` are both set, then the min of the two will be used.
+ */
+public class MemoryLimit {
+    Long absoluteValue;
+    Double percentOfMaxDirectMemory;
+    public static Optional<Long> calculateClientMemoryLimit(Optional<MemoryLimit> memoryLimit) {
+        if (memoryLimit.isPresent()) {
+
+            Long absolute = memoryLimit.get().getAbsoluteValue();
+            Double percentOfDirectMem = memoryLimit.get().getPercentOfMaxDirectMemory();
+            if (absolute != null) {
+                checkArgument(absolute > 0, "Absolute memory limit for Pulsar client has to be positive");
+            }
+            if (percentOfDirectMem != null) {
+                checkArgument(percentOfDirectMem > 0 && percentOfDirectMem <= 100,
+                        "Percent of max direct memory limit for Pulsar client must be between 0 and 100");
+            }
+
+            if (absolute != null && percentOfDirectMem != null) {
+                return Optional.of(Math.min(absolute, getBytesPercentDirectMem(percentOfDirectMem)));
+            }
+
+            if (absolute != null) {
+                return Optional.of(absolute);
+            }
+
+            if (percentOfDirectMem != null) {
+                return Optional.of(getBytesPercentDirectMem(percentOfDirectMem));
+            }
+        }
+        return Optional.empty();
+    }
+
+    private static long getBytesPercentDirectMem(double percent) {
+        return (long) (DirectMemoryUtils.jvmMaxDirectMemory() * (percent / 100));
+    }

Review Comment:
   update: https://github.com/apache/pulsar/pull/23959#discussion_r1952299178



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #23959:
URL: https://github.com/apache/pulsar/pull/23959#issuecomment-6228415894

   This PR had been idle since the last re-review request, so I continued it on this branch. @walkinggo's commits are kept as they were. I merged current `master` (no rebase) and added one commit on top. The changes since the last review:
   
   - **Merge with `master`:** the functions protos are now LightProto, and `FunctionDetails` field 24 is taken by `clientApi`, so the limit is field 25.
   - **The limit now reaches process/Kubernetes instances.** `FunctionActioner` uses the function's limit, or else the new `WorkerConfig` default. `RuntimeUtils.getCmd` passes it as `--pulsar_client_memory_limit_bytes` / `--pulsar_client_memory_limit_percent`, and `JavaInstanceStarter` reads these flags. The earlier `--memory_limit` option had no converter and nothing passed it to the instance.
   - **The field is renamed to `pulsarClientMemoryLimit`** in `FunctionConfig` and the proto, to match `InstanceConfig`/`WorkerConfig`. The conversions are fixed: a partially set limit no longer throws an NPE, `convertFromDetails` keeps the limit, function updates merge it, and invalid values are rejected when the function is submitted.
   - **`ThreadRuntimeFactoryConfig.MemoryLimit` (`absoluteValue`) is restored**, so existing `functions_worker.yml` files keep working. The thread runtime falls back to the worker default when it isn't set. `MemoryLimit` stays a plain DTO without a Netty dependency, as agreed in the review.
   - **Added the `pulsar-admin functions` flags** `--client-memory-limit` / `--client-memory-limit-percent`, plus tests for each part.
   
   The PR description is updated with the full write-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org