You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/01/05 07:25:27 UTC

[GitHub] [incubator-kyuubi] pan3793 opened a new pull request #1683: [BUILD] Reduce Flink local cluster memory usage

pan3793 opened a new pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683


   ### _Why are the changes needed?_
   
   Make CI stable
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778604871



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       I can not find the way to overwrite the configure defined in `$FLINK_HOME/conf/flink-conf.yaml`, thus copied the whole `$FLINK_HOME/conf` to `flink_conf`, and then modify the `flink_conf/flink-conf.yaml`, `zoo.cfg` is not required for our test, because we don't turn on HA.
   
   The current approach is copy ``$FLINK_HOME/conf` with minimal modification and keep others as-is, or do u think we should only pick the required conf files to run `bin/start-cluster`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778631104



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       I don't follow your thoughts, 'when you need to modify one config, you cp the whole dir.`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778604871



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       I can not find the way to overwrite the configure defined in `$FLINK_HOME/conf/flink-conf.yaml`, thus copied the whole `$FLINK_HOME/conf` to `flink_conf`, and then modify the `flink_conf/flink-conf.yaml`, `zoo.cfg` is not required for our test, because we don't turn on HA.
   
   The current approach is copy ``$FLINK_HOME/conf` with minimal modification and keep others as-is, or we just pick the required conf files?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778601423



##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m

Review comment:
       The only change is L44 and L51, I tuned it to use the minimal valid value. (maybe we can reduce more, but need to tune other memory confs)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778599502



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       Can zookeeper peer be skipped?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 closed pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778601165



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       why?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yanghua commented on pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#issuecomment-1009660877


   IMO, we can close this PR. WDYT? @yaooqinn @pan3793 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778604871



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       I can not find the way to overwrite the configure defined in `$FLINK_HOME/conf/flink-conf.yaml`, thus copied the whole `$FLINK_HOME/conf` to `flink_conf`, and then modify the `flink_conf/flink-conf.yaml`, `zoo.cfg` is not required for our test, because we don't turn on HA.
   
   The current approach is copy `$FLINK_HOME/conf` with minimal modification and keep others as-is, or do u think we should only pick the required conf files to run `bin/start-cluster`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#issuecomment-1005492907


   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#1683](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e542fe) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/e0d9458144e68411f1dd62afb91efd1e1ef92cc5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e0d9458) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/graphs/tree.svg?width=650&height=150&src=pr&token=925D4tb9AH&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #1683   +/-   ##
   =========================================
     Coverage     59.56%   59.56%           
     Complexity      193      193           
   =========================================
     Files           265      265           
     Lines         13088    13094    +6     
     Branches       1648     1648           
   =========================================
   + Hits           7796     7800    +4     
   - Misses         4637     4638    +1     
   - Partials        655      656    +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...he/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9yZXBsL0t5dXViaVNwYXJrSUxvb3Auc2NhbGE=) | `90.16% <0.00%> (-3.28%)` | :arrow_down: |
   | [...org/apache/kyuubi/operation/ExecuteStatement.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vRXhlY3V0ZVN0YXRlbWVudC5zY2FsYQ==) | `87.50% <0.00%> (+0.14%)` | :arrow_up: |
   | [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `97.22% <0.00%> (+0.16%)` | :arrow_up: |
   | [...org/apache/kyuubi/server/EventLoggingService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvRXZlbnRMb2dnaW5nU2VydmljZS5zY2FsYQ==) | `91.66% <0.00%> (+0.36%)` | :arrow_up: |
   | [...ala/org/apache/kyuubi/util/KyuubiHadoopUtils.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS91dGlsL0t5dXViaUhhZG9vcFV0aWxzLnNjYWxh) | `68.75% <0.00%> (+2.08%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e0d9458...5e542fe](https://codecov.io/gh/apache/incubator-kyuubi/pull/1683?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778633739



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       1. we can only use `FLINK_CONF_DIR`, there is no option like `--conf-file=xxx.yaml`
   2. I can not found a way to overwrite a property when `bin/start-cluster`
   
   any suggestion?

##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       1. we can only use `FLINK_CONF_DIR`, there is no option like `--conf-file=xxx.yaml`
   2. I can not find a way to overwrite a property when `bin/start-cluster`
   
   any suggestion?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778600519



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       Copied from `$FLINK_HOME/conf`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778601423



##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m

Review comment:
       The only change is L44 and L51, and tune it to use the minimal valid value. (may we can reduce more, but need to tune other memory confs)

##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m
+
+
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usage within the TaskManager process, including JVM metaspace and other overhead.
+
+taskmanager.memory.process.size: 1024m

Review comment:
       mark




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778601423



##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m

Review comment:
       The only change is L44 and L51, I tuned it to use the minimal valid value. (maybe we can reduce more, but need to tune other memory properties)

##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m

Review comment:
       The only change is L44 and L51, I tuned it to use the minimal valid value. (maybe we can reduce more, but need to tune other memory properties as well)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778604871



##########
File path: kyuubi-server/src/test/resources/flink_conf/zoo.cfg
##########
@@ -0,0 +1,36 @@
+################################################################################

Review comment:
       I can not find the way to overwrite the configure defined in `$FLINK_HOME/conf/flink-conf.yaml`, thus copied the whole `$FLINK_HOME/conf` to `flink_conf`, and then modify the `flink_conf/flink-conf.yaml`, `zoo.cfg` is not required for our test, because we don't turn on HA.
   
   The current approach is copy ``$FLINK_HOME/conf` with minimal modification and keep others as-is, or do u think we should only pick the required conf files?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1683: [BUILD] Reduce Flink local cluster memory usage

Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1683:
URL: https://github.com/apache/incubator-kyuubi/pull/1683#discussion_r778601423



##########
File path: kyuubi-server/src/test/resources/flink_conf/flink-conf.yaml
##########
@@ -0,0 +1,272 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+
+#==============================================================================
+# Common
+#==============================================================================
+
+# The external address of the host on which the JobManager runs and can be
+# reached by the TaskManagers and any clients which want to connect. This setting
+# is only used in Standalone mode and may be overwritten on the JobManager side
+# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
+# In high availability mode, if you use the bin/start-cluster.sh script and setup
+# the conf/masters file, this will be taken care of automatically. Yarn
+# automatically configure the host name based on the hostname of the node where the
+# JobManager runs.
+
+jobmanager.rpc.address: localhost
+
+# The RPC port where the JobManager is reachable.
+
+jobmanager.rpc.port: 6123
+
+
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
+
+jobmanager.memory.process.size: 768m

Review comment:
       The only change is L44 and L51, I tuned it to use the minimal valid value. (may we can reduce more, but need to tune other memory confs)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org