You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by "Reylak (via GitHub)" <gi...@apache.org> on 2023/04/20 11:59:20 UTC

[GitHub] [openwhisk] Reylak opened a new issue, #5397: Building is very slow because of Python 2 and `ulimit` of number of open files

Reylak opened a new issue, #5397:
URL: https://github.com/apache/openwhisk/issues/5397

   ## Environment details:
   
   (all that matters is that I run on ArchLinux)
   
   * host: ArchLinux (2023-04-20), Linux 6.2.11
   * Gradlew:
   ```
   ------------------------------------------------------------
   Gradle 6.9.1
   ------------------------------------------------------------
   
   Build time:   2021-08-20 11:15:18 UTC
   Revision:     f0ddb54aaae0e44f0a7209c3c0274d506ea742a0
   
   Kotlin:       1.4.20
   Groovy:       2.5.12
   Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
   JVM:          1.8.0_362 (Oracle Corporation 25.362-b09)
   OS:           Linux 6.2.11-arch1-1 amd64
   ```
   
   ## Steps to reproduce the issue:
   
   1. clone OpenWhisk source repository
   2. `./gradlew distDocker`
   
   ## Provide the expected results and outputs:
   
   OpenWhisk should build in reasonable time (a matter of minutes, less than 3min on my computer).
   
   ## Provide the actual results and outputs:
   
   OpenWhisk is very slow to build.
   The very slow phase is building ow-utils, in particular installing Python 2 native modules which also compiles them (>=30min and counting, I stopped after this time).
   
   ## Proposed fix
   
   Move to Python 3-only base Docker images.
   
   Workaround: set a `ulimit` directive on Docker build commands in "gradle/docker.gradle" to lower the limit on the number of open files (`nofile`).
   For example:
   ```gradle
   dockerBuildArg = ['build', '--ulimit', 'nofile=4096:4096']
   ```
   
   ## Additional information you deem important:
   
   The underlying problem has been identified in an issue about another topic in [openwhisk-deploy-kube](https://github.com/apache/openwhisk-deploy-kube/issues/673#issuecomment-869067074): Python 2 does not handle well high numbers of possibly open FDs when asked to close them when calling `subprocess.Popen` (see [Python bug report](https://bugs.python.org/issue1663329)).
   What this has to do with building OpenWhisk, is that the Docker image for ow-utils (possibly others) uses Python 2 and installs Python packages with native modules; the latter must be compiled on-the-fly, incurring a lot of calls to `subprocesss.Popen` (just a guess here) that trigger this behavior.
   
   It is very visible on ArchLinux because the default ulimit on the number of open files in a Docker container is very high in this distribution (not sure why not in others, I guess they set a lower default value system-wide).
   
   Honestly, I think effort should be made to definitely migrate from Python 2 in every OpenWhisk component.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] dgrove-oss commented on issue #5397: Building is very slow because of Python 2 and `ulimit` of number of open files

Posted by "dgrove-oss (via GitHub)" <gi...@apache.org>.
dgrove-oss commented on issue #5397:
URL: https://github.com/apache/openwhisk/issues/5397#issuecomment-1516954667

   I agree that we should just be using Python3 everywhere.  I don't recall exactly why ow-utils is using Python 2, but its well past the Python 2 EOL data of 1/1/2020.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] Reylak commented on issue #5397: Building is very slow because of Python 2 and `ulimit` of number of open files

Posted by "Reylak (via GitHub)" <gi...@apache.org>.
Reylak commented on issue #5397:
URL: https://github.com/apache/openwhisk/issues/5397#issuecomment-1517468276

   The same applies to CouchDB (see [the issue at openwhisk-deploy-kube](https://github.com/apache/openwhisk-deploy-kube/issues/673#issuecomment-869067074)). But in this case, it has more to do with using an old version of CouchDB, and I cannot tell if using a more recent version would break things for OpenWhisk.
   
   For the record, the issue can be fixed in an Ansible deployment by adding a ulimit to the deploy task of CouchDB:
   
   ```yaml
   - name: "(re)start CouchDB from '{{ couchdb_image }} ' "
     vars:
       couchdb_image: "{{ couchdb.docker_image | default('apache/couchdb:' ~ couchdb.version ) }}"
       # [...]
       ulimits:
         - "nofile:4096:4096"
       pull: "{{ couchdb.pull_couchdb | default(true) }}"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org