You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/11/15 13:07:59 UTC
[GitHub] [incubator-kvrocks] PragmaTwice opened a new pull request, #1125: Use core number as the default concurrent compile task number in x.py
PragmaTwice opened a new pull request, #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125
`4` is not a good value for default concurrent compile task number.
In devices which core number is less than 4 (like vm of github actions), it can make the compile time even longer than `-j $(nproc)`, and if ore number is more than 4, it will also take longer than `-j $(nproc)` in normal situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1316272771
> > I also suspect so. But I do not know how to retrieve the correct core number. 🤣
>
> Try this? @PragmaTwice https://donghao.org/2022/01/20/how-to-get-the-number-of-cpu-cores-inside-a-container/
Thanks! I will try it. It would be nice if it can be cross-platform (e.g. linux/mac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
git-hulk commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1316282972
> I think it is hard to find a sound & robust way to retrieve current core number with respect to cpuset.
>
> `psutil` did this, but its implementation is complex.
It should be not worth to do that if we need to import a new module or add too much complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun closed pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
tisonkun closed pull request #1125: Use core number as the default concurrent compile task number in x.py
URL: https://github.com/apache/incubator-kvrocks/pull/1125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1316278975
I think it is hard to find a sound & robust way to retrieve current core number with respect to cpuset.
`psutil` did this, but its implementation is complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
git-hulk commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315444486
> > https://app.travis-ci.com/github/apache/incubator-kvrocks/jobs/588512318 Compilation failed. Please take a look @PragmaTwice .
>
> I notice that `cpu_count` returns `80` in travis ci, and then `make -j80` is run, causing OOM. It is weird.
Maybe the `getcpu()` retrieves the number of physical host cores, but the building process is running in docker environment with limited resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315482199
> > > https://app.travis-ci.com/github/apache/incubator-kvrocks/jobs/588512318 Compilation failed. Please take a look @PragmaTwice .
> >
> >
> > I notice that `cpu_count` returns `80` in travis ci, and then `make -j80` is run, causing OOM. It is weird.
>
> Maybe the `getcpu()` retrieves the number of physical host cores, but the building process is running in the docker environment with limited resources.
I also suspect so. But I do not know how to retrieve the correct core numbers. :rofl:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315428522
> https://app.travis-ci.com/github/apache/incubator-kvrocks/jobs/588512318 Compilation failed. Please take a look @PragmaTwice .
I notice that `cpu_count` returns 80 in travis ci, and then `make -j80` is run, causing memory overflow. It is weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315408933
https://app.travis-ci.com/github/apache/incubator-kvrocks/jobs/588512318 Compilation failed. Please take a look @PragmaTwice .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tanruixiang commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
tanruixiang commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1316291009
> ```
> $ cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
> cat: /sys/fs/cgroup/cpu/cpu.cfs_quota_us: No such file or directory
> ```
>
> It is weird. It always reports "no such file" in my local host machine and docker container.
In cgroup2, it seems to have been migrated to `cpu.max`.
https://stackoverflow.com/questions/65551215/get-docker-cpu-memory-limit-inside-container/65554131#65554131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
git-hulk commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315300753
I like this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1316276331
```
$ cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu/cpu.cfs_quota_us: No such file or directory
```
It is weird. It always reports "no such file" in my local host machine and docker container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tanruixiang commented on pull request #1125: Use core number as the default concurrent compile task number in x.py
Posted by GitBox <gi...@apache.org>.
tanruixiang commented on PR #1125:
URL: https://github.com/apache/incubator-kvrocks/pull/1125#issuecomment-1315589639
> I also suspect so. But I do not know how to retrieve the correct core number. 🤣
Try this?
https://donghao.org/2022/01/20/how-to-get-the-number-of-cpu-cores-inside-a-container/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org