You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/07/13 11:41:25 UTC
[GitHub] [lucene-jira-archive] mocobeta opened a new pull request, #41: Allow to specify number of worker processes for jira2github_import.py
mocobeta opened a new pull request, #41:
URL: https://github.com/apache/lucene-jira-archive/pull/41
Close #36
`jira2github_import.py` processes Jira dump files one by one and does not call any HTTP APIs. It should be able to parallelize it with [multiprocessing](https://docs.python.org/3/library/multiprocessing.html).
One problem is how to handle the log file. I implemented the "log listener" pattern following this cookbook.
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
Usage:
```
# use four worker processes. a log listener process is also started.
python src/jira2github_import.py --min 9000 --max 9100 --num_workers=4
# all forked processes should be stopped by sending SIGINT to the main process
(Ctrl-C on Linux)
```
If `--num_workers` option is committed, only one worker and listener processes are started.
Note: I think this code is OS-agnostic, but I haven't used `multiprocessing` on Windows. There might be some pitfalls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-jira-archive] mocobeta commented on pull request #41: Allow to specify number of worker processes for jira2github_import.py
Posted by GitBox <gi...@apache.org>.
mocobeta commented on PR #41:
URL: https://github.com/apache/lucene-jira-archive/pull/41#issuecomment-1183240812
@mikemccand just to let you know, I'm going to merge this tomorrow (in JST) so that we are able to iterate conversion tests more often.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-jira-archive] mocobeta commented on pull request #41: Allow to specify number of worker processes for jira2github_import.py
Posted by GitBox <gi...@apache.org>.
mocobeta commented on PR #41:
URL: https://github.com/apache/lucene-jira-archive/pull/41#issuecomment-1183303117
This code shares a Logger object between workers, that seems to work on Linux but might not work on windows.
```
# The worker configuration is done at the start of the worker process run.
# Note that on Windows you can't rely on fork semantics, so each process
# will run the logging configuration code when it starts.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-jira-archive] mocobeta merged pull request #41: Allow to specify number of worker processes for jira2github_import.py
Posted by GitBox <gi...@apache.org>.
mocobeta merged PR #41:
URL: https://github.com/apache/lucene-jira-archive/pull/41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-jira-archive] mocobeta commented on pull request #41: Allow to specify number of worker processes for jira2github_import.py
Posted by GitBox <gi...@apache.org>.
mocobeta commented on PR #41:
URL: https://github.com/apache/lucene-jira-archive/pull/41#issuecomment-1183234776
Works fine for me - it now processes the whole Jira dump in about 80 minutes with four workers.
```
python src/jira2github_import.py --min 1 --max 10648 --num_workers 4
```
All logs were correctly written in a single file as before (the order was not sequential this time).
```
[2022-07-13 20:50:19,001] INFO:jira2github_import: Converting Jira issues to GitHub issues in /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data. num_workers=4
[2022-07-13 20:50:19,173] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-1.json
[2022-07-13 20:50:19,295] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-2.json
[2022-07-13 20:50:19,583] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-3.json
[2022-07-13 20:50:19,799] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-5.json
[2022-07-13 20:50:19,920] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-6.json
...
[2022-07-13 22:09:17,144] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-10646.json
[2022-07-13 22:09:17,237] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-10645.json
[2022-07-13 22:09:17,395] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-10648.json
[2022-07-13 22:09:17,577] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-10647.json
[2022-07-13 22:09:20,118] DEBUG:jira2github_import: GitHub issue data created: /mnt/hdd/repo/lucene-jira-archive/migration/github-import-data/GH-LUCENE-10643.json
[2022-07-13 22:09:20,122] INFO:jira2github_import: Done.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org