You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/23 18:28:29 UTC
[GitHub] [pulsar] michaeljmarshall opened a new pull request #14825: Skip crowdin download
michaeljmarshall opened a new pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825
Master issue: #14824
### Motivation
The crowdin translations do not get updated often enough to warrant 3 downloads a day, especially given that the download and upload are currently timing out and preventing the english version of the website to get published.
### Modifications
* Make the `crowdin-download` step only run once a day.
* Add license to the yarn `package.json`.
### Verifying this change
This is a trivial change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14825: Skip crowdin download
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825#discussion_r833603104
##########
File path: site2/tools/build-site.sh
##########
@@ -66,13 +66,14 @@ cd ${ROOT_DIR}/site2/website
yarn
yarn write-translations
-if [ "$CROWDIN_DOCUSAURUS_API_KEY" != "UNSET" ]; then
+# The crowdin upload and download take a long time to run, and have resulted in timeouts. In order to ensure that the
+# website is still able to get published, we only run the download and upload if current hour is 0-5.
+# This leads to executing crowdin-upload and crowdin-download once per day when website build is scheduled
+# to run with cron expression '0 */6 * * *'
+CURRENT_HOUR=$(date +%H)
+if [[ "$CROWDIN_DOCUSAURUS_API_KEY" != "UNSET" || $CURRENT_HOUR -lt 6 ]]; then
# upload only if environment variable CROWDIN_UPLOAD=1 is set
- # or current hour is 0-5
- # this leads to executing crowdin-upload once per day when website build is scheduled
- # to run with cron expression '0 */6 * * *'
- CURRENT_HOUR=$(date +%H)
- if [[ "$CROWDIN_UPLOAD" == "1" || $CURRENT_HOUR -lt 6 ]]; then
+ if [[ "$CROWDIN_UPLOAD" == "1" ]]; then
yarn run crowdin-upload
fi
yarn run crowdin-download
Review comment:
I'm fine in making the change that you have proposed.
I'm just thinking that when both crowdin-upload and crowdin-download will happen once a day, that might be so slow that it never completes. For context, I made the original `$CURRENT_HOUR -lt 6` hack in https://github.com/apache/pulsar/pull/10794 . That was a quick fix to get the website build to complete by running crowdin-upload once per day.
One possible solution here would be to schedule crowdin-upload and crowdin-download in different time slots.
However, it would be better to get them scheduled in completely separate jobs to fix the problem. That might be the only way to get the website build stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall merged pull request #14825: Skip crowdin download
Posted by GitBox <gi...@apache.org>.
michaeljmarshall merged pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #14825: Skip crowdin download
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825#discussion_r833606603
##########
File path: site2/tools/build-site.sh
##########
@@ -66,13 +66,14 @@ cd ${ROOT_DIR}/site2/website
yarn
yarn write-translations
-if [ "$CROWDIN_DOCUSAURUS_API_KEY" != "UNSET" ]; then
+# The crowdin upload and download take a long time to run, and have resulted in timeouts. In order to ensure that the
+# website is still able to get published, we only run the download and upload if current hour is 0-5.
+# This leads to executing crowdin-upload and crowdin-download once per day when website build is scheduled
+# to run with cron expression '0 */6 * * *'
+CURRENT_HOUR=$(date +%H)
+if [[ "$CROWDIN_DOCUSAURUS_API_KEY" != "UNSET" || $CURRENT_HOUR -lt 6 ]]; then
# upload only if environment variable CROWDIN_UPLOAD=1 is set
- # or current hour is 0-5
- # this leads to executing crowdin-upload once per day when website build is scheduled
- # to run with cron expression '0 */6 * * *'
- CURRENT_HOUR=$(date +%H)
- if [[ "$CROWDIN_UPLOAD" == "1" || $CURRENT_HOUR -lt 6 ]]; then
+ if [[ "$CROWDIN_UPLOAD" == "1" ]]; then
yarn run crowdin-upload
fi
yarn run crowdin-download
Review comment:
I think the long-term fix is to upgrade the crowdin cli. Their docs mention that it is more efficient and has multi-threading for uploads.
I agree that separate jobs would make sense too. It only makes sense to upload files when docs have actually changed. I wonder if we can upload _just_ the changed files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #14825: Skip crowdin download
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825#issuecomment-1076965648
FYI @urfreespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #14825: Skip crowdin download
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on pull request #14825:
URL: https://github.com/apache/pulsar/pull/14825#issuecomment-1077165972
@Anonymitaet @urfreespace - I think we need to look at upgrading the crowdin cli version, so I created this issue: https://github.com/apache/pulsar/issues/14837. Also, is there a way committers can get access to the crowdin account?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org