You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/20 14:39:49 UTC
[GitHub] [spark-website] attilapiros opened a new pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
attilapiros opened a new pull request #307:
URL: https://github.com/apache/spark-website/pull/307
After https://github.com/apache/spark-website/commit/1bb4e571b6a575d1be55552569da46e843beccc4 the next logical step is automate the checking of the documentation generation:
- check the generation was complete (contains all the files for the last version)
- check the whether the right version of Jekyll was used
With a github workflow we can rebuild the documentation and check whether there is any unstaged changes after it.
For testing there is a temporary extra commit, which adds a new file and modifies an existing one both are added without the corresponding HTML:
```
$ git show --stat
commit c67186399fa3b444dbdaf0e09226eb954ef1e947 (HEAD -> add-workflow)
Author: attilapiros <pi...@gmail.com>
Date: Sat Feb 20 14:21:07 2021 +0100
temporary commit
community.md | 2 +-
new_file.md | 10 ++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon edited a comment on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782817894
That "Settings", even committers or PMCs don't have an access to that. We just have the write access. For such requests, we should go through [INFRA ticket](https://issues.apache.org/jira/projects/INFRA) or ask builds@apache.org mailing list.
I remember GA actions are all enabled by default so we did not face such problem in Apache Spark repo. However, looks it's now disabled for some reasons. There have been many changes w/ GA actions in ASF, e.g.) security issue, etc. So I think it's best to interact with them directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782894282
According to INFRA-21450, the policy seems to enable GitHub Action Workflow by default. I'm not sure how we can test this on PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon closed pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #307:
URL: https://github.com/apache/spark-website/pull/307
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-783673625
Oh there is a typo "documention" => "documentation".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782721677
Dongjoon and Hyukjin know it much better but if they don't pick this up in a while I can try. I need to merge it first I assume?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782729217
I checked this out one of my own repo and I assume this must be set:
But I am fine for waiting for the others just to be in the safe side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782831025
Thanks @HyukjinKwon I have opened: https://issues.apache.org/jira/browse/INFRA-21450.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782960002
@attilapiros, can you test this in your fork with changing the branch to test? Then I think we can just merge and see if works good in the main repo too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782817894
That "Settings", even committers or PMCs don't have an access to that. We just have the write access. For such requests, we should go through [INFRA ticket](https://issues.apache.org/jira/projects/INFRA) or ask builds@apache.org mailing list.
I remember GA actions are all enabled by default so we did not face such problem in Apache Spark repo. However, looks it's not disabled for some reasons. There have been many changes w/ GA actions in ASF, e.g.) security issue, etc. So I think it's best to interact with them directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782695062
I found this page:
https://docs.github.com/en/actions/managing-workflow-runs/disabling-and-enabling-a-workflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros edited a comment on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros edited a comment on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782695062
Thanks Sean! I found this page:
https://docs.github.com/en/actions/managing-workflow-runs/disabling-and-enabling-a-workflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782692747
@dongjoon-hyun @srowen @HyukjinKwon as I see we might need to enable github workflow for this to work
What do you think about this idea? Could you help me be enabling the workflow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on a change in pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #307:
URL: https://github.com/apache/spark-website/pull/307#discussion_r579768241
##########
File path: .github/workflows/doc_gen.yml
##########
@@ -0,0 +1,40 @@
+name: Check document generation
+
+on:
+ push:
+ branches:
+ - asf-site
+ pull_request:
+ branches:
+ - asf-site
+ workflow_dispatch:
+ inputs:
+ target:
+ description: 'Target branch to run'
+ required: true
Review comment:
I think you can remove this `workflow_dispatch`. It's pretty unlikely that people manually trigger the GA build ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on a change in pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #307:
URL: https://github.com/apache/spark-website/pull/307#discussion_r579783598
##########
File path: .github/workflows/doc_gen.yml
##########
@@ -0,0 +1,40 @@
+name: Check document generation
+
+on:
+ push:
+ branches:
+ - asf-site
+ pull_request:
+ branches:
+ - asf-site
+ workflow_dispatch:
+ inputs:
+ target:
+ description: 'Target branch to run'
+ required: true
Review comment:
Thanks! Removed. To keep the commit order I have modified the first commit and force pushed the commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-783791040
Merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros edited a comment on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros edited a comment on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-783672863
@HyukjinKwon thanks, this was a great idea.
I tried it with some slight modification: simply cherry picked this change into my fork's asf-site branch and pushed it.
It found a problem: the `sitemap.xml` content is different based on the ordering of the sites:
```
diff --git a/site/sitemap.xml b/site/sitemap.xml
index d39b3a212..b2c4901c6 100644
--- a/site/sitemap.xml
+++ b/site/sitemap.xml
@@ -876,11 +876,11 @@
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/screencasts/</loc>
+ <loc>https://spark.apache.org/sql/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/sql/</loc>
+ <loc>https://spark.apache.org/screencasts/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
```
This is why I remove this file from the check via "grep -v sitemap.xml".
Here is a successful run for the case when there is no change at all: https://github.com/attilapiros/spark-website/runs/1955870878
Here is the result for the temporary commit which contains markdown changes without corresponding HTMLs: https://github.com/attilapiros/spark-website/runs/1955930541
Where the error was:
```
Not all documention was generated and/or not the right Jekyll version was used! Modified / untracked files (excluding sitemap.xml):
site/new_file.html site/community.html
Git diff (excluding sitemap.xml):
diff --git a/site/community.html b/site/community.html
index c7b3d0369..10b8bf3fc 100644
--- a/site/community.html
+++ b/site/community.html
@@ -207,7 +207,7 @@
<h4>StackOverflow</h4>
-<p>For usage questions and help (e.g. how to use this Spark API), it is recommended you use the
+<p>A. For usage questions and help (e.g. how to use this Spark API), it is recommended you use the
StackOverflow tag <a href="https://stackoverflow.com/questions/tagged/apache-spark"><code class="language-plaintext highlighter-rouge">apache-spark</code></a>
as it is an active forum for Spark users’ questions and answers.</p>
Error: Process completed with exit code 1.
```
Finding the `shell: /bin/bash {0}` settings took quite a long time. The simple `bash` setting falled back to Bourne Shell.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782896440
What about removing my 2nd commit (the temporary test). Merging this PR then testing the idea with a new PR which won't be merged? If there is a problem we can revert this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #307: [WIP] Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-782693086
Seems like a fine idea. I don't know how to enable workflows myself but can click buttons if I know where to click.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] attilapiros commented on pull request #307: Add github workflow for checking all doc is generated with the right Jekyll version
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #307:
URL: https://github.com/apache/spark-website/pull/307#issuecomment-783672863
@HyukjinKwon thanks, this was a great idea.
I tried it with some slight modification: simply cherry picked this change into my fork's asf-site branch and pushed it.
It found a problem: the `sitemap.xml` content is different based on the ordering of the sites:
```
diff --git a/site/sitemap.xml b/site/sitemap.xml
index d39b3a212..b2c4901c6 100644
--- a/site/sitemap.xml
+++ b/site/sitemap.xml
@@ -876,11 +876,11 @@
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/screencasts/</loc>
+ <loc>https://spark.apache.org/sql/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/sql/</loc>
+ <loc>https://spark.apache.org/screencasts/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
```
This is why I remove this file from the check via "grep -v sitemap.xml".
Here is a successful run for the case when there is no change at all: https://github.com/attilapiros/spark-website/runs/1955870878
Here is the result for the temporary commit which contains markdown changes without corresponding HTMLs: https://github.com/attilapiros/spark-website/runs/1955930541
```
Not all documention was generated and/or not the right Jekyll version was used! Modified / untracked files (excluding sitemap.xml):
site/new_file.html site/community.html
Git diff (excluding sitemap.xml):
diff --git a/site/community.html b/site/community.html
index c7b3d0369..10b8bf3fc 100644
--- a/site/community.html
+++ b/site/community.html
@@ -207,7 +207,7 @@
<h4>StackOverflow</h4>
-<p>For usage questions and help (e.g. how to use this Spark API), it is recommended you use the
+<p>A. For usage questions and help (e.g. how to use this Spark API), it is recommended you use the
StackOverflow tag <a href="https://stackoverflow.com/questions/tagged/apache-spark"><code class="language-plaintext highlighter-rouge">apache-spark</code></a>
as it is an active forum for Spark users’ questions and answers.</p>
Error: Process completed with exit code 1.
```
Finding the `shell: /bin/bash {0}` settings took quite a long time. The simple `bash` setting falled back to Bourne Shell.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org