You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@whimsical.apache.org by cu...@apache.org on 2018/05/09 16:35:12 UTC
[whimsy] branch master updated (46b681b -> 908e1f5)
This is an automated email from the ASF dual-hosted git repository.
curcuru pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/whimsy.git.
from 46b681b be more forgiving
new 0f813c1 Pull site/vs/pods specific data out
new 8abb8ca Encapsulate details of site checking data
new 908e1f5 Rearchitect to use new sitestandards module
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lib/whimsy/sitestandards.rb | 203 +++++++++++++++++++
www/site.cgi | 467 ++++++++++++++++++++------------------------
2 files changed, 413 insertions(+), 257 deletions(-)
create mode 100644 lib/whimsy/sitestandards.rb
--
To stop receiving notification emails like this one, please contact
curcuru@apache.org.
[whimsy] 01/03: Pull site/vs/pods specific data out
Posted by cu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
curcuru pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/whimsy.git
commit 0f813c16dde7efc23bdde221d7f0e31350b54598
Author: Shane Curcuru <as...@shanecurcuru.org>
AuthorDate: Tue May 8 12:36:47 2018 -0400
Pull site/vs/pods specific data out
Starting to work on making common utility methods that can make
site|pods.cgi both use all same code
---
www/site.cgi | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/www/site.cgi b/www/site.cgi
index 6479d26..f2c43e1 100755
--- a/www/site.cgi
+++ b/www/site.cgi
@@ -13,7 +13,7 @@ require 'json'
require 'net/http'
require 'time' # for httpdate
-PAGETITLE = "Apache TLP Website Checks" # Wvisible:sites,brand
+PAGETITLE = "Apache Project Website Checks" # Wvisible:sites,brand
SITE_PASS = 'label-success'
SITE_WARN = 'label-warning'
SITE_FAIL = 'label-danger'
@@ -53,7 +53,8 @@ DOCS = {
'thanks' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
'"Thanks" should link to: http://www.apache.org/foundation/thanks.html'],
}
-DATAURI = 'https://whimsy.apache.org/public/site-scan.json'
+DATAURI = 'https://whimsy.apache.org/public/'
+SCAN_DATA_FILE = 'site-scan.json'
def analyze(sites)
success = Hash.new { |h, k| h[k] = Hash.new(&h.default_proc) }
@@ -75,20 +76,20 @@ def analyze(sites)
]
end
-def getsites
- local_copy = File.expand_path('../public/site-scan.json', __FILE__).untaint
+def getsites(filename)
+ local_copy = File.expand_path("../public/#{filename}", __FILE__).untaint
if File.exist? local_copy
crawl_time = File.mtime(local_copy).httpdate # show time in same format as last-mod
sites = JSON.parse(File.read(local_copy))
else
- response = Net::HTTP.get_response(URI(DATAURI))
+ response = Net::HTTP.get_response(URI("#{DATAURI}#{filename}"))
crawl_time = response['last-modified']
sites = JSON.parse(response.body)
end
return sites, crawl_time
end
-sites, crawl_time = getsites()
+sites, crawl_time = getsites(SCAN_DATA_FILE)
analysis = analyze(sites)
@@ -205,7 +206,7 @@ _html do
_body? do
_whimsy_body(
title: PAGETITLE,
- subtitle: 'Checking TLP Websites For required content',
+ subtitle: 'Checking Project/Podling Websites For required content',
related: {
"/committers/tools" => "Whimsy Tool Listing",
"https://www.apache.org/foundation/marks/pmcs#navigation" => "Required PMC Links Policy",
@@ -214,7 +215,7 @@ _html do
},
helpblock: -> {
_p do
- _ 'This script periodically crawls all Apache project websites to check them for a few specific links or text blocks that all projects are expected to have.'
+ _ 'This script periodically crawls all Apache project and podling websites to check them for a few specific links or text blocks that all projects are expected to have.'
_ 'The checks include verifying that all '
_a 'required links', href: 'https://www.apache.org/foundation/marks/pmcs#navigation'
_ ' appear on a project homepage, along with an "image" check if project logo files are in apache.org/img'
@@ -222,9 +223,9 @@ _html do
_p! do
_a 'View the crawler code', href: 'https://github.com/apache/whimsy/blob/master/tools/site-scan.rb'
_ ', '
- _a 'website display code', href: 'https://github.com/apache/whimsy/blob/master/www/site.cgi'
+ _a 'website display code', href: "https://github.com/apache/whimsy/blob/master/www#{ENV['SCRIPT_NAME']}"
_ ', and '
- _a 'raw JSON data', href: DATAURI
+ _a 'raw JSON data', href: "#{DATAURI}#{SCAN_DATA_FILE}"
_ '.'
_br
_ "Last crawl time: #{crawl_time} over #{sites.size} websites."
@@ -299,7 +300,7 @@ _html do
else
# overview
_whimsy_panel_table(
- title: "Site Check - All Projects Results",
+ title: "Site Check - All Project/Podling Results",
helpblock: -> {
_ul.list_inline do
_li.small "Data key: "
--
To stop receiving notification emails like this one, please contact
curcuru@apache.org.
[whimsy] 02/03: Encapsulate details of site checking data
Posted by cu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
curcuru pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/whimsy.git
commit 8abb8ca6757c8d4291d5cb73808dd6597f070949
Author: Shane Curcuru <as...@shanecurcuru.org>
AuthorDate: Wed May 9 12:33:04 2018 -0400
Encapsulate details of site checking data
---
lib/whimsy/sitestandards.rb | 203 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 203 insertions(+)
diff --git a/lib/whimsy/sitestandards.rb b/lib/whimsy/sitestandards.rb
new file mode 100644
index 0000000..5bda6d8
--- /dev/null
+++ b/lib/whimsy/sitestandards.rb
@@ -0,0 +1,203 @@
+#!/usr/bin/env ruby
+# Defines partial standards for Apache website checker
+# TODO better document with specific policies
+
+# Encapsulate (most) scans/validations done on website content
+module SiteStandards
+ extend self
+ CHECK_TEXT = 'text' # (optional) Regex of <a ...>Text to scan for</a>, of a.text.downcase.strip
+ CHECK_CAPTURE = 'capture' # a_href minimal regex to capture - for license, we capture the link if it points to apache.org somewhere
+ CHECK_VALIDATE = 'validate' # a_href detailed regex to expect for compliance; it must point to one of our actual licenses to pass
+ CHECK_TYPE = 'type' # true = validation checks href/url; false = checks text node
+ CHECK_POLICY = 'policy' # URL to policy statement for this check
+ CHECK_DOC = 'doc' # Explanation of what the check is looking for
+
+ # Checks done only for TLPs (i.e. not podlings)
+ TLP_CHECKS = {
+ 'uri' => { # Custom: merely saves uri of site
+ CHECK_TEXT => nil,
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{https?://[^.]+\.apache\.org},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#websites',
+ CHECK_DOC => 'The homepage for any ProjectName must be served from http://ProjectName.apache.org',
+ },
+ }
+ # Checks done only for Incubator podlings
+ PODLING_CHECKS = {
+ 'uri' => {
+ CHECK_TEXT => nil,
+ CHECK_CAPTURE => %r{https?://[^.]+\.incubator\.apache\.org},
+ CHECK_VALIDATE => %r{https?://[^.]+\.incubator\.apache\.org},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#websites',
+ CHECK_DOC => 'The homepage for any ProjectName must be served from http://ProjectName(.incubator).apache.org',
+ },
+ 'disclaimer' => { # textnode_check: txt =~ / Incubation is required of all newly accepted projects /
+ CHECK_TEXT => %r{Incubation is required of all newly accepted projects},
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{Apache \S+( \S+)?( \([Ii]ncubating\))? is an effort undergoing [Ii]ncubation at [Tt]he Apache Software Foundation \(ASF\),? sponsored by the (Apache )?\S+( PMC)?. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the complet [...]
+ CHECK_TYPE => false,
+ CHECK_POLICY => 'https://incubator.apache.org/guides/branding.html#disclaimers',
+ CHECK_DOC => 'All Apache Incubator Podling sites must contain the incubating disclaimer.',
+ },
+ }
+ # Checks done for all podlings|projects
+ COMMON_CHECKS = {
+ 'foundation' => { # Custom: a_href =~ ... then custom checking for hover/title text
+ CHECK_TEXT => %r{apache|asf|foundation}i,
+ CHECK_CAPTURE => %r{.}i,
+ CHECK_VALIDATE => %r{apache|asf|foundation}i,
+ CHECK_TYPE => false,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#navigation',
+ CHECK_DOC => 'All projects must feature some prominent link back to the main ASF homepage at http://www.apache.org/',
+ },
+ 'events' => { # Custom: a_href.include? 'apache.org/events/' then custom check for img
+ CHECK_TEXT => nil,
+ CHECK_CAPTURE => %r{apache\.org\/events},
+ CHECK_VALIDATE => %r{^https?://.*apache.org/events/current-event},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/events/README.txt',
+ CHECK_DOC => 'Projects SHOULD include a link to any current ApacheCon event, as provided by VP, Conferences.',
+ },
+ 'license' => { # link_check a_text =~ /^license$/ and a_href.include? 'apache.org'
+ CHECK_TEXT => /^license$/,
+ CHECK_CAPTURE => %r{apache\.org},
+ CHECK_VALIDATE => %r{^https?://.*apache.org/licenses/$},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#navigation',
+ CHECK_DOC => '"License" should link to: http://www.apache.org/licenses/',
+ },
+ 'thanks' => { # link_check a_text =~ /\Athanks[!]?\z/
+ CHECK_TEXT => /\Athanks[!]?\z/,
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{^https?://.*apache.org/foundation/thanks},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#navigation',
+ CHECK_DOC => '"Thanks" should link to: http://www.apache.org/foundation/thanks.html',
+ },
+ 'security' => { # link_check a_text == 'security'
+ CHECK_TEXT => /security/,
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{^https?://.*apache.org/[Ss]ecurity},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#navigation',
+ CHECK_DOC => '"Security" should link to either to a project-specific page [...], or to the main http://www.apache.org/security/ page.',
+ },
+ 'sponsorship' => { # link_check ['sponsorship', 'donate', 'sponsor apache','sponsoring apache'].include? a_text
+ CHECK_TEXT => %r{sponsorship|donate|sponsor\sapache|sponsoring\sapache},
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{^https?://.*apache.org/foundation/sponsorship},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#navigation',
+ CHECK_DOC => '"Sponsorship", "Sponsor Apache", or "Donate" should link to: http://www.apache.org/foundation/sponsorship.html',
+ },
+
+ 'trademarks' => { # textnode_check: if (txt =~ /\btrademarks\b/ and not data[:trademarks]) or txt =~/are trademarks of [Tt]he Apache Software/
+ CHECK_TEXT => %r{\btrademarks\b},
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{trademarks of [Tt]he Apache Software Foundation},
+ CHECK_TYPE => false,
+ CHECK_POLICY => 'https://www.apache.org/foundation/marks/pmcs#attributions',
+ CHECK_DOC => 'All project or product homepages must feature a prominent trademark attribution of all applicable Apache trademarks.',
+ },
+ 'copyright' => { # textnode_check: txt =~ /Copyright / or txt =~ /©/
+ CHECK_TEXT => %r{Copyright|©},
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{[Cc]opyright [^.]+ Apache Software Foundation}, # Do we need '[Tt]he ASF'?
+ CHECK_TYPE => false,
+ CHECK_POLICY => 'https://www.apache.org/legal/src-headers.html#headers',
+ CHECK_DOC => 'All website content SHOULD include a copyright notice for the ASF.',
+ },
+
+ 'image' => { # Custom: merely looks in IMAGE_DIR for #{id}.*
+ CHECK_TEXT => nil,
+ CHECK_CAPTURE => nil,
+ CHECK_VALIDATE => %r{projectname.jpg},
+ CHECK_TYPE => true,
+ CHECK_POLICY => 'https://www.apache.org/img/',
+ CHECK_DOC => 'Projects SHOULD include a 212px wide copy of their logo in https://www.apache.org/img/ to be included in ASF homepage.',
+ },
+ }
+
+ SITE_PASS = 'label-success'
+ SITE_WARN = 'label-warning'
+ SITE_FAIL = 'label-danger'
+ # Determine the color of a given table cell, given:
+ # - overall analysis of the sites, in particular the third column
+ # which is a list projects that successfully matched the check
+ # - list of links for the project in question
+ # - the column in question (which indicates the check being reported on)
+ # - the name of the project
+ def label(analysis, links, col, name)
+ if not links[col]
+ SITE_FAIL
+ elsif analysis[2].include? col and not analysis[2][col].include? name
+ SITE_WARN
+ else
+ SITE_PASS
+ end
+ end
+
+ # Get hash of checks to be done for tlp | podling
+ # @param tlp true if project; podling otherwise
+ def get_checks(tlp = true)
+ tlp ? (return TLP_CHECKS.merge(COMMON_CHECKS)) : (return PODLING_CHECKS.merge(COMMON_CHECKS))
+ end
+
+ # Get filename of check data for tlp | podling
+ # @param tlp true if project; podling otherwise
+ def get_filename(tlp = true)
+ tlp ? (return 'site-scan.json') : (return 'pods-scan.json')
+ end
+
+ # Get URL to default filename location on server
+ def get_url(is_local = true)
+ is_local ? (return '../public/') : (return 'https://whimsy.apache.org/public/')
+ end
+
+ # Get check data for tlp | podling
+ # Uses a local_copy if available; w.a.o/public otherwise
+ # @param tlp true if project; podling otherwise
+ # @return [hash of site data, crawl_time]
+ def get_sites(tlp = true)
+ local_copy = File.expand_path("#{get_url(true)}#{get_filename(tlp)}", __FILE__).untaint
+ if File.exist? local_copy
+ crawl_time = File.mtime(local_copy).httpdate # show time in same format as last-mod
+ sites = JSON.parse(File.read(local_copy))
+ else
+ response = Net::HTTP.get_response(URI("#{get_url(false)}#{get_filename(tlp)}"))
+ crawl_time = response['last-modified']
+ sites = JSON.parse(response.body)
+ end
+ return sites, crawl_time
+ end
+
+ # Analyze data returned from site-scan.rb by using checks[CHECK_VALIDATE] regex
+ # If value =~ CHECK_VALIDATE, SITE_PASS
+ # If value is present (presumably from CHECK_TEXT|CAPTURE), then SITE_WARN
+ # If value not present, SITE_FAIL (i.e. site-scan.rb didn't find it)
+ # @param sites hash of site-scan data collected
+ # @param checks to apply to sites to determine status
+ # @return [overall counts, description of statuses, success listings]
+ def analyze(sites, checks)
+ success = Hash.new { |h, k| h[k] = Hash.new(&h.default_proc) }
+ counts = Hash.new { |h, k| h[k] = Hash.new(&h.default_proc) }
+ checks.each do |nam, check_data|
+ success[nam] = sites.select{ |k, site| site[nam] =~ check_data[SiteStandards::CHECK_VALIDATE] }.keys
+ counts[nam][SITE_PASS] = success[nam].count
+ counts[nam][SITE_WARN] = 0 # Reorder output
+ counts[nam][SITE_FAIL] = sites.select{ |k, site| site[nam].nil? }.count
+ counts[nam][SITE_WARN] = sites.size - counts[nam][SITE_PASS] - counts[nam][SITE_FAIL]
+ end
+
+ return [
+ counts, {
+ SITE_PASS => '# Sites with links to primary ASF page',
+ SITE_WARN => '# Sites with link, but not an expected ASF one',
+ SITE_FAIL => '# Sites with no link for this topic'
+ }, success
+ ]
+ end
+end
+
--
To stop receiving notification emails like this one, please contact
curcuru@apache.org.
[whimsy] 03/03: Rearchitect to use new sitestandards module
Posted by cu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
curcuru pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/whimsy.git
commit 908e1f5c4a7e3ebe36414621565737e12831bfe7
Author: Shane Curcuru <as...@shanecurcuru.org>
AuthorDate: Wed May 9 12:34:09 2018 -0400
Rearchitect to use new sitestandards module
In preparation for making site.cgi and pods.cgi both use same core
data/functions
---
www/site.cgi | 468 +++++++++++++++++++++++++++--------------------------------
1 file changed, 210 insertions(+), 258 deletions(-)
diff --git a/www/site.cgi b/www/site.cgi
index f2c43e1..6054c9d 100755
--- a/www/site.cgi
+++ b/www/site.cgi
@@ -1,4 +1,5 @@
#!/usr/bin/env ruby
+PAGETITLE = "Apache Project Website Checks" # Wvisible:sites,brand
# ensure that there is a path (even a slash will do) after the script name
unless ENV['PATH_INFO'] and not ENV['PATH_INFO'].empty?
@@ -12,86 +13,13 @@ $LOAD_PATH.unshift File.realpath(File.expand_path('../../lib', __FILE__))
require 'json'
require 'net/http'
require 'time' # for httpdate
+require 'whimsy/sitestandards'
-PAGETITLE = "Apache Project Website Checks" # Wvisible:sites,brand
-SITE_PASS = 'label-success'
-SITE_WARN = 'label-warning'
-SITE_FAIL = 'label-danger'
-cols = %w( uri events foundation image license sponsorship security thanks copyright trademarks )
-CHECKS = {
- 'uri' => %r{https?://[^.]+\.apache\.org},
- 'copyright' => %r{[Cc]opyright [^.]+ Apache Software Foundation}, # Do we need '[Tt]he ASF'?
- 'foundation' => %r{.},
- 'image' => %r{.},
- # TODO more checks needed here, e.g. ASF registered and 3rd party marks
- 'trademarks' => %r{trademarks of [Tt]he Apache Software Foundation},
- 'events' => %r{^https?://.*apache.org/events/current-event},
- 'license' => %r{^https?://.*apache.org/licenses/$}, # should link to parent license page only
- 'sponsorship' => %r{^https?://.*apache.org/foundation/sponsorship},
- 'security' => %r{^https?://.*apache.org/[Ss]ecurity},
- 'thanks' => %r{^https?://.*apache.org/foundation/thanks},
-}
-DOCS = {
- 'uri' => ['https://www.apache.org/foundation/marks/pmcs#websites',
- 'The homepage for any ProjectName must be served from http://ProjectName.apache.org'],
- 'copyright' => ['https://www.apache.org/legal/src-headers.html#headers',
- 'All website content SHOULD include a copyright notice for the ASF.'],
- 'foundation' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
- 'All projects must feature some prominent link back to the main ASF homepage at http://www.apache.org/'],
- 'image' => ['https://www.apache.org/img/',
- 'Projects SHOULD include a 212px wide copy of their logo in https://www.apache.org/img/ to be included in ASF homepage.'],
- 'trademarks' => ['https://www.apache.org/foundation/marks/pmcs#attributions',
- 'All project or product homepages must feature a prominent trademark attribution of all applicable Apache trademarks'],
- 'events' => ['https://www.apache.org/events/README.txt',
- 'Projects SHOULD include a link to any current ApacheCon event, as provided by VP, Conferences.'],
- 'license' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
- '"License" should link to: http://www.apache.org/licenses/'],
- 'sponsorship' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
- '"Sponsorship" or "Donate" should link to: http://www.apache.org/foundation/sponsorship.html'],
- 'security' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
- '"Security" should link to either to a project-specific page [...], or to the main http://www.apache.org/security/ page'],
- 'thanks' => ['https://www.apache.org/foundation/marks/pmcs#navigation',
- '"Thanks" should link to: http://www.apache.org/foundation/thanks.html'],
-}
-DATAURI = 'https://whimsy.apache.org/public/'
-SCAN_DATA_FILE = 'site-scan.json'
-
-def analyze(sites)
- success = Hash.new { |h, k| h[k] = Hash.new(&h.default_proc) }
- counts = Hash.new { |h, k| h[k] = Hash.new(&h.default_proc) }
- CHECKS.each do |nam, pat|
- success[nam] = sites.select{ |k, site| site[nam] =~ pat }.keys
- counts[nam][SITE_PASS] = success[nam].count
- counts[nam][SITE_WARN] = 0 # Reorder output
- counts[nam][SITE_FAIL] = sites.select{ |k, site| site[nam].nil? }.count
- counts[nam][SITE_WARN] = sites.size - counts[nam][SITE_PASS] - counts[nam][SITE_FAIL]
- end
-
- [
- counts, {
- SITE_PASS => '# Sites with links to primary ASF page',
- SITE_WARN => '# Sites with link, but not an expected ASF one',
- SITE_FAIL => '# Sites with no link for this topic'
- }, success
- ]
-end
-
-def getsites(filename)
- local_copy = File.expand_path("../public/#{filename}", __FILE__).untaint
- if File.exist? local_copy
- crawl_time = File.mtime(local_copy).httpdate # show time in same format as last-mod
- sites = JSON.parse(File.read(local_copy))
- else
- response = Net::HTTP.get_response(URI("#{DATAURI}#{filename}"))
- crawl_time = response['last-modified']
- sites = JSON.parse(response.body)
- end
- return sites, crawl_time
-end
-
-sites, crawl_time = getsites(SCAN_DATA_FILE)
-
-analysis = analyze(sites)
+# Gather and analyze scans for TLP websites
+cgi_for_tlps = true
+sites, crawl_time = SiteStandards.get_sites(cgi_for_tlps)
+checks_performed = SiteStandards.get_checks(cgi_for_tlps)
+analysis = SiteStandards.analyze(sites, checks_performed)
# Allow CLI testing, e.g. "PATH_INFO=/ ruby www/site.cgi >test.json"
# SCRIPT_NAME will always be set for a CGI invocation
@@ -108,25 +36,68 @@ require 'wunderbar'
require 'wunderbar/bootstrap'
require 'wunderbar/jquery/stupidtable'
-# Determine the color of a given table cell, given:
-# - overall analysis of the sites, in particular the third column
-# which is a list projects that successfully matched the check
-# - list of links for the project in question
-# - the column in question (which indicates the check being reported on)
-# - the name of the project
-def label(analysis, links, col, name)
- if not links[col]
- SITE_FAIL
- elsif analysis[2].include? col and not analysis[2][col].include? name
- SITE_WARN
- else
- SITE_PASS
+_html do
+ _head do
+ _style %{
+ .table td {font-size: smaller;}
+ }
+ end
+ _body? do
+ _whimsy_body(
+ title: PAGETITLE,
+ subtitle: "Checking #{cgi_for_tlps ? 'Project' : 'Podling'} Websites For required content",
+ related: {
+ "/committers/tools" => "Whimsy Tool Listing",
+ "https://www.apache.org/foundation/marks/pmcs#navigation" => "Required PMC Links Policy",
+ "https://github.com/apache/whimsy/blob/master/www#{ENV['SCRIPT_NAME']}" => "See This Source Code",
+ "mailto:dev@whimsical.apache.org?subject=[SITE] Website Checker Question" => "Questions? Email Whimsy PMC"
+ },
+ helpblock: -> {
+ _p do
+ _ 'This script periodically crawls all Apache project and podling websites to check them for a few specific links or text blocks that all projects are expected to have.'
+ _ 'The checks include verifying that all '
+ _a 'required links', href: 'https://www.apache.org/foundation/marks/pmcs#navigation'
+ _ ' appear on a project homepage, along with an "image" check if project logo files are in apache.org/img'
+ end
+ _p! do
+ _a 'View the crawler code', href: 'https://github.com/apache/whimsy/blob/master/tools/site-scan.rb'
+ _ ', '
+ _a 'website display code', href: "https://github.com/apache/whimsy/blob/master/www#{ENV['SCRIPT_NAME']}"
+ _ ', and '
+ _a 'raw JSON data', href: "#{SiteStandards.get_url()}#{SiteStandards.get_filename(cgi_for_tlps)}"
+ _ '.'
+ _br
+ _ "Last crawl time: #{crawl_time} over #{sites.size} websites."
+ end
+ }
+ ) do
+ # Encapsulate data display (same for projects and podlings)
+ display_application(path_info, sites, analysis, checks_performed, cgi_for_tlps)
+ end
+
+ _script %{
+ var table = $(".table").stupidtable();
+ table.on("aftertablesort", function (event, data) {
+ var th = $(this).find("th");
+ th.find(".arrow").remove();
+ var dir = $.fn.stupidtable.dir;
+ var arrow = data.direction === dir.ASC ? "↑" : "↓";
+ th.eq(data.column).append('<span class="arrow">' + arrow +'</span>');
+ });
+ }
end
end
-def displayProject(project, links, cols, analysis)
+# Encapsulate (most) display of website checks between projects|podlings
+# Display data for a single project's checks
+# @param project id of project
+# @param links site data for that specific project
+# @param columns list of check types to report on
+# @param analysis complete scan data
+# @param tlp true if project (default); podling otherwise
+def display_project(project, links, analysis, checks, tlp = true)
_whimsy_panel_table(
- title: "Site Check For Project - #{links['display_name']}",
+ title: "Site Check For #{tlp ? 'Project' : 'Podling'} - #{links['display_name']}",
helpblock: -> {
_a href: '../', aria_label: 'Home to site checker' do
_span.glyphicon.glyphicon_home :aria_hidden
@@ -146,8 +117,8 @@ def displayProject(project, links, cols, analysis)
_th! 'Check Description'
end
end
- cols.each do |col|
- cls = label(analysis, links, col, project)
+ checks.keys.each do |col|
+ cls = SiteStandards.label(analysis, links, col, project)
_tr do
_td do
_a col.capitalize, href: "../check/#{col}"
@@ -162,14 +133,17 @@ def displayProject(project, links, cols, analysis)
end
_td do
- if cls != SITE_PASS
- if CHECKS.include? col
- _ 'Expected to match regular expression: '
- _code CHECKS[col].source
- if DOCS.include? col
- _ ' '
- _a DOCS[col][1], href: DOCS[col][0]
+ if cls != SiteStandards::SITE_PASS
+ if checks.keys.include? col
+ if checks[col][SiteStandards::CHECK_TYPE]
+ _ 'URL expected to match regular expression: '
+ _code checks[col][SiteStandards::CHECK_VALIDATE].source
+ else
+ _ 'Text of a link expected to match regular expression: '
+ _code checks[col][SiteStandards::CHECK_TEXT].source
end
+ _br
+ _a checks[col][SiteStandards::CHECK_DOC], href: checks[col][SiteStandards::CHECK_POLICY]
else
_ ''
end
@@ -182,185 +156,163 @@ def displayProject(project, links, cols, analysis)
end
end
-def displayError(path)
+# Display data for a single check across all projects/podlings
+# @param col id of check to display
+# @param sites site data for all projects
+# @param analysis complete scan data
+# @param checks complete set of checks performed
+# @param tlp true if project (default); podling otherwise
+def display_check(col, sites, analysis, checks, tlp = true)
_whimsy_panel_table(
- title: "ERROR",
+ title: "Site Check Of Type - #{col.capitalize}",
helpblock: -> {
_a href: '../', aria_label: 'Home to site checker' do
_span.glyphicon.glyphicon_home :aria_hidden
end
_span.glyphicon.glyphicon_menu_right
- _span.text_danger "ERROR: The path #{path} is not a recognized command for this tool, sorry! "
+ if checks.keys.include? col
+ if checks[col][SiteStandards::CHECK_TYPE]
+ _ 'Check Results URL expected to match regular expression: '
+ _code checks[col][SiteStandards::CHECK_VALIDATE].source
+ else
+ _ 'Check Results Text of a link expected to match regular expression: '
+ _code checks[col][SiteStandards::CHECK_TEXT].source
+ end
+ if checks.include? col
+ _br
+ _a checks[col][SiteStandards::CHECK_DOC], href: checks[col][SiteStandards::CHECK_POLICY]
+ end
+ _li.small " Click column badges to sort"
+ else
+ _span.text_danger "WARNING: the site checker may not understand type: #{col}, results may not be complete/available."
+ end
}
) do
- _p.bold 'ERROR - please try again.'
+ _table.table.table_condensed.table_striped do
+ _thead do
+ _tr do
+ _th! "#{tlp ? 'Project' : 'Podling'}", data_sort: 'string-ins'
+ _th! data_sort: 'string' do
+ _ 'Check Results'
+ _br
+ analysis[0][col].each do |cls, val|
+ _ ' '
+ _span.label val, class: cls
+ end
+ end
+ end
+ end
+ _tbody do
+ sites.each do |n, links|
+ _tr do
+ _td do
+ _a links['display_name'], href: "../project/#{n}"
+ end
+
+ if links[col] =~ /^https?:/
+ _td class: SiteStandards.label(analysis, links, col, n) do
+ _a links[col], href: links[col]
+ end
+ else
+ _td links[col], class: SiteStandards.label(analysis, links, col, n)
+ end
+ end
+ end
+ end
+ end
end
end
-_html do
- _head do
- _style %{
- .table td {font-size: smaller;}
- }
- end
- _body? do
- _whimsy_body(
- title: PAGETITLE,
- subtitle: 'Checking Project/Podling Websites For required content',
- related: {
- "/committers/tools" => "Whimsy Tool Listing",
- "https://www.apache.org/foundation/marks/pmcs#navigation" => "Required PMC Links Policy",
- "https://github.com/apache/whimsy/blob/master/www#{ENV['SCRIPT_NAME']}" => "See This Source Code",
- "mailto:dev@whimsical.apache.org?subject=[SITE] Website Checker Question" => "Questions? Email Whimsy PMC"
- },
+# Display an overview of all checks/sites
+# @param sites site data for all projects
+# @param analysis complete scan data
+# @param checks complete set of checks performed
+# @param tlp true if project (default); podling otherwise
+def display_overview(sites, analysis, checks, tlp = true)
+ _whimsy_panel_table(
+ title: "Site Check - All #{tlp ? 'Project' : 'Podling'} Results",
helpblock: -> {
- _p do
- _ 'This script periodically crawls all Apache project and podling websites to check them for a few specific links or text blocks that all projects are expected to have.'
- _ 'The checks include verifying that all '
- _a 'required links', href: 'https://www.apache.org/foundation/marks/pmcs#navigation'
- _ ' appear on a project homepage, along with an "image" check if project logo files are in apache.org/img'
- end
- _p! do
- _a 'View the crawler code', href: 'https://github.com/apache/whimsy/blob/master/tools/site-scan.rb'
- _ ', '
- _a 'website display code', href: "https://github.com/apache/whimsy/blob/master/www#{ENV['SCRIPT_NAME']}"
- _ ', and '
- _a 'raw JSON data', href: "#{DATAURI}#{SCAN_DATA_FILE}"
- _ '.'
- _br
- _ "Last crawl time: #{crawl_time} over #{sites.size} websites."
+ _ul.list_inline do
+ _li.small "Data key: "
+ analysis[1].each do |cls, desc|
+ _li.label desc, class: cls
+ end
+ _li.small " Click column badges to sort"
end
}
- ) do
-
- if path_info =~ %r{/project/(.+)}
- # details for an individual project
- project = $1
- if sites[project]
- displayProject(project, sites[project], cols, analysis)
- else
- displayError(path_info)
- end
- elsif path_info =~ %r{/check/(.+)}
- # details for a single check
- col = $1
- _whimsy_panel_table(
- title: "Site Check Of Type - #{col.capitalize}",
- helpblock: -> {
- _a href: '../', aria_label: 'Home to site checker' do
- _span.glyphicon.glyphicon_home :aria_hidden
- end
- _span.glyphicon.glyphicon_menu_right
- if CHECKS.include? col
- _ ' Check Results are expected to match the regular expression: '
- _code CHECKS[col].source
- if DOCS.include? col
+ ) do
+ _table.table.table_condensed.table_striped do
+ _thead do
+ _tr do
+ _th! "#{tlp ? 'Project' : 'Podling'}", data_sort: 'string-ins'
+ checks.keys.each do |col|
+ _th! data_sort: 'string' do
+ _a col.capitalize, href: "check/#{col}"
+ _br
+ analysis[0][col].each do |cls, val|
_ ' '
- _a DOCS[col][1], href: DOCS[col][0]
- end
- _li.small " Click column badges to sort"
- else
- _span.text_danger "WARNING: the site checker may not understand type: #{col}, results may not be complete/available."
- end
- }
- ) do
- _table.table.table_condensed.table_striped do
- _thead do
- _tr do
- _th! 'Project', data_sort: 'string-ins'
- _th! data_sort: 'string' do
- _ 'Check Results'
- _br
- analysis[0][col].each do |cls, val|
- _ ' '
- _span.label val, class: cls
- end
- end
- end
- end
- _tbody do
- sites.each do |n, links|
- _tr do
- _td do
- _a links['display_name'], href: "../project/#{n}"
- end
-
- if links[col] =~ /^https?:/
- _td class: label(analysis, links, col, n) do
- _a links[col], href: links[col]
- end
- else
- _td links[col], class: label(analysis, links, col, n)
- end
- end
+ _span.label val, class: cls
end
end
end
end
- else
- # overview
- _whimsy_panel_table(
- title: "Site Check - All Project/Podling Results",
- helpblock: -> {
- _ul.list_inline do
- _li.small "Data key: "
- analysis[1].each do |cls, desc|
- _li.label desc, class: cls
- end
- _li.small " Click column badges to sort"
- end
- }
- ) do
- _table.table.table_condensed.table_striped do
- _thead do
- _tr do
- _th! 'Project', data_sort: 'string-ins'
- cols.each do |col|
- _th! data_sort: 'string' do
- _a col.capitalize, href: "check/#{col}"
- _br
- analysis[0][col].each do |cls, val|
- _ ' '
- _span.label val, class: cls
- end
- end
- end
- end
+ end
+
+ sort_order = {
+ SiteStandards::SITE_PASS => 1,
+ SiteStandards::SITE_WARN => 2,
+ SiteStandards::SITE_FAIL => 3
+ }
+
+ _tbody do
+ sites.each do |n, links|
+ _tr do
+ _td do
+ _a "#{links['display_name']}", href: "project/#{n}"
end
-
- sort_order = {
- SITE_PASS => 1,
- SITE_WARN => 2,
- SITE_FAIL => 3
- }
-
- _tbody do
- sites.each do |n, links|
- _tr do
- _td do
- _a "#{links['display_name']}", href: "project/#{n}"
- end
- cols.each do |c|
- cls = label(analysis, links, c, n)
- _td '', class: cls, data_sort_value: sort_order[cls]
- end
- end
- end
+ checks.keys.each do |c|
+ cls = SiteStandards.label(analysis, links, c, n)
+ _td '', class: cls, data_sort_value: sort_order[cls]
end
end
- end # of _whimsy_panel_table
+ end
end
end
-
- _script %{
- var table = $(".table").stupidtable();
- table.on("aftertablesort", function (event, data) {
- var th = $(this).find("th");
- th.find(".arrow").remove();
- var dir = $.fn.stupidtable.dir;
- var arrow = data.direction === dir.ASC ? "↑" : "↓";
- th.eq(data.column).append('<span class="arrow">' + arrow +'</span>');
- });
+ end
+end
+
+# Display an error page if a suburl we're given isn't supported
+def display_error(path)
+ _whimsy_panel_table(
+ title: "ERROR - bad url provided",
+ helpblock: -> {
+ _a href: '../', aria_label: 'Home to site checker' do
+ _span.glyphicon.glyphicon_home :aria_hidden
+ end
+ _span.glyphicon.glyphicon_menu_right
+ _span.text_danger "ERROR: The path #{path} is not a recognized command for this tool, sorry! "
}
+ ) do
+ _a.bold 'ERROR - please try again.', href: '../'
end
end
+
+# Display our application's data - handles / and project|check/id paths
+def display_application(path, sites, analysis, checks, tlp = true)
+ if path =~ %r{/project/(.+)} # Display a single project
+ if sites[$1]
+ display_project($1, sites[$1], analysis, checks, tlp)
+ else
+ display_error(path)
+ end
+ elsif path =~ %r{/check/(.+)} # Display a single check
+ if checks[$1]
+ display_check($1, sites, analysis, checks, tlp)
+ else
+ display_error(path)
+ end
+ else
+ display_overview(sites, analysis, checks, tlp)
+ end
+end
+
--
To stop receiving notification emails like this one, please contact
curcuru@apache.org.