Adding more website checks to Whismy?

[Christian Grobmeier <gr...@apache.org>]

Hello Whimsy developers,

today I found this: 
https://whimsy.apache.org/site/

Is it possible for me to add a few more checks? In example, I would like to check if a site does contain Google Analytics, TagManager or Google Fonts. There might be more to check. GA is a privacy related problem, so I thought how I could check how many of our sites are using known, critical things like GA.

My question is, if you would be willing to accept code to perform those checks and where would I add it?

Kind regards,
Christian 

Re: Adding more website checks to Whismy?

[Shane Curcuru <as...@shanecurcuru.org>]

Christian Grobmeier wrote on 11/13/20 9:17 AM:
> Hello Whimsy developers,
> 
> today I found this: 
> https://whimsy.apache.org/site/
> 
> Is it possible for me to add a few more checks? In example, I would like to check if a site does contain Google Analytics, TagManager or Google Fonts. There might be more to check. GA is a privacy related problem, so I thought how I could check how many of our sites are using known, critical things like GA.
> 
> My question is, if you would be willing to accept code to perform those checks and where would I add it?

Great idea!

As Sam notes elsethread, the code is structured to separate out the
various checks as much as possible from the crawling and the potential
validation display of the data.

In particular, be sure to carefully note the status of any new checks,
especially if they are scanning best practices vs. strictly required
website policies.  We've had issues in the past with edge cases where
some projects were worried they were "red", even though they otherwise
met whatever policies were actually required.

For example, while it's useful to know where Google Fonts is being used,
it's not a policy to use (or not) that service.  Likewise, if we have a
specific policy forbidding Google Analytics, be sure to link to the
policy in any code comments.

-- 

- Shane
  Whimsy PMC
  The Apache Software Foundation

Re: Adding more website checks to Whismy?

[Matt Sicker <bo...@gmail.com>]

Absolutely! You can open a PR on
https://github.com/apache/whimsy

On Fri, Nov 13, 2020 at 08:17 Christian Grobmeier <gr...@apache.org>
wrote:

> Hello Whimsy developers,
>
> today I found this:
> https://whimsy.apache.org/site/
>
> Is it possible for me to add a few more checks? In example, I would like
> to check if a site does contain Google Analytics, TagManager or Google
> Fonts. There might be more to check. GA is a privacy related problem, so I
> thought how I could check how many of our sites are using known, critical
> things like GA.
>
> My question is, if you would be willing to accept code to perform those
> checks and where would I add it?
>
> Kind regards,
> Christian
>

Re: Adding more website checks to Whismy?

[Sam Ruby <ru...@intertwingly.net>]

On Fri, Nov 13, 2020 at 9:17 AM Christian Grobmeier
<gr...@apache.org> wrote:
>
> Hello Whimsy developers,
>
> today I found this:
> https://whimsy.apache.org/site/
>
> Is it possible for me to add a few more checks? In example, I would like to check if a site does contain Google Analytics, TagManager or Google Fonts. There might be more to check. GA is a privacy related problem, so I thought how I could check how many of our sites are using known, critical things like GA.
>
> My question is, if you would be willing to accept code to perform those checks and where would I add it?

The following code performs the scan:

https://github.com/apache/whimsy/blob/master/tools/site-scan.rb

The results of the scan are placed here:

https://whimsy.apache.org/public/site-scan.json

The page you are referencing is produced by this script:

https://github.com/apache/whimsy/blob/master/www/site.cgi

That script delegates the production of the page to be displayed to

https://github.com/apache/whimsy/blob/master/lib/whimsy/sitewebsite.rb

> Kind regards,
> Christian

- Sam Ruby