Content scanning during checkout/update

[Toplak Daniel <D....@cadenas.de>]

Hi,

I want to implement a server side scanning for malicious content in both ways: when commits arrive and when checkouts updates are send to the client.
The first way is running fine via a pre-commit hook which extracts the content from the transaction and scan the content with clamav.

My problem is to get the 2nd thing working.

I have a running version with mod_clamav (http://software.othello.ch/mod_clamav/) as an apache output filter, but this works only if I turn of the bulk updates with "SVNAllowBulkUpdates off" in the apache config.
Disadvantage of this solutions is, that the checkouts/updates are extremely slow due to the "SVNAllowBulkUpdates off"
If I omit this setting (which is the default) then the checkouts/updates are fast, but the scanning is no longer working.
Does anyone have a good idea to solve this problem?

My system:

-          Debian Linux 6.0 AMD64

-          Apache 2.2.16

-          Subversion 1.6.12

Regards
Daniel Toplak

Re: Content scanning during checkout/update

[Ryan Schmidt <su...@ryandesign.com>]

On Jul 19, 2011, at 12:11, Toplak Daniel wrote:

> My approach via the mod_clamav output filter blocks the content be delivered to the client and breaks the checkout/update with a http status 500 and a information in the http status line.

I'm still curious: does this really work? For an "svn update" for example Subversion only transfers the differences between what the user already has in their working copy and what's in the requested revision on the server, plus it's compressed. Will clamav detect malware that is compressed? Will clamav detect malware that is inserted into an existing file the user already had?

More importantly, is this really a big problem for your setup -- malware getting into the repository? It seems like a rather uncommon situation to me. But if you think it is common for your situation, would it be sufficient to scan the HEAD of the repository for malware periodically -- daily or weekly, or whenever malware definitions are updated? Maybe that would be simpler to implement and perform better than scanning on every access.

AW: Content scanning during checkout/update

[Toplak Daniel <D....@cadenas.de>]

Ryan, as you mention the svnhookdispatcher-fake does not block the checkout so the malicious content is delivered to the client, which I want to avoid.

The scanning on commit is only the half of the security, as you wrote, at the time of the commit the malicious content could not be detected by the heuristic of the scanning process. (f.e. signature is unknown).

My approach via the mod_clamav output filter blocks the content be delivered to the client and breaks the checkout/update with a http status 500 and a information in the http status line.
But this only works when "SVNAllowBulkUpdates off" is set in the server settings, which leads to slow performance.

Regards.
Daniel Toplak

-----Ursprüngliche Nachricht-----
Von: Ryan Schmidt [mailto:subversion-2011a@ryandesign.com] 
Gesendet: Dienstag, 19. Juli 2011 19:03
An: Toplak Daniel
Cc: users@subversion.apache.org
Betreff: Re: Content scanning during checkout/update


On Jul 19, 2011, at 07:06, Toplak Daniel wrote:

> I want to implement a server side scanning for malicious content in both ways: when commits arrive and when checkouts updates are send to the client.
> The first way is running fine via a pre-commit hook which extracts the content from the transaction and scan the content with clamav.
>  
> My problem is to get the 2nd thing working.

Right, that's more difficult since Subversion doesn't have a pre- or post-checkout or -update hook.

I wrote a script to help you fake it, if you're serving the repository using Apache:

http://www.ryandesign.com/svnhookdispatcher/

This wouldn't prevent someone from checking out or updating, but would give you a chance to run a script on the server when they do. If the script finds something it needs to alert the user about, it could do so using external means, like by sending them an email or an instant message.

But why is it that you want to scan not only at commit time but also at checkout/update time? If you scan files for malicious content at commit, isn't that enough? Once you've verified the commit is clean, it'll stay clean; revisions can't be modified later. Or are you worried that someone commits some new malware that's not yet identified by your scanning software, and you want to scan it again later with updated versions of the scanning software?

Re: Content scanning during checkout/update

[Ryan Schmidt <su...@ryandesign.com>]

On Jul 19, 2011, at 07:06, Toplak Daniel wrote:

> I want to implement a server side scanning for malicious content in both ways: when commits arrive and when checkouts updates are send to the client.
> The first way is running fine via a pre-commit hook which extracts the content from the transaction and scan the content with clamav.
>  
> My problem is to get the 2nd thing working.

Right, that's more difficult since Subversion doesn't have a pre- or post-checkout or -update hook.

I wrote a script to help you fake it, if you're serving the repository using Apache:

http://www.ryandesign.com/svnhookdispatcher/

This wouldn't prevent someone from checking out or updating, but would give you a chance to run a script on the server when they do. If the script finds something it needs to alert the user about, it could do so using external means, like by sending them an email or an instant message.

But why is it that you want to scan not only at commit time but also at checkout/update time? If you scan files for malicious content at commit, isn't that enough? Once you've verified the commit is clean, it'll stay clean; revisions can't be modified later. Or are you worried that someone commits some new malware that's not yet identified by your scanning software, and you want to scan it again later with updated versions of the scanning software?

Re: Content scanning during checkout/update

[Mark Phippard <ma...@gmail.com>]

On Tue, Jul 19, 2011 at 8:06 AM, Toplak Daniel <D....@cadenas.de> wrote:


> I want to implement a server side scanning for malicious content in both
> ways: when commits arrive and when checkouts updates are send to the client.
>
> The first way is running fine via a pre-commit hook which extracts the
> content from the transaction and scan the content with clamav.****
>
> ** **
>
> My problem is to get the 2nd thing working.****
>
> ** **
>
> I have a running version with mod_clamav (
> http://software.othello.ch/mod_clamav/) as an apache output filter, but
> this works only if I turn of the bulk updates with “SVNAllowBulkUpdates off”
> in the apache config.****
>
> Disadvantage of this solutions is, that the checkouts/updates are extremely
> slow due to the “SVNAllowBulkUpdates off”****
>
> If I omit this setting (which is the default) then the checkouts/updates
> are fast, but the scanning is no longer working.****
>
> Does anyone have a good idea to solve this problem?
>

You can configure your clients to use ra_serf.  Serf natively uses the
technique that this directive turns on for Neon, except that Serf is
designed for this approach and uses modern techniques to make it perform
acceptable.  Specifically it opens multiple connections to the server and
also uses pipelining.

Add http-library = serf to the end of your servers file on each client to do
this.

Unfortunately, with SVN 1.6 serf is not as stable as Neon.  You might still
find it usable though. It gets better with SVN 1.7 where serf may even
become the default HTTP client.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/