You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by javalishixml <ja...@163.com> on 2015/05/19 17:22:23 UTC

[users@httpd] Re:Re: [users@httpd] how to block the duplicated requests?


Great thank you, Yehuda.

I see you speak "Though those modules exist, this might be best implemented in your application"

How can I understand it? I should write a c program for apache? or I should write some java web application at tomcat level?

Because this is a web site, it has to face concurrency issue. So I prefer to resolve this issue at httpd level.
Not sure if my thinking is good or not? 

Can you plz explain it more clear...

Looking forward to your nice feedback    :)


At 2015-05-19 22:01:45, "Yehuda Katz" <ye...@ymkatz.net> wrote:

There are a few modules that you might be able to modify to meet your needs - although these things are generally designed for DDoS protection.
mod_security - http://www.modsecurity.org/
mod_evasive - http://www.zdziarski.com/blog/?page_id=442
mod_cband - http://dembol.org/blog/mod_cband/

mod_limitipconn - http://dominia.org/djao/limitipconn2.html
Though those modules exist, this might be best implemented in your application.



Another way to do it would be with a log parsing tool like Fail2Ban.


- Y


On Tue, May 19, 2015 at 4:29 AM, javalishixml <ja...@163.com> wrote:

Hi,

I have a website. It is built by apache + tomcat.

Now we make a lottery activity at this website. But we find that some robots always raise the duplicated requests to hit this lottery activity. It causes that robots almost get all the awards.

So we just want to block these kind of duplicated requests at every interval unit.
For example, we set the interval unit is 3 seconds. The if the robot want to hit the lottery activity in 3 seconds, the website could block this action.

So how to do it? I suppose if we do it at tomcat level, is it a very low performance? Can I do it at apache level? how to do it?
If I could not do it apache level?

Thanks in advance,
Java Coder

Re: [users@httpd] Re:Re: [users@httpd] how to block the duplicated requests?

Posted by Yehuda Katz <ye...@ymkatz.net>.
I would recommend that you write it into your java application. An
application that does a "lottery" with each request should probably include
rate-limiting itself. That way, if you ever make changes, all the logic of
the application is in one place.

- Y

Sent from a gizmo with a very small keyboard and hyperactive autocorrect.
On May 19, 2015 11:22 AM, "javalishixml" <ja...@163.com> wrote:

>
> Great thank you, Yehuda.
>
> I see you speak "Though those modules exist, this might be best
> implemented in your application"
> How can I understand it? I should write a c program for apache? or I
> should write some java web application at tomcat level?
> Because this is a web site, it has to face concurrency issue. So I prefer
> to resolve this issue at httpd level.
> Not sure if my thinking is good or not?
> Can you plz explain it more clear...
>
> Looking forward to your nice feedback    :)
>
> At 2015-05-19 22:01:45, "Yehuda Katz" <ye...@ymkatz.net> wrote:
>
> There are a few modules that you might be able to modify to meet your
> needs - although these things are generally designed for DDoS protection.
>
>    - mod_security - http://www.modsecurity.org/
>    - mod_evasive - http://www.zdziarski.com/blog/?page_id=442
>    - mod_cband - http://dembol.org/blog/mod_cband/
>    - mod_limitipconn - http://dominia.org/djao/limitipconn2.html
>
> Though those modules exist, this might be best implemented in your
> application.
>
> Another way to do it would be with a log parsing tool like Fail2Ban.
>
> - Y
>
> On Tue, May 19, 2015 at 4:29 AM, javalishixml <ja...@163.com>
> wrote:
>
>> Hi,
>>
>> I have a website. It is built by apache + tomcat.
>>
>> Now we make a lottery activity at this website. But we find that some robots always raise the duplicated requests to hit this lottery activity. It causes that robots almost get all the awards.
>>
>> So we just want to block these kind of duplicated requests at every interval unit.
>> For example, we set the interval unit is 3 seconds. The if the robot want to hit the lottery activity in 3 seconds, the website could block this action.
>>
>> So how to do it? I suppose if we do it at tomcat level, is it a very low performance? Can I do it at apache level? how to do it?
>> If I could not do it apache level?
>>
>> Thanks in advance,
>> Java Coder
>>
>>
>>
>>
>
>
>

Re: [users@httpd] Re:Re: [users@httpd] Re:Re: [users@httpd] how to block the duplicated requests?

Posted by Kurtis Rader <kr...@skepticism.us>.
On Wed, May 20, 2015 at 1:26 AM, javalishixml <ja...@163.com> wrote:

> 4. later, we setup a new register page. We change its url from "
> http://mywebsite.com/register1.jsp" to "http://mywebsite.com/register2.jsp
> "
> For the first several days, we find everthing is good.
> But after several days, we find the robot(crackers) will successfully
> register the thousands different users for this web site during only
> several minutes.
>

You just made my point. The fact that you changed the URI should not
require updating the filtering rules. Similarly if you add an RPC interface
(or some other means of interacting with your lottery service) the
front-ends to that interface should not have to know about the rules for
rejecting duplicate requests. That filtering is the responsibility of your
lottery service. It is not the responsibility of the web server.


> Is it a DDOS attack? Is there a good way to resolve it at httpd level?
> Which apache mod is best for my issue?
>

No Apache module is best for your issue because the Apache web server is
the wrong place to implement the request filtering. If Yehuda's and my
arguments haven't convinced you that implementing this logic in the web
server is the wrong place to do it then I would recommend ModSecurity:
http://www.modsecurity.org/.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

[users@httpd] Re:Re: [users@httpd] Re:Re: [users@httpd] how to block the duplicated requests?

Posted by javalishixml <ja...@163.com>.
Great thank you all.


More detail information is as below.


presudo-code step:


1. a register page named "http://mywebsite.com/register1.jsp" is set up, and this page contains a CAPTCHA image 
2. the robot(crackers) will successfully register the thousands different users for this web site during only several minutes.
3. if it is a human beings, these thousands different users should have different IPs. But we find  these thousands different users are from same IPs.
By the way, we get the IP from HttpServletRequest header.
4. later, we setup a new register page. We change its url from "http://mywebsite.com/register1.jsp" to "http://mywebsite.com/register2.jsp"
For the first several days, we find everthing is good.
But after several days, we find the robot(crackers) will successfully register the thousands different users for this web site during only several minutes.


It's just reproduced steps for our case.


Our requirements are that:
1. we have a URL for register page. we don't want the thousands different users with same IP could successfully registered during a very short time window.
2. We can have a policy to set a interval time window. Based on this interval time window, the same IP should not register users again and again.
3. This policy should manage a group of URLs. We can always add the different URLs for this policy.  Because based on our maintaining activities, we may set up many different register page again and again.


Is it a DDOS attack? Is there a good way to resolve it at httpd level? Which apache mod is best for my issue?




在 2015-05-20 05:11:13，"Kurtis Rader" <kr...@skepticism.us> 写道：

On Tue, May 19, 2015 at 8:22 AM, javalishixml <ja...@163.com> wrote:

I see you speak "Though those modules exist, this might be best implemented in your application"

How can I understand it? I should write a c program for apache? or I should write some java web application at tomcat level?

Because this is a web site, it has to face concurrency issue. So I prefer to resolve this issue at httpd level.
Not sure if my thinking is good or not? 





Yehuda is correct. You should implement the rate limiting in your lottery code, not the web server. You are rate limiting lottery attempts, not HTTP requests. Nor are you eliminating duplicate requests. If you've never written a rate limiter the concept is fairly straightforward. Google "token bucket". Below is a simple implementation I use in my httpd and sshd log monitoring app to detect malware that is flooding my server with bogus requests. It uses a single token bucket and is thus much simpler and uses less memory than more flexible implementations.


Here is an example of how it is used:


blackhole_source = False
if web_req.http_method == 'POST':
    rate_limiter = self.rate_limiter_post.get(ip_addr, None)
    if rate_limiter is not None:
        blackhole_source = rate_limiter()
    else:
        # Allow no more than two POST requests every five seconds.
        # In my experience legitimate WordPress administration doesn't
        # exceed this limit.
        self.rate_limiter_post[ip_addr] = util.RateLimiter(2, 5)


if blackhole_source:


And here is the implementation:


class RateLimiter():
    """Return true if the allowable rate has been exceeded."""
    # pylint: disable=too-few-public-methods


    def __init__(self, rate, per):
        """Initialize object for a specific situation (e.g., IP address).


        Note that rate and per must have the same temporal units; e.g., seconds.
        """
        if not isinstance(rate, int) or not isinstance(per, int):
            raise TypeError('rate and per must be integers')
        self.rate = float(rate)
        self.per = float(per)
        self.ratio = self.rate / self.per
        self.allowance = float(rate)
        self.last_check = time.time()


    def __call__(self):
        """Return True if the rate has been exceeded."""
        now = time.time()
        time_delta = now - self.last_check
        self.last_check = now
        self.allowance += time_delta * self.ratio
        if self.allowance > self.rate:
            self.allowance = int(self.rate)  # throttle
        if self.allowance < 1.0:
            return True
        else:
            self.allowance -= 1.0
            return False




--

Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: [users@httpd] Re:Re: [users@httpd] how to block the duplicated requests?

Posted by Kurtis Rader <kr...@skepticism.us>.
On Tue, May 19, 2015 at 8:22 AM, javalishixml <ja...@163.com> wrote:

> I see you speak "Though those modules exist, this might be best
> implemented in your application"
> How can I understand it? I should write a c program for apache? or I
> should write some java web application at tomcat level?
> Because this is a web site, it has to face concurrency issue. So I prefer
> to resolve this issue at httpd level.
> Not sure if my thinking is good or not?
>


Yehuda is correct. You should implement the rate limiting in your lottery
code, not the web server. You are rate limiting lottery attempts, not HTTP
requests. Nor are you eliminating duplicate requests. If you've never
written a rate limiter the concept is fairly straightforward. Google "token
bucket". Below is a simple implementation I use in my httpd and sshd log
monitoring app to detect malware that is flooding my server with bogus
requests. It uses a single token bucket and is thus much simpler and uses
less memory than more flexible implementations.

Here is an example of how it is used:

blackhole_source = False
if web_req.http_method == 'POST':
    rate_limiter = self.rate_limiter_post.get(ip_addr, None)
    if rate_limiter is not None:
        blackhole_source = rate_limiter()
    else:
        # Allow no more than two POST requests every five seconds.
        # In my experience legitimate WordPress administration doesn't
        # exceed this limit.
        self.rate_limiter_post[ip_addr] = util.RateLimiter(2, 5)

if blackhole_source:


And here is the implementation:

class RateLimiter():
    """Return true if the allowable rate has been exceeded."""
    # pylint: disable=too-few-public-methods

    def __init__(self, rate, per):
        """Initialize object for a specific situation (e.g., IP address).

        Note that rate and per must have the same temporal units; e.g.,
seconds.
        """
        if not isinstance(rate, int) or not isinstance(per, int):
            raise TypeError('rate and per must be integers')
        self.rate = float(rate)
        self.per = float(per)
        self.ratio = self.rate / self.per
        self.allowance = float(rate)
        self.last_check = time.time()

    def __call__(self):
        """Return True if the rate has been exceeded."""
        now = time.time()
        time_delta = now - self.last_check
        self.last_check = now
        self.allowance += time_delta * self.ratio
        if self.allowance > self.rate:
            self.allowance = int(self.rate)  # throttle
        if self.allowance < 1.0:
            return True
        else:
            self.allowance -= 1.0
            return False


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank