You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by zw...@apache.org on 2022/04/05 20:12:23 UTC
[trafficserver] 02/02: Adds an IP reputation system to the SNI rate limiter (#8459)
This is an automated email from the ASF dual-hosted git repository.
zwoop pushed a commit to branch 9.2.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
commit a6d1f24a429740baeb43e7c98a2a8eef5b85463b
Author: Leif Hedstrom <zw...@apache.org>
AuthorDate: Tue Apr 5 14:10:37 2022 -0600
Adds an IP reputation system to the SNI rate limiter (#8459)
(cherry picked from commit fac9ad15e388540c366234b4c8873059c3994c29)
---
doc/admin-guide/plugins/rate_limit.en.rst | 62 ++++-
doc/static/images/sdk/SieveLRU.png | Bin 0 -> 137860 bytes
plugins/experimental/rate_limit/Makefile.inc | 1 +
plugins/experimental/rate_limit/ip_reputation.cc | 323 +++++++++++++++++++++++
plugins/experimental/rate_limit/ip_reputation.h | 236 +++++++++++++++++
plugins/experimental/rate_limit/iprep_simu.cc | 299 +++++++++++++++++++++
plugins/experimental/rate_limit/sni_limiter.cc | 104 +++++++-
plugins/experimental/rate_limit/sni_limiter.h | 22 ++
plugins/experimental/rate_limit/sni_selector.cc | 15 +-
9 files changed, 1048 insertions(+), 14 deletions(-)
diff --git a/doc/admin-guide/plugins/rate_limit.en.rst b/doc/admin-guide/plugins/rate_limit.en.rst
index ef3f3246f..5d51ac895 100644
--- a/doc/admin-guide/plugins/rate_limit.en.rst
+++ b/doc/admin-guide/plugins/rate_limit.en.rst
@@ -30,6 +30,17 @@ The limit counters and queues are per remap rule only, i.e. there is
(currently) no way to group transaction limits from different remap rules
into a single rate limiter.
+.. Note::
+ This is still work in progress, in particularly the configuration and
+ the IP reputation system needs some work. In particular:
+
+ * We need a proper YAML configuration overall, allowing us to configure
+ better per service controls as well as sharing resources between remap
+ rules or SNI.
+ * We need reloadable configurations.
+ * The IP reputation currently only works with the global plugin settings.
+ * There is no support for adding allow listed IPs to the IP reputation.
+
Remap Plugin
------------
@@ -96,7 +107,10 @@ Global Plugin
-------------
As a global plugin, the rate limiting currently applies only for TLS enabled
-connections, based on the SNI from the TLS handshake. The basic use is as::
+connections, based on the SNI from the TLS handshake. As a global plugin we
+also have the support of an IP reputation system, see below for configurations.
+
+The basic use is as::
rate_limit.so SNI=www1.example.com,www2.example.com --limit=2 --queue=2 --maxage=10000
@@ -144,6 +158,37 @@ The following options are available:
the plugin will use the FQDN of the SNI associated with each rate limiter instance
created during plugin initialization.
+.. option:: --iprep_buckets
+ The number of LRU buckets to use for the IP reputation. A good number here
+ is 10, but can be configured. The reason for the different buckets is to
+ account for a pseudo-sorted list of IPs on the frequency seen. Too few buckets
+ will not be enough to keep such a sorting, rendering the algorithm useless. To
+ function in our setup, the number of buckets must be less than ``100``.
+
+.. option:: --iprep_bucketsize
+ This is the size of the largest LRU bucket (the `entry bucket`), `15` is a good
+ value. This is a power of 2, so `15` means the largest LRU can hold `32768` entries.
+ Note that this option must be bigger then the `--iprep_buckets` setting, for the
+ bucket halfing to function.
+
+.. option:: --iprep_maxage
+ This is used for aging out entries out of the LRU, the default is `0` which means
+ no aging happens. Even with no aging, entries will eventually fall out of buckets
+ because of the LRU mechanism that kicks in. The aging is here to make sure a spike
+ in traffic from an IP doesn't keep the entry for too long in the LRUs.
+
+.. option:: --iprep_permablock_limit
+ The minimum number of hits an IP must reach to get moved to the permanent bucket.
+ In this bucket, entries will stay for 2x
+
+.. option:: --iprep_permablock_pressure
+ This option specifies from which bucket an IP is allowed to move from into the
+ perma block bucket. A good value here is likely `0` or `1`, which is very conservative.
+
+.. option:: --iprep_permablock_maxage
+ Similar to `--iprep_maxage` above, but only applies to the long term (`perma-block`)
+ bucket. Default is `0`, which means no aging to this bucket is applied.
+
Metrics
-------
Metric names are generated either using defaults or user-supplied values. In either
@@ -189,6 +234,21 @@ A user can specify their own prefixes and tags, but not types or metrics.
``resumed`` Queued connection is resumed.
============== ===================================================================
+IP Reputation
+-------------
+
+The goal of the IP reputation system is to simply try to identify IPs which are more
+likely to be abusive than others. It's not a perfect system, and it relies heavily on
+the notion of pressure. The Sieve LRUs are always filled, so you have to make sure that
+you only start using them when the system thinks it's under pressure.
+
+The Sieve LRU is a chained set of (configurable) LRUs, each with smaller and smaller
+capacity. This essentially adds a notion of partially sorted elements; All IPs in
+LRU <n> generally are more active than the IPs in LRU <n+1>. LRU is specially marked
+for longer term blocking, only the most abusive elements would end up here.
+
+.. figure:: /static/images/sdk/SieveLRU.png
+
Examples
--------
diff --git a/doc/static/images/sdk/SieveLRU.png b/doc/static/images/sdk/SieveLRU.png
new file mode 100644
index 000000000..3e138e46d
Binary files /dev/null and b/doc/static/images/sdk/SieveLRU.png differ
diff --git a/plugins/experimental/rate_limit/Makefile.inc b/plugins/experimental/rate_limit/Makefile.inc
index 72469de5c..95ab01f43 100644
--- a/plugins/experimental/rate_limit/Makefile.inc
+++ b/plugins/experimental/rate_limit/Makefile.inc
@@ -21,4 +21,5 @@ experimental_rate_limit_rate_limit_la_SOURCES = \
experimental/rate_limit/txn_limiter.cc \
experimental/rate_limit/sni_limiter.cc \
experimental/rate_limit/sni_selector.cc \
+ experimental/rate_limit/ip_reputation.cc \
experimental/rate_limit/utilities.cc
diff --git a/plugins/experimental/rate_limit/ip_reputation.cc b/plugins/experimental/rate_limit/ip_reputation.cc
new file mode 100644
index 000000000..129cd9584
--- /dev/null
+++ b/plugins/experimental/rate_limit/ip_reputation.cc
@@ -0,0 +1,323 @@
+/** @file
+
+ Implementation details for the IP reputation classes.
+
+ @section license License
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+#include <iostream>
+#include <cmath>
+
+#include "ip_reputation.h"
+
+namespace IpReputation
+{
+// These static class members are here to calculate a uint64_t hash of an IP
+uint64_t
+SieveLru::hasher(const sockaddr *sock)
+{
+ switch (sock->sa_family) {
+ case AF_INET: {
+ const sockaddr_in *sa = reinterpret_cast<const sockaddr_in *>(sock);
+
+ return (0xffffffff00000000 | sa->sin_addr.s_addr);
+ } break;
+ case AF_INET6: {
+ const sockaddr_in6 *sa6 = reinterpret_cast<const sockaddr_in6 *>(sock);
+
+ return (*reinterpret_cast<uint64_t const *>(sa6->sin6_addr.s6_addr) ^
+ *reinterpret_cast<uint64_t const *>(sa6->sin6_addr.s6_addr + sizeof(uint64_t)));
+ } break;
+ default:
+ // Clearly shouldn't happen ...
+ return 0;
+ break;
+ }
+}
+
+uint64_t
+SieveLru::hasher(const std::string &ip, u_short family) // Mostly a convenience function for testing
+{
+ switch (family) {
+ case AF_INET: {
+ sockaddr_in sa4;
+
+ inet_pton(AF_INET, ip.c_str(), &(sa4.sin_addr));
+ sa4.sin_family = AF_INET;
+ return hasher(reinterpret_cast<const sockaddr *>(&sa4));
+ } break;
+ case AF_INET6: {
+ sockaddr_in6 sa6;
+
+ inet_pton(AF_INET6, ip.c_str(), &(sa6.sin6_addr));
+ sa6.sin6_family = AF_INET6;
+ return hasher(reinterpret_cast<const sockaddr *>(&sa6));
+ } break;
+ default:
+ // Really shouldn't happen ...
+ return 0;
+ }
+}
+// Constructor, setting up the pre-sized LRU buckets etc.
+SieveLru::SieveLru(uint32_t num_buckets, uint32_t size) : _lock(TSMutexCreate())
+{
+ initialize(num_buckets, size);
+}
+
+// Initialize the Sieve LRU object
+void
+SieveLru::initialize(uint32_t num_buckets, uint32_t size)
+{
+ TSMutexLock(_lock);
+ TSAssert(!_initialized); // Don't allow it to be initialized more than once!
+ TSReleaseAssert(size > num_buckets); // Otherwise we can't half the bucket sizes
+
+ _initialized = true;
+ _num_buckets = num_buckets;
+ _size = size;
+
+ uint32_t cur_size = pow(2, 1 + _size - num_buckets);
+
+ _map.reserve(pow(2, size + 2)); // Allow for all the sieve LRUs, and extra room for the allow list
+ _buckets.reserve(_num_buckets + 2); // Two extra buckets, for the block list and allow list
+
+ // Create the other buckets, in smaller and smaller sizes (power of 2)
+ for (uint32_t i = lastBucket(); i <= entryBucket(); ++i) {
+ _buckets[i] = new SieveBucket(cur_size);
+ cur_size *= 2;
+ }
+
+ _buckets[blockBucket()] = new SieveBucket(cur_size / 2); // Block LRU, same size as entry bucket
+ _buckets[allowBucket()] = new SieveBucket(0); // Allow LRU, this is unlimited
+ TSMutexUnlock(_lock);
+}
+
+// Increment the count for an element (will be created / added if new).
+std::tuple<uint32_t, uint32_t>
+SieveLru::increment(KeyClass key)
+{
+ TSMutexLock(_lock);
+ TSAssert(_initialized);
+
+ auto map_it = _map.find(key);
+
+ if (_map.end() == map_it) {
+ // This is a new entry, this can only be added to the last LRU bucket
+ SieveBucket *lru = _buckets[entryBucket()];
+
+ if (lru->full()) { // The LRU is full, replace the last item with a new one
+ auto last = std::prev(lru->end());
+ auto &[l_key, l_count, l_bucket, l_added] = *last;
+
+ lru->moveTop(lru, last);
+ _map.erase(l_key);
+ *last = {key, 1, entryBucket(), SystemClock::now()};
+ } else {
+ // Create a new entry, the date is not used now (unless perma blocked), but could be useful for aging out stale elements.
+ lru->push_front({key, 1, entryBucket(), SystemClock::now()});
+ }
+ _map[key] = lru->begin();
+ TSMutexUnlock(_lock);
+
+ return {entryBucket(), 1};
+ } else {
+ auto &[map_key, map_item] = *map_it;
+ auto &[list_key, count, bucket, added] = *map_item;
+ auto lru = _buckets[bucket];
+ auto max_age = (bucket == blockBucket() ? _perma_max_age : _max_age);
+
+ // Check if the entry is older than max_age (if set), if so just move it to the entry bucket and restart
+ // Yes, this will move likely abusive IPs but they will earn back a bad reputation; The goal here is to
+ // not let "spiked" entries sit in small buckets indefinitely. It also cleans up the code. We only check
+ // the actual system time every 10 request for an IP, if traffic is less frequent than that, the LRU will
+ // age it out properly.
+ if ((_max_age > std::chrono::seconds::zero()) && ((count % 10) == 0) &&
+ (std::chrono::duration_cast<std::chrono::seconds>(SystemClock::now() - added) > max_age)) {
+ auto last_lru = _buckets[entryBucket()];
+
+ count >>= 3; // Age the count by a factor of 1/8th
+ bucket = entryBucket();
+ last_lru->moveTop(lru, map_item);
+ } else {
+ ++count;
+
+ if (bucket > lastBucket()) { // Not in the smallest bucket, so we may promote
+ auto p_lru = _buckets[bucket - 1]; // Move to previous bucket
+
+ if (!p_lru->full()) {
+ p_lru->moveTop(lru, map_item);
+ --bucket;
+ } else {
+ auto p_item = std::prev(p_lru->end());
+ auto &[p_key, p_count, p_bucket, p_added] = *p_item;
+
+ if (p_count <= count) {
+ // Swap places on the two elements, moving both to the top of their respective LRU buckets
+ p_lru->moveTop(lru, map_item);
+ lru->moveTop(p_lru, p_item);
+ --bucket;
+ ++p_bucket;
+ }
+ }
+ } else {
+ // Just move it to the top of the current LRU
+ lru->moveTop(lru, map_item);
+ }
+ }
+ TSMutexUnlock(_lock);
+
+ return {bucket, count};
+ }
+}
+
+// Lookup the status of the IP in the current tables, without modifying anything
+std::tuple<uint32_t, uint32_t>
+SieveLru::lookup(KeyClass key) const
+{
+ TSMutexLock(_lock);
+ TSAssert(_initialized);
+
+ auto map_it = _map.find(key);
+
+ if (_map.end() == map_it) {
+ TSMutexUnlock(_lock);
+
+ return {0, entryBucket()}; // Nothing found, return 0 hits and the entry bucket #
+ } else {
+ auto &[map_key, map_item] = *map_it;
+ auto &[list_key, count, bucket, added] = *map_item;
+
+ TSMutexUnlock(_lock);
+
+ return {bucket, count};
+ }
+}
+
+// A little helper function, to properly move an IP to one of the two special buckets,
+// allow-bucket and block-bucket.
+int32_t
+SieveLru::move_bucket(KeyClass key, uint32_t to_bucket)
+{
+ TSMutexLock(_lock);
+ TSAssert(_initialized);
+
+ auto map_it = _map.find(key);
+
+ if (_map.end() == map_it) {
+ // This is a new entry, add it directly to the special bucket
+ SieveBucket *lru = _buckets[to_bucket];
+
+ if (lru->full()) { // The LRU is full, replace the last item with a new one
+ auto last = std::prev(lru->end());
+ auto &[l_key, l_count, l_bucket, l_added] = *last;
+
+ lru->moveTop(lru, last);
+ _map.erase(l_key);
+ *last = {key, 1, to_bucket, SystemClock::now()};
+ } else {
+ // Create a new entry
+ lru->push_front({key, 1, to_bucket, SystemClock::now()});
+ }
+ _map[key] = lru->begin();
+ } else {
+ auto &[map_key, map_item] = *map_it;
+ auto &[list_key, count, bucket, added] = *map_item;
+ auto lru = _buckets[bucket];
+
+ if (bucket != to_bucket) { // Make sure it's not already blocked
+ auto move_lru = _buckets[to_bucket];
+
+ // Free a space for a new entry, if needed
+ if (move_lru->size() >= move_lru->max_size()) {
+ auto d_entry = std::prev(move_lru->end());
+ auto &[d_key, d_count, d_bucket, d_added] = *d_entry;
+
+ move_lru->erase(d_entry);
+ _map.erase(d_key);
+ }
+ move_lru->moveTop(lru, map_item); // Move the LRU item to the perma-blocks
+ bucket = to_bucket;
+ added = SystemClock::now();
+ }
+ }
+ TSMutexUnlock(_lock);
+
+ return to_bucket; // Just as a convenience, return the destination bucket for this entry
+}
+
+void
+SieveLru::dump()
+{
+ TSMutexLock(_lock);
+ TSAssert(_initialized);
+
+ for (uint32_t i = 0; i < _num_buckets + 1; ++i) {
+ long long cnt = 0, sum = 0;
+ auto lru = _buckets[i];
+
+ std::cout << std::endl
+ << "Dumping bucket " << i << " (size=" << lru->size() << ", max_size=" << lru->max_size() << ")" << std::endl;
+ for (auto &it : *lru) {
+ auto &[key, count, bucket, added] = it;
+
+ ++cnt;
+ sum += count;
+#if 0
+ if (0 == i) { // Also dump the content of the top bucket
+ std::cout << "\t" << key << "; Count=" << count << ", Bucket=" << bucket << std::endl;
+ }
+#endif
+ }
+
+ std::cout << "\tAverage count=" << (cnt > 0 ? sum / cnt : 0) << std::endl;
+ }
+ TSMutexUnlock(_lock);
+}
+
+// Debugging tools, these memory sizes are best guesses to how much memory the containers will actually use
+size_t
+SieveBucket::memorySize() const
+{
+ size_t total = sizeof(SieveBucket);
+
+ total += size() * (2 * sizeof(void *) + sizeof(LruEntry)); // Double linked list + object
+
+ return total;
+}
+
+size_t
+SieveLru::memoryUsed() const
+{
+ TSMutexLock(_lock);
+ TSAssert(_initialized);
+
+ size_t total = sizeof(SieveLru);
+
+ for (uint32_t i = 0; i <= _num_buckets + 1; ++i) {
+ total += _buckets[i]->memorySize();
+ }
+
+ total += _map.size() * (sizeof(void *) + sizeof(SieveBucket::iterator));
+ total += _map.bucket_count() * (sizeof(size_t) + sizeof(void *));
+ TSMutexUnlock(_lock);
+
+ return total;
+}
+
+} // namespace IpReputation
diff --git a/plugins/experimental/rate_limit/ip_reputation.h b/plugins/experimental/rate_limit/ip_reputation.h
new file mode 100644
index 000000000..4abbbcaa0
--- /dev/null
+++ b/plugins/experimental/rate_limit/ip_reputation.h
@@ -0,0 +1,236 @@
+/** @file
+
+ Include file for all the IP reputation classes.
+
+ @section license License
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+#pragma once
+
+#include <string>
+#include <cstdint>
+#include <tuple>
+#include <unordered_map>
+#include <list>
+#include <vector>
+#include <chrono>
+#include <arpa/inet.h>
+
+#include "ts/ts.h"
+
+namespace IpReputation
+{
+using KeyClass = uint64_t;
+using SystemClock = std::chrono::system_clock;
+
+// Key / Count / bucket # (rank, 0-<n>) / time added
+using LruEntry = std::tuple<KeyClass, uint32_t, uint32_t, std::chrono::time_point<SystemClock>>;
+
+// This is a wrapper around a std::list which lets us size limit the list to a
+// certain size.
+class SieveBucket : public std::list<LruEntry>
+{
+public:
+ SieveBucket(uint32_t max_size) : _max_size(max_size) {}
+
+ bool
+ full() const
+ {
+ return (_max_size > 0 ? (size() >= _max_size) : false);
+ }
+
+ size_t
+ max_size() const
+ {
+ return _max_size;
+ }
+
+ // Move an element to the top of an LRU. This *can* move it from the current LRU (bucket)
+ // to another, when promoted to a higher rank.
+ void
+ moveTop(SieveBucket *source_lru, SieveBucket::iterator &item)
+ {
+ splice(begin(), *source_lru, item);
+ }
+
+ // Debugging tools
+ size_t memorySize() const;
+
+private:
+ uint32_t _max_size;
+};
+
+using HashMap = std::unordered_map<KeyClass, SieveBucket::iterator>; // The hash map for finding the entry
+
+// This is a concept / POC: Ranked LRU buckets
+//
+// Also, obviously the std::string here is not awesome, rather, we ought to save the
+// hashed value from the IP as the key (just like the hashed in cache_promote).
+class SieveLru
+{
+public:
+ SieveLru() : _lock(TSMutexCreate()){}; // The unitialized version
+ SieveLru(uint32_t num_buckets, uint32_t size);
+ ~SieveLru()
+ {
+ for (uint32_t i = 0; i <= _num_buckets + 1; ++i) { // Remember to delete the two special allow/block buckets too
+ delete _buckets[i];
+ }
+ }
+
+ void initialize(uint32_t num_buckets = 10, uint32_t size = 15);
+
+ // Return value is the bucket (0 .. num_buckets) that the IP is in, and the
+ // current count of "hits". The lookup version is similar, except it doesn't
+ // modify the status of the IP (read-only).
+ std::tuple<uint32_t, uint32_t> increment(KeyClass key);
+
+ std::tuple<uint32_t, uint32_t>
+ increment(const sockaddr *sock)
+ {
+ return increment(hasher(sock));
+ }
+
+ // Move an IP to the perm-block or perma-allow LRUs. A zero (default) maxage is indefinite (no timeout).
+ uint32_t
+ block(KeyClass key)
+ {
+ return move_bucket(key, blockBucket());
+ }
+
+ uint32_t
+ allow(KeyClass key)
+ {
+ return move_bucket(key, allowBucket());
+ }
+
+ uint32_t
+ block(const sockaddr *sock)
+ {
+ return move_bucket(hasher(sock), blockBucket());
+ }
+
+ uint32_t
+ allow(const sockaddr *sock)
+ {
+ return move_bucket(hasher(sock), allowBucket());
+ }
+
+ // Lookup the current state of an IP
+ std::tuple<uint32_t, uint32_t> lookup(KeyClass key) const;
+
+ std::tuple<uint32_t, uint32_t>
+ lookup(const sockaddr *sock) const
+ {
+ return lookup(hasher(sock));
+ }
+
+ // A helper function to hash an INET or INET6 sockaddr to a 64-bit hash.
+ static uint64_t hasher(const sockaddr *sock);
+ static uint64_t hasher(const std::string &ip, u_short family = AF_INET);
+
+ // Identifying some of the special buckets:
+ //
+ // entryBucket == the highest bucket, where new IPs enter (also the biggest bucket)
+ // lastBucket == the last bucket, which is most likely to be abusive
+ // blockBucket == the bucket where we "permanently" block bad IPs
+ // allowBucket == the bucket where we "permanently" allow good IPs (can not be blocked)
+ uint32_t
+ entryBucket() const
+ {
+ return _num_buckets;
+ }
+
+ constexpr uint32_t
+ lastBucket() const
+ {
+ return 1;
+ }
+
+ constexpr uint32_t
+ blockBucket() const
+ {
+ return 0;
+ }
+
+ uint32_t
+ allowBucket() const
+ {
+ return _num_buckets + 1;
+ }
+
+ size_t
+ bucketSize(uint32_t bucket) const
+ {
+ if (bucket <= (_num_buckets + 1)) {
+ return _buckets[bucket]->size();
+ } else {
+ return 0;
+ }
+ }
+
+ bool
+ initialized() const
+ {
+ return _initialized;
+ }
+
+ // Aging getters and setters
+ std::chrono::seconds
+ maxAge() const
+ {
+ return _max_age;
+ }
+
+ std::chrono::seconds
+ permaMaxAge() const
+ {
+ return _perma_max_age;
+ }
+
+ void
+ maxAge(std::chrono::seconds maxage)
+ {
+ _max_age = maxage;
+ }
+
+ void
+ permaMaxAge(std::chrono::seconds maxage)
+ {
+ _perma_max_age = maxage;
+ }
+
+ // Debugging tool, dumps some info around the buckets
+ void dump();
+ size_t memoryUsed() const;
+
+protected:
+ int32_t move_bucket(KeyClass key, uint32_t to_bucket);
+
+private:
+ HashMap _map;
+ std::vector<SieveBucket *> _buckets;
+ uint32_t _num_buckets = 10; // Leave this at 10 ...
+ uint32_t _size = 0; // Set this up to initialize
+ std::chrono::seconds _max_age = std::chrono::seconds::zero(); // Aging time in the SieveLru (default off)
+ std::chrono::seconds _perma_max_age = std::chrono::seconds::zero(); // Aging time in the SieveLru for perma-blocks
+ bool _initialized = false; // If this has been properly initialized yet
+ TSMutex _lock; // The lock around all data access
+};
+
+} // namespace IpReputation
diff --git a/plugins/experimental/rate_limit/iprep_simu.cc b/plugins/experimental/rate_limit/iprep_simu.cc
new file mode 100644
index 000000000..887df68b8
--- /dev/null
+++ b/plugins/experimental/rate_limit/iprep_simu.cc
@@ -0,0 +1,299 @@
+
+/** @file
+
+ Simulator application, for testing the behavior of the SieveLRU. This does
+ not build as part of the system, but put here for future testing etc.
+
+ @section license License
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+#include <getopt.h>
+
+#include <vector>
+#include <fstream>
+#include <chrono>
+#include <iostream>
+#include <iomanip>
+#include <cstdint>
+
+// Yeh well, sue me, boost is useful here, and this is not part of the actual core code
+#include <boost/algorithm/string.hpp>
+
+#include "ip_reputation.h"
+
+// Convenience class declarations
+using IpMap = std::unordered_map<IpReputation::KeyClass, std::tuple<int, bool>>; // count / false = good, true = bad
+using IpList = std::vector<IpMap::iterator>;
+
+// Holds all command line options
+struct CmdConfigs {
+ uint32_t start_buckets, end_buckets, incr_buckets;
+ uint32_t start_size, end_size, incr_size;
+ uint32_t start_threshold, end_threshold, incr_threshold;
+ uint32_t start_permablock, end_permablock, incr_permablock;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+// Command line options / parsing, returns the parsed and populate CmdConfig
+// structure (from above).
+//
+std::tuple<int32_t, int32_t, int32_t>
+splitArg(std::string str)
+{
+ int32_t start = 0, end = 0, incr = 1;
+ std::vector<std::string> results;
+
+ boost::split(results, str, [](char c) { return c == '-' || c == '/'; });
+
+ if (results.size() > 0) {
+ start = std::stoi(results[0]);
+ if (results.size() > 1) {
+ end = std::stoi(results[1]);
+ if (results.size() > 2) {
+ incr = std::stoi(results[2]);
+ }
+ } else {
+ end = start;
+ }
+ } else {
+ std::cerr << "Malformed argument: " << str << std::endl;
+ }
+
+ return {start, end, incr};
+}
+
+CmdConfigs
+parseArgs(int argc, char **argv)
+{
+ CmdConfigs options;
+ int c;
+ constexpr struct option long_options[] = {
+ {"help", no_argument, NULL, 'h'}, {"buckets", required_argument, NULL, 'b'}, {"perma", required_argument, NULL, 'p'},
+ {"size", required_argument, NULL, 's'}, {"threshold", required_argument, NULL, 't'}, {NULL, 0, NULL, 0}};
+
+ // Make sure the optional values have been set
+
+ options.start_permablock = 0;
+ options.end_permablock = 0;
+ options.incr_permablock = 1;
+
+ while (1) {
+ int ix = 0;
+
+ c = getopt_long(argc, argv, "b:f:p:s:t:h?", long_options, &ix);
+ if (c == -1)
+ break;
+
+ switch (c) {
+ case 'h':
+ case '?':
+ std::cerr << "usage: iprep_simu -b|--buckets <size>[-<end bucket range>[/<increment>]]" << std::endl;
+ std::cerr << " -s|--size <bucket size>[-<end bucket size range>[/<increment>]]" << std::endl;
+ std::cerr << " -t|--threshold <bucket num>[-<end bucket num range>[/<increment>]]" << std::endl;
+ std::cerr << " [-p|--perma <permablock>[-<end permablock range>[/<increment>]]]" << std::endl;
+ std::cerr << " [-h|--help" << std::endl;
+ exit(0);
+ break;
+ case 'b':
+ std::tie(options.start_buckets, options.end_buckets, options.incr_buckets) = splitArg(optarg);
+ break;
+ case 's':
+ std::tie(options.start_size, options.end_size, options.incr_size) = splitArg(optarg);
+ break;
+ case 'p':
+ std::tie(options.start_permablock, options.end_permablock, options.incr_permablock) = splitArg(optarg);
+ break;
+ case 't':
+ std::tie(options.start_threshold, options.end_threshold, options.incr_threshold) = splitArg(optarg);
+ break;
+ default:
+ fprintf(stderr, "getopt returned weird stuff: 0%o\n", c);
+ exit(-1);
+ break;
+ }
+ }
+
+ return options; // RVO
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Load a configuration file, and populate the two structures with the
+// list of IPs (and their status) as well as the full sequence of requests.
+//
+// Returns a tuple with the number of good requests and bad requests, respectively.
+//
+std::tuple<uint32_t, uint32_t>
+loadFile(std::string fname, IpMap &all_ips, IpList &ips)
+{
+ std::ifstream infile(fname);
+
+ float timestamp; // The timestamp from the request(relative)
+ std::string ip; // The IP
+ bool status; // Bad (false) or Good (true) request?
+
+ uint32_t good_ips = 0;
+ uint32_t bad_ips = 0;
+ uint32_t good_requests = 0;
+ uint32_t bad_requests = 0;
+
+ // Load in the entire file, and fill the request vector as well as the IP lookup table (state)
+ while (infile >> timestamp >> ip >> status) {
+ auto ip_hash = IpReputation::SieveLru::hasher(ip, ip.find(':') != std::string::npos ? AF_INET6 : AF_INET);
+ auto it = all_ips.find(ip_hash);
+
+ if (!status) {
+ ++good_requests;
+ } else {
+ ++bad_requests;
+ }
+
+ if (all_ips.end() != it) {
+ auto &[key, data] = *it;
+ auto &[count, d_status] = data;
+
+ ++count;
+ ips.push_back(it);
+ } else {
+ all_ips[ip_hash] = {0, status};
+ ips.push_back(all_ips.find(ip_hash));
+ if (!status) {
+ ++good_ips;
+ } else {
+ ++bad_ips;
+ }
+ }
+ }
+
+ std::cout << std::setprecision(3);
+ std::cout << "Total number of requests: " << ips.size() << std::endl;
+ std::cout << "\tGood requests: " << good_requests << " (" << 100.0 * good_requests / ips.size() << "%)" << std::endl;
+ std::cout << "\tBad requests: " << bad_requests << " (" << 100.0 * bad_requests / ips.size() << "%)" << std::endl;
+ std::cout << "Unique IPs in set: " << all_ips.size() << std::endl;
+ std::cout << "\tGood IPs: " << good_ips << " (" << 100.0 * good_ips / all_ips.size() << "%)" << std::endl;
+ std::cout << "\tBad IPs: " << bad_ips << " (" << 100.0 * bad_ips / all_ips.size() << "%)" << std::endl;
+ std::cout << std::endl;
+
+ return {good_requests, bad_requests};
+}
+
+int
+main(int argc, char *argv[])
+{
+ auto options = parseArgs(argc, argv);
+
+ // All remaining arguments should be files, so lets process them one by one
+ for (int file_num = optind; file_num < argc; ++file_num) {
+ IpMap all_ips;
+ IpList ips;
+
+ // Load the data from file
+ auto [good_requests, bad_requests] = loadFile(argv[file_num], all_ips, ips);
+
+ // Here starts the actual simulation, loop through variations
+ for (uint32_t size = options.start_size; size <= options.end_size; size += options.incr_size) {
+ for (uint32_t buckets = options.start_buckets; buckets <= options.end_buckets; buckets += options.incr_buckets) {
+ for (uint32_t threshold = options.start_threshold; threshold <= options.end_threshold;
+ threshold += options.incr_threshold) {
+ for (uint32_t permablock = options.start_permablock; permablock <= options.end_permablock;
+ permablock += options.incr_permablock) {
+ // Setup the buckets and metrics for this loop
+ IpReputation::SieveLru ipt(buckets, size);
+
+ auto start = std::chrono::system_clock::now();
+
+ // Some metrics
+ uint32_t good_blocked = 0;
+ uint32_t good_allowed = 0;
+ uint32_t bad_blocked = 0;
+ uint32_t bad_allowed = 0;
+ uint32_t good_perm_blocked = 0;
+ uint32_t bad_perm_blocked = 0;
+
+ for (auto iter : ips) {
+ auto &[ip, data] = *iter;
+ auto &[count, status] = data;
+ auto [bucket, cur_cnt] = ipt.increment(ip);
+
+ // Currently we only allow perma-blocking on items in bucket 1, so check for that first.
+ if (cur_cnt > permablock && bucket == ipt.lastBucket()) {
+ bucket = ipt.block(ip);
+ }
+
+ if (bucket == ipt.blockBucket()) {
+ if (!status) {
+ ++good_perm_blocked;
+ } else {
+ ++bad_perm_blocked;
+ }
+ } else if (bucket <= threshold) {
+ if (!status) {
+ ++good_blocked;
+ } else {
+ ++bad_blocked;
+ }
+ } else {
+ if (!status) {
+ ++good_allowed;
+ } else {
+ ++bad_allowed;
+ }
+ }
+ }
+
+ auto end = std::chrono::system_clock::now();
+
+ uint32_t total_blocked = bad_blocked + good_blocked;
+ uint32_t total_perm_blocked = bad_perm_blocked + good_perm_blocked;
+ uint32_t total_allowed = bad_allowed + good_allowed;
+
+ // ipt.dump();
+
+ std::chrono::duration<double> elapsed_seconds = end - start;
+
+ std::cout << "Running with size=" << size << ", buckets=" << buckets << ", threshold=" << threshold
+ << ", permablock=" << permablock << std::endl;
+ std::cout << "Processing time: " << elapsed_seconds.count() << std::endl;
+ std::cout << "Denied requests: " << total_blocked + total_perm_blocked << std::endl;
+ std::cout << "\tGood requests denied: " << good_blocked + good_perm_blocked << " ("
+ << 100.0 * (good_blocked + good_perm_blocked) / good_requests << "%)" << std::endl;
+ std::cout << "\tBad requests denied: " << bad_blocked + bad_perm_blocked << " ("
+ << 100.0 * (bad_blocked + bad_perm_blocked) / bad_requests << "%)" << std::endl;
+ std::cout << "Allowed requests: " << total_allowed << std::endl;
+ std::cout << "\tGood requests allowed: " << good_allowed << " (" << 100.0 * good_allowed / good_requests << "%)"
+ << std::endl;
+ std::cout << "\tBad requests allowed: " << bad_allowed << " (" << 100.0 * bad_allowed / bad_requests << "%)"
+ << std::endl;
+ if (permablock) {
+ std::cout << "Permanently blocked IPs: " << ipt.bucketSize(ipt.blockBucket()) << std::endl;
+ std::cout << "\tGood requests permanently denied: " << good_perm_blocked << " ("
+ << 100.0 * good_perm_blocked / good_requests << "%)" << std::endl;
+ std::cout << "\tBad requests permanently denied: " << bad_perm_blocked << " ("
+ << 100.0 * bad_perm_blocked / bad_requests << "%)" << std::endl;
+ }
+ std::cout << "Estimated score (lower is better): "
+ << 100.0 * ((100.0 * good_blocked / good_requests + 100.0 * bad_allowed / bad_requests) /
+ (100.0 * good_allowed / good_requests + 100.0 * bad_blocked / bad_requests))
+ << std::endl;
+ std::cout << "Memory used for IP Reputation data: " << ipt.memoryUsed() / (1024.0 * 1024.0) << "MB" << std::endl
+ << std::endl;
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/plugins/experimental/rate_limit/sni_limiter.cc b/plugins/experimental/rate_limit/sni_limiter.cc
index b63c50b1d..fcc8fb838 100644
--- a/plugins/experimental/rate_limit/sni_limiter.cc
+++ b/plugins/experimental/rate_limit/sni_limiter.cc
@@ -43,18 +43,58 @@ sni_limit_cont(TSCont contp, TSEvent event, void *edata)
int len;
const char *server_name = TSVConnSslSniGet(vc, &len);
std::string_view sni_name(server_name, len);
+ SniRateLimiter *limiter = selector->find(sni_name);
- if (!sni_name.empty()) { // This should likely always succeed, but without it we can't do anything
- SniRateLimiter *limiter = selector->find(sni_name);
+ if (limiter) {
+ // Check if we have an IP reputation for this SNI, and if we should block
+ if (limiter->iprep.initialized()) {
+ const sockaddr *sock = TSNetVConnRemoteAddrGet(vc);
+ int pressure = limiter->pressure();
+
+ TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s, pressure=%d", static_cast<int>(sni_name.length()), sni_name.data(), pressure);
+
+ // TSDebug(PLUGIN_NAME, "IP Reputation: pressure is currently %d", pressure);
+
+ if (pressure >= 0) { // When pressure is < 0, we're not yet at a level of pressure to be concerned about
+ char client_ip[INET6_ADDRSTRLEN] = "[unknown]";
+ auto [bucket, cur_cnt] = limiter->iprep.increment(sock);
+
+ // Get the client IP string if debug is enabled
+ if (TSIsDebugTagSet(PLUGIN_NAME)) {
+ if (sock->sa_family == AF_INET) {
+ inet_ntop(AF_INET, &(((struct sockaddr_in *)sock)->sin_addr), client_ip, INET_ADDRSTRLEN);
+ } else if (sock->sa_family == AF_INET6) {
+ inet_ntop(AF_INET6, &(((struct sockaddr_in6 *)sock)->sin6_addr), client_ip, INET6_ADDRSTRLEN);
+ }
+ }
+
+ if (cur_cnt > limiter->iprep_permablock_count &&
+ bucket <= limiter->iprep_permablock_threshold) { // Mark for long-term blocking
+ TSDebug(PLUGIN_NAME, "Marking IP=%s for perma-blocking", client_ip);
+ bucket = limiter->iprep.block(sock);
+ }
+
+ if (static_cast<uint32_t>(pressure) > bucket) { // Remember the perma-block bucket is always 0, and we are >=0 already
+ // Block this IP from finishing the handshake
+ TSDebug(PLUGIN_NAME, "Rejecting connection from IP=%s, we're at pressure and IP was chosen to be blocked", client_ip);
+ TSUserArgSet(vc, gVCIdx, nullptr);
+ TSVConnReenableEx(vc, TS_EVENT_ERROR);
+
+ return TS_ERROR;
+ }
+ }
+ } else {
+ TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s, no IP reputation", static_cast<int>(sni_name.length()), sni_name.data());
+ }
- TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s", static_cast<int>(sni_name.length()), sni_name.data());
- if (limiter && !limiter->reserve()) {
+ // If we passed the IP reputation filter, continue rate limiting these connections
+ if (!limiter->reserve()) {
if (!limiter->max_queue || limiter->full()) {
// We are running at limit, and the queue has reached max capacity, give back an error and be done.
- TSVConnReenableEx(vc, TS_EVENT_ERROR);
TSDebug(PLUGIN_NAME, "Rejecting connection, we're at capacity and queue is full");
TSUserArgSet(vc, gVCIdx, nullptr);
limiter->incrementMetric(RATE_LIMITER_METRIC_REJECTED);
+ TSVConnReenableEx(vc, TS_EVENT_ERROR);
return TS_ERROR;
} else {
@@ -69,11 +109,11 @@ sni_limit_cont(TSCont contp, TSEvent event, void *edata)
TSVConnReenable(vc);
}
} else {
+ // No limiter for this SNI at all, clear the args etc. just in case
+ TSUserArgSet(vc, gVCIdx, nullptr);
TSVConnReenable(vc);
}
-
- break;
- }
+ } break;
case TS_EVENT_VCONN_CLOSE: {
SniRateLimiter *limiter = static_cast<SniRateLimiter *>(TSUserArgGet(vc, gVCIdx));
@@ -107,10 +147,19 @@ SniRateLimiter::initialize(int argc, const char *argv[])
{const_cast<char *>("maxage"), required_argument, nullptr, 'm'},
{const_cast<char *>("prefix"), required_argument, nullptr, 'p'},
{const_cast<char *>("tag"), required_argument, nullptr, 't'},
+ // These are all for the IP reputation system. ToDo: These should be global rather than per SNI ?
+ {const_cast<char *>("iprep_maxage"), required_argument, nullptr, 'a'},
+ {const_cast<char *>("iprep_buckets"), required_argument, nullptr, 'B'},
+ {const_cast<char *>("iprep_bucketsize"), required_argument, nullptr, 'S'},
+ {const_cast<char *>("iprep_permablock_limit"), required_argument, nullptr, 'L'},
+ {const_cast<char *>("iprep_permablock_pressure"), required_argument, nullptr, 'P'},
+ {const_cast<char *>("iprep_permablock_maxage"), required_argument, nullptr, 'A'},
// EOF
{nullptr, no_argument, nullptr, '\0'},
};
+ TSDebug(PLUGIN_NAME, "Initializing an SNI Rate Limiter");
+
while (true) {
int opt = getopt_long(argc, const_cast<char *const *>(argv), "", longopt, nullptr);
@@ -130,11 +179,50 @@ SniRateLimiter::initialize(int argc, const char *argv[])
case 't':
this->tag = std::string(optarg);
break;
+ case 'a':
+ this->_iprep_max_age = std::chrono::seconds(strtol(optarg, nullptr, 10));
+ break;
+ case 'B':
+ this->_iprep_num_buckets = strtol(optarg, nullptr, 10);
+ if (this->_iprep_num_buckets >= 100) {
+ TSError("sni_limiter: iprep_num_buckets must be in the range 1 .. 99, IP reputation disabled");
+ this->_iprep_num_buckets = 0;
+ }
+ break;
+ case 'S':
+ this->_iprep_size = strtol(optarg, nullptr, 10);
+ break;
+ case 'L':
+ this->iprep_permablock_count = strtol(optarg, nullptr, 10);
+ break;
+ case 'P':
+ this->iprep_permablock_threshold = strtol(optarg, nullptr, 10);
+ break;
+ case 'A':
+ this->_iprep_perma_max_age = std::chrono::seconds(strtol(optarg, nullptr, 10));
+ break;
}
if (opt == -1) {
break;
}
}
+ // Enable and initialize the IP reputation if asked for
+ if (this->_iprep_num_buckets > 0 && this->_iprep_size > 0) {
+ TSDebug(PLUGIN_NAME, "Calling and _initialized is %d\n", this->iprep.initialized());
+ this->iprep.initialize(this->_iprep_num_buckets, this->_iprep_size);
+ TSDebug(PLUGIN_NAME, "IP-reputation enabled with %u buckets, max size is 2^%u", this->_iprep_num_buckets, this->_iprep_size);
+
+ TSDebug(PLUGIN_NAME, "Called and _initialized is %d\n", this->iprep.initialized());
+
+ // These settings are optional
+ if (this->_iprep_max_age != std::chrono::seconds::zero()) {
+ this->iprep.maxAge(this->_iprep_max_age);
+ }
+ if (this->_iprep_perma_max_age != std::chrono::seconds::zero()) {
+ this->iprep.permaMaxAge(this->_iprep_perma_max_age);
+ }
+ }
+
return true;
}
diff --git a/plugins/experimental/rate_limit/sni_limiter.h b/plugins/experimental/rate_limit/sni_limiter.h
index 3889a0819..93b1b5558 100644
--- a/plugins/experimental/rate_limit/sni_limiter.h
+++ b/plugins/experimental/rate_limit/sni_limiter.h
@@ -18,6 +18,7 @@
#pragma once
#include "limiter.h"
+#include "ip_reputation.h"
#include "ts/ts.h"
int sni_limit_cont(TSCont contp, TSEvent event, void *edata);
@@ -40,4 +41,25 @@ public:
}
bool initialize(int argc, const char *argv[]);
+
+ // ToDo: this ought to go into some better global IP reputation pool / settings. Waiting for YAML...
+ IpReputation::SieveLru iprep;
+ uint32_t iprep_permablock_count = 0; // "Hits" limit for blocking permanently
+ uint32_t iprep_permablock_threshold = 0; // Pressure threshold for permanent block
+
+ // Calculate the pressure, which is either a negative number (ignore), or a number 0-<buckets>.
+ // 0 == block only perma-blocks.
+ int32_t
+ pressure() const
+ {
+ return ((active() - 1) / static_cast<float>(limit) * 100) - (99 - _iprep_num_buckets);
+ }
+
+private:
+ // ToDo: These should be moved to global configurations to have one shared IP Reputation.
+ // today the configuration of this is so klunky, that there is no easy way to make it "global".
+ std::chrono::seconds _iprep_max_age = std::chrono::seconds::zero(); // Max age in the SieveLRUs for regular buckets
+ std::chrono::seconds _iprep_perma_max_age = std::chrono::seconds::zero(); // Max age in the SieveLRUs for perma-block buckets
+ uint32_t _iprep_num_buckets = 10; // Number of buckets. ToDo: leave this at 10 always
+ uint32_t _iprep_size = 15; // Size of the biggest bucket; 15 == 2^15 == 32768
};
diff --git a/plugins/experimental/rate_limit/sni_selector.cc b/plugins/experimental/rate_limit/sni_selector.cc
index d41b4df06..e4c22d02d 100644
--- a/plugins/experimental/rate_limit/sni_selector.cc
+++ b/plugins/experimental/rate_limit/sni_selector.cc
@@ -86,6 +86,10 @@ SniSelector::insert(std::string_view sni, SniRateLimiter *limiter)
SniRateLimiter *
SniSelector::find(std::string_view sni)
{
+ if (sni.empty()) { // Likely shouldn't happen, but we can shortcircuit
+ return nullptr;
+ }
+
auto limiter = _limiters.find(sni);
if (limiter != _limiters.end()) {
@@ -105,17 +109,18 @@ SniSelector::factory(const char *sni_list, int argc, const char *argv[])
char *saveptr;
char *sni = strdup(sni_list); // We make a copy of the sni list, to not touch the original string
char *token = strtok_r(sni, ",", &saveptr);
- SniRateLimiter def_limiter;
-
- def_limiter.initialize(argc, argv); // Creates the template limiter
- _needs_queue_cont = (def_limiter.max_queue > 0);
+ // Todo: We are repeating initializing here with the same configurations, but once we move this to
+ // YAML, and refactor this, it'll be better. And this is not particularly expensive.
while (nullptr != token) {
- SniRateLimiter *limiter = new SniRateLimiter(def_limiter); // Make a shallow copy
+ SniRateLimiter *limiter = new SniRateLimiter();
TSReleaseAssert(limiter);
+ limiter->initialize(argc, argv);
limiter->description = token;
+ _needs_queue_cont = (limiter->max_queue > 0);
+
insert(std::string_view(limiter->description), limiter);
token = strtok_r(nullptr, ",", &saveptr);
}