Learn2ban

Learn2ban

Overview

Learn2Ban is an open source module designed to introduce intelligent, adaptive features to botnet detection and banning by using a machine-learning approach.

While regular expressions work to frustrate most brute force attacks, they are still vulnerable to even minor changes in the pattern or content of requests sent during a Denial of Service attack. By simply rotating the User Agent field or tuning the timeframe of attacks, assailants are able to work around a traditional Fail2Ban-style system. This failing in regex-only banning occurs because each request is considered on its own rather than in the context of all the requests made by a particular IP. Thus each time the IP makes a request, its past behaviour is essentially wiped clean, akin to changing its fingerprints every time. Learn2ban, by contrast, works as an adjunct to the Banjax module by taking an abstract viewpoint of HTTP requests and differentiates legitimate users from attacking botnets. In so doing, it blocks only those requests which meet the criteria for being considered a bot.

Using machine-learning techniques, the system can make decisions about whether or not a given request is spurious. This decision-making capability is contingent upon a range of available information, thereby increasing the number of attack dimensions that can be considered at once. In order to detect botnet behaviour, it is not enough simply to match a recurring pattern; rather, based on a broad range of example data, Learn2Ban is capable of highlighting and capturing significant patterns of behaviour. These patterns act as the fingerprints which identify a DDoS attack. Learn2Ban is an open source module designed to introduce intelligent, adaptive features to botnet detection and banning by using a machine-learning approach.

Machine-learning in brief

In the case of Learn2Ban, our use of the term machine-learning refers to a process of classification. For any given set of data, we want to determine algorithmically which pieces of information represent genuine requests to our servers and which are malicious attempts to damage the system. Machine- learning works by training the system to correctly classify requests, based on presenting it with the sample data of past DDoS attacks. The greater the number and variety of sample data presented, the broader the capability of the model will be. In other words, the more types of DDoS attacks the system sees, the more accurately it will be able to deflect them.

To this end, eQualit.ie has collected a wide range of example logs that detail the many different narratives of DDoS attacks. These logs are annotated, initially using Banjax filters such as Javascript and Captcha Challenges, in order to distinguish between genuine and malicious requests. The annotated logs are then presented to the Learn2Ban system and a model is constructed and tested against attacks it has never seen before. The model is measured for initial accuracy and then multiple experiments are run to refine it until the model is regarded as no longer improvable. The goal is to create a model that is very good at catching specific attacks and still able to recognise new patterns when they first arise.

The determination as to whether a given IP address is malicious is done based on comparisons with pre-computed Support Vector Machine (SVM) models. These models are learned from existing attack examples chosen to allow the greatest coverage of potential attack types.

Learn2Ban concept

Learn2Ban has been designed to work as an additional filter for the Fail2Ban system, building on the existing protocols for blacklisting, banning and blocking attacks. This new extension of its capabilities does not affect the standard operability for Systems Administrators and Operations Systems. Once a suspect IP address has been identified, the predefined rules for (temporarily or permanently) banning the IP are enforced through the Fail2Ban system. Learn2Ban has the added advantage of reducing administration overheads, since it is no longer necessary for Admins to notice attacks - either through system alerts or, in a worst-case scenario, through systems going offline. In its place, Learn2ban proactively detects attacks and responds accordingly to the utilised patterns.

Implementation

As with all BotnetDBP components, Learn2ban works as a standalone module. It can integrate seamlessly with the other moving parts of the Deflect system but can also be easily adapted by others for their own projects.

It takes into account several features of an HTTP request in making its determination. These range from frequency of request to rotation of User Agent for the same IP address, as well as browsing behaviour. Rather than dealing with a single dimension of information, as in the case of a regex filter, Learn2ban is able to consider a requesting IP address’s behaviour as a whole over a given period of time.

The primary algorithm used is an SVM as a binary classifier due to its efficiency in the detection phase. A number of features were implemented to model user browsing behaviour as a vector; features 1 through 7 were implemented in the primary phase and features 8 through 10 were added in the optimization phase. The following is a list of the features considered:

  1. Average request interval - this feature essentially captures the rate of request. An abnormally high rate would be a strong indication that the associated IP is performing an attack
  2. User Agent cycling - in order to frustrate attempts by a traditional regex or pattern-based system, such as Fail2Ban, attackers will generally rotate the User Agent they make per each request. This has the effect of limiting the efficacy of a hard-coded regex. By considering this behaviour in the context of the associated IP’s overall behaviour, Learn2Ban aims to see through this obfuscation.
  3. Page depth request - human users will generally browse through a site in ways which lead to a deeper link exploration than an automated bot is inclined to do. Thus if an IP is generally only requesting certain front-end pages rather than digging into the site, this feature will highlight that behaviour.
  4. Request Depth Standard Deviation - As an adjunct to the depth of a request, this feature considers the standard deviation of a bot’s request.
  5. HTML to Image/CSS ratio - again, unlike a human user, bots will not usually cause requests to load additional content such as images or CSS, since the request is neither focused widely nor browser-based in general.
  6. Variance of request interval - in this case, bots are again unmasked by their fundamental behaviour. Unless the attacker has the foresight to build in an algorithmic random-wait interval between requests, the request interval variance will once again expose the attack. It is quite difficult to model the stochastic behaviour of a human user, so given a large enough sample set, a pattern will generally emerge in the behaviour of a bot.
  7. Payload size - if an IP is consistently requesting large files - such as PDFs - this is a good indication that it is an intentional attack to waste server resources.
  8. HTTP Response Code Rate - This feature considers the type of request that a requester is causing. In many cases, a bot will deliberately cause an error request or make requests for non-existent pages. The latter is a deliberate attempt at a cache-busting attack. In this scenario, the botnet attempts to turn the caching servers against the origin site forcing them to attempt to update their caches for non-existent pages. In this circumstance, the proxy servers essentially switch sides and become an extension of the attacking force, bombarding the protected host with requests.
  9. Session Length - This feature also elucidates general behaviour considering the requester’s interaction with a given sight in terms of session time.
  10. Percentage Consecutive Requests - To further elucidate the requester’s interaction with a given site, we additionally consider how many of the requests made were consecutive as another window onto frequency.

Supervised learning was employed during the training phase. We distinguished between bots and human users by recognizing attacking users in attack logs and identifying them using regular expressions. Later classification was based on this initial training.

Minimum Redundancy Maximum Relevance (mRMR) and a second method based on Principal Component Analysis (PCA) were used to rank the effectiveness of the features in detecting bots from human users. mRMR indicates that, overall, features 1, 9, 4 and 3 were of most importance respectively, with feature 1 appearing significantly more important than any other feature. PCA results are mostly consistent with mRMR measures, indicating features 8, 4, 3 and 1 as the most important features. In a recent attack on the Deflect network, feature 2 was by itself able to detect all bots with 100% accuracy. The addition of features 8 and 9 in the optimization phase did much to improve the accuracy of some models, in some cases raising efficacy of detection from 70% to 90%.

Through experimentation, the accuracy of the Learn2ban tool has been determined to be at 90% and above (i.e. both false positives and true negatives amounted to less than 10%). In several cases, accuracy of 99% was achieved.

Learn2ban 1 Learn2ban 2

Learn2ban is fully configurable by end users. For example in the event of a high rate of attack it may be preferable to ban all IP addresses with greater than 75% certainty of being a bot whereas in low load situations a threshold of 95% would be the optimal setting. The tool, as well as a variety of use cases and configurations, are all fully available and fully open sourced on Github.