The Four Generations of Bots and the Upcoming Fifth Generation

Before reading this topic, we encourage you to understand what are bots and how they work by reading our topic on the DiamWall knowledge base (What are bots and how do they work?).

Bots have advanced greatly from their beginnings as basic scripting tools that employed command line interfaces thanks to the intensifying competition between bot creators and security experts, as well as the growing use of JavaScript and HTML5 web technologies. Bots today use fully functional browsers and are programmed to mimic human behavior when navigating websites and applications, using the mouse, tapping and swiping on mobile devices, and generally attempting to pass for genuine users in order to get over security measures.

Here, at DiamWall we like to qualify the bots based on their generations, so we can divide behaviors and patterns according to each bot, with the achievement of detecting those, based on the current and past technologies, we've defined 4 bots generations being them:

First Generation

First-generation bots were simple crawlers, generally used to carry out simple activities such as:

• Scraping
• Carding
• Form spam

First-generation bots were usually executed from automation programs rather than from browsers and from a limited, fixed set of locations, often from servers rather than desktop locations.

This limited spread of host locations as well as the limited sophistication of these bots means they are relatively easy to spot and put defenses in place to stop them. The first generation of bot defenses typically monitor network activity and take actions such as block listing known bad actor IP addresses or non-residential sources (such as data centers), and putting in rate limits that stop more than a predetermined number of requests from a specific location in a certain amount of time.

Making this easier to understand, these bots have fairly easy structures and they can run simple automatic patterns. They don't have any understanding of the web's basic contents like Cookies and Sessions and they are not difficult to detect and block. A simple script that calls web content or an API with the GET / POST method could be referred to as a first-generation bot.

Second Generation

These bots work with later versions of Chrome and Firefox that support headless mode operation, as well as "headless" web development and testing tools like PhantomJS, SimpleBrowser, and Selenium frameworks. They are capable of running JavaScript and maintaining cookies, unlike first-generation bots. In response to the increasing prevalence of JavaScript issues in websites and applications, botmasters started employing headless browsers.

These bots can perform demands step by step and execute challenges that are used to tackle the bots from the first generation, or bypass attempts to execute those who are used to tackle this generation, however, these bots have a slower run-time compared to the first and second-generation bots.

Impact: These bots are employed in ad fraud, form spam, skewed analytics, and DDoS attacks on applications.

Mitigation: These bots can be identified by the browser and device characteristics of the user's computer, such as the existence of particular JavaScript variables, tampered iframes, sessions, and cookies. Once the bot has been located, its fingerprints can be used to block it.

Analyzing analytics and normal user journeys, followed by a careful examination of traffic patterns in various website parts, is another approach to spotting these bots. These differences can serve as red flags for bots planning to conduct various types of attacks, like unauthorized parties and scraping.

Third Generation

These bots operate by using fully functional browsers that have been either customized or infected with malware. Simple mouse motions and keystrokes, for example, can be simulated by them. They might, however, fail to behave in a random manner akin to humans.

Impact: Third-generation bots are employed for a variety of activities, including account takeover, application DDoS, API abuse, carding, and ad fraud.

Mitigation: Based on device and browser characteristics, third-generation bots are challenging to identify.

Fourth Generation

The most recent generation of bots exhibits more sophisticated human-like interaction traits, such as moving the mouse pointer randomly rather than in straight lines. These bots can even modify their user agents while switching between thousands of IP addresses. Growing evidence suggests that "behavior hijacking" by bot developers, involves recording real users' touches and swipes on hijacked mobile apps in order to more precisely resemble human behavior on a website or app. Due to behavior hijacking, it is considerably more difficult to distinguish between their actions and those of actual users, making them far more difficult to identify. Additionally, the enormous number of users whose browsers and devices have been compromised is responsible for their widespread propagation.

Note that, like the second and third generation, these bots are also running in headless browsers, the normal cases aren't PhantomJS or Selenium but instead, they are running in a pure headless Chromium, so they can mimic the actions of a genuine browser.

Impact: Fourth-generation bots are used for ad fraud, carding, account takeover, application DDoS, high bypass attempts, and API abuse.

Mitigation: To get through security barriers, these bots frequently conduct "low and sluggish" attacks across tens of thousands of IP addresses. There would be a lot of false positives if these bots are detected using superficial interface traits, like mouse movement patterns. Therefore, current strategies are insufficient to mitigate such malware. In order to accurately detect fourth-generation bots with zero false positives, machine learning-based technologies are needed. One such technology is intent-based deep packet inspection (DPI) or deep behavioral analysis (IDBA), which uses semi-supervised machine learning models to identify the intent of bots with the highest precision.

Fifth Generation

As said at the start of the topic, DiamWall prefers to qualify the bots based on their web 'knowledge' being the World Wide Web currently facing fourth-generations bots which are really complex, we should get ready for a fifth-generation.
This generation doesn't exist yet, but it is expected to be a high "upgrade" from its last generation since, we do believe that the fifth generation will be a full Artificial Intelligence bot, which will bring many difficulties to the mitigation systems to avoid those.

Since they might be smart enough to "understand" what is happening and get ready to bypass the security measures, on this day Artificials Intelligence might also be applied to mitigation systems, and we might see a war between two AIs, one trying to bypass and the other trying to mitigate.
Let's wait to see what the future brings to the internet.

Final Words

Such analysis covers the full online property's visitor's journey, with a focus on interaction patterns such as mouse movements, scrolling, and taps, as well as the order of URLs, visited the referrers used, and the amount of time spent on each page. DiamWall can record additional metrics pertaining to the browser stack, IP reputation, fingerprints, Packet Inspections, Managed Challenges, and other features that should be included in this study as well.

Important DiamWall note

Nowadays most companies are still trying to fight third-generation bots, and failing on it, without even being aware that a fourth generation even exists, and weird as it seems, most of these companies are security ones who provide security services to their customers.
Being this, soon or later the fifth generation will be out, and we, as security companies, need to deploy measures and solutions to avoid fifth-generation attacks on our infrastructures before they really exist.
At DiamWall, we can fully mitigate the first, the second, and the third generations of bots without the needing for any human interaction such as Captchas, Javascript challenges, or other kinds of JavaScript mitigation systems, we can do it without Deep Packet Inspection which will analyze all the packets of the connection and search for malicious behavior or malicious patterns.
Regarding the fourth generation of bots, most of them are also mitigated with Deep Packet inspection, the most advanced bots that are out of the internet are also fully detected by us, in case of any bypass attempt to our Packet Inspection systems we automatically detect those and initialize the mitigation with our Managed Challenge.

Learn here more about our Deep Packet Inspection, and also about our managed challenge.

We might also soon, release a blog topic on how DiamWall is getting ready for the fifth generation of bots, stay tuned on the blog!