Gremlin now uses AI to break distributed systems faster

What better way to find out what your system can take than by breaking it? Popularized by Netflix more than a decade ago, chaos testing is the practice of deliberately introducing controlled failures into working systems to test their resilience.

Now, a startup that built its business around this practice is using AI to automate more of the testing and troubleshooting process.

Gremlin’s Foresight AI, a new add-on to its current “chaos engineering” services, autonomously breaks software infrastructure to uncover hidden bugs before they pop up in production.

Chaos engineering can be deliciously dangerous for an engineer: Turn off a random server, load balancer or even an entire region of a cloud deployment, then observe how well the rest of the system tries to keep serving customers.

If it does, job well done.

If it's borked, more reliability work is afoot.

Kolton Andrus, one of the Netflix engineers who did chaos engineering at the streaming service, founded Gremlin to bring fault injection to enterprises, along with business-friendly accoutrements such as scoring, executive reporting, and organizational accountability tools.

Adding AI into the mix speeds the process considerably, Andrus told The Register, by automating many of the preparatory and post-testing tasks that used to be done by hand.

Failure-as-a-Service AIOps services are nothing new for the site reliability engineer (SRE), but most of these tools diagnose issues only after the system has already crashed.

"A lot of AI solutions are, 'Hey, we took a guess.

Here you go.

Good luck,'” Andrus said.

Instead, Gremlin purposefully tips over the system and then offers expert advice on how to prevent that from happening again.

To use Gremlin, the user installs agents on their system that can inject latency, kill pods, or max out memory.