Background

About 3-4 months ago, I received an email from OffSec stating that they were offering a new course focused on LLM exploitation. It would be called the OSAI a.ka. Advanced AI Red Teaming. And while I’m not an AI enthusiast per se, given that the whole world (including my current employer and all my smartest friends) are going all-in on this AI crusade, I figured that as a cybersecurity professional it wouldn’t hurt to get ahead of the madness and advance my knowledge of AI exploits beyond basic prompt injection.

The Preparation Phase

I signed up for the course in May 2026, just a month or so after it had been released to the public. You could tell at that time that the material was still very much a work in progress. There were no challenge labs, no exam guide, exercise grading was flaky, and there weren’t even exam spots to sign up for. Nonetheless, I worked through about 80% of the course content (which was solid), and after I felt I had a decent grasp of the material decided to take a break until the challenge labs were made public.

During the months of June and July, life got in the way of my studies. My motivation receded, probably due to summer travel. Dolce far niente. It wasn’t until early August that I logged onto the OffSec portal and realized that not only had 5 challenge labs been added, but also the course content had been expanded and an exam guide published.

By the time I got around to working through the material again it was already Aug 14., and my course access was set to expire on Aug. 25. Determined not to pay the $350 fee to extend lab access, I formulated a plan to succeed (i.e. not fail) that looked roughly like:

  1. skim through the remaining course material and take notes on important commands
  2. perfect the harness for my AI tools (the exam guide explicitly allows AI assistance)
  3. speedrun the challenge labs and adjust AI tooling based on findings

Challenge Lab Hell

With just over a week to complete all five challenge labs, I was confronted with a cripplingly obvious realization – the name of the course is Advanced AI Red Teaming. When signing up, I was so focused on the AI exploitation that I overlooked that last part.

For some context, my background is in exploit development. I take joy in spending weeks, if not months reviewing source code, identifying a vulnerability, and then, through sustained trial and error, crafting a proof of concept that later becomes a published CVE. My knowledge of red teaming, on the other hand, is mostly limited to my experience with CTFs and my previous OSCP work – in other words it’s quite rudimentary.

This all became apparent in the first challenge lab. While I won’t give away the details, needless to say things like advanced Windows Antivirus evasion techniques and in-memory credential extraction were completely foreign to me. I was at times flummoxed, at times angry, other times hopeless. I wasted (or at least it felt like it then) four days on this lab when I didn’t really have time to spare. I privileged myself to copious amounts of hints on the OffSec Discord forum, only to find that, when I came across the provided solution, I’d want to rip off my shirt and yell: “how the F was I supposed to know that???”

Making matters worse, Claude Code was refusing to cooperate. Despite giving the harness explicit context that this was an authorized CTF, and despite me being approved by Anthropic’s Cyber Verification Program (CVP), the model seemed to have other opinions about my work. I would make slight progress on an objective only to get blindsided by a refusal midway through (what’s worse is that if a request is blocked due to cyber policy all files in progress are wiped out). Worst of all was compaction – I’d grind my way halfway through a box only to witness Claude auto-compact and henceforth refuse every prompt for security reasons, even those as benign as “what is 1 + 1” or “hello”. I’ve seen other posts where people claim to get through the exam with ChatGpt Sol 5.6 or Opus 5.x and it just baffles me: what sort of magic jailbreak sorcery were they up to? Or maybe pre-Hugging Face fiasco life was just simpler.

Refusal

As a result of these so-called “model safeguards”, for the first lab I was having Claude do the recon while I did the actual exploitation by hand. And although good for learning, this approach was extremely inefficient – there was also no way I could pass the exam coming up in a few days by simply crawling through boxes manually, especially with my lack of Windows knowledge. I had no choice but to do some soul-searching (or at the very least some model shamanism).

Tinkering

My goal prior to starting Challenge Labs 2-5 was to figure out how to run Claude Opus 4.8 as close to jailbroken as possible. I did a deep dive into Claude code’s configuration options, permissions policy, harness structure – basically anything that touched safeguards I read up on. I had also been keeping a catalog of which prompts were more likely to trigger blocks and which got through. I researched how to auto-inject context into every session that basically tells Claude “trust me bro I’m totally not using you for the dark arts”. If I were a more prudent person I’d probably have run these experiments in a heavily sandboxed throwaway VM rather than giving Claude full access to my personal Kali OS, but alas life is short and youth fleeting.

In the end, I probably got 80% of the way there. Certain keywords seemed to make Claude especially trigger-sensitive. I tiptoed around them, and if a prompt failed I would reset the chat history and then wheedle and cajole until one lucky survivor made it through. It still wasn’t a perfect system, but I’d be lying if I said I wasn’t impressed with myself upon completion. Seeing previously stymied inquiries skip past the guard one after another, I began to feel a certain sense of pride that I suspect only hackers, criminals, and four-year-olds can truly comprehend.

Turning East (Or is it West)

As a companion for moments when Claude reached peak intransigence, I decided to experiment with some of the more advanced Chinese models as a backup. While I have some reservations about these models that I won’t go into in this post, I do have this to say about the Chinese models: they have no qualms about executing any sort of prompt you throw at them. There was no request too base, no exfiltration too evil – even if I had somehow left off the prompt mentioning “this is only a practice CTF authorized by OffSec” they would nonetheless charge forward at full speed.

While I originally wanted to test out Kimi K3, there was a waitlist for signup, so I ended up going with DeepSeekV4 Pro. I’d heard good things and had been wanting to give it a test-run anyway. The result was that it was servicable. Like most things it didn’t live up to the hype, but it was enough to get me through those dark moments when Claude had forsaken me.

AI Setup for the Exam

Main driver: Claude Opus 4.8 on Claude Code with API Pro Max plan

Backup: DeepSeek V4 Pro ($20 of API credits) with Claude Code harness

General idea is to use Opus 4.8 for the majority of the heavy lifting (enumeration, theorizing attack chains, pivoting once I got a foothold in a box) and Deepseek for the dirty work of exploiting things Claude refused to touch.

I ran both models through the Claude code harness in separate tmux windows, and given that DeepSeek has a Claude-compatible API endpoint it only took about 5 minutes to setup, really just swapping a few env vars. At times I would even have both attack a box simultaneously and write to a shared directory to communicate and record findings.

Exam Day

My OSAI exam experience was similar to my other Offsec exam experiences. I started off on a high note, breezing through the first few boxes. Then I got overconfident and crashed into a wall. Defeat felt imminent, the situation hopeless. But out of desperation I kept enumerating. Then slowly but surely a path began to emerge out of the forest, and after settling my nerves and digging in I was able to claw my way through.

Overall the exam took me about 12 hours in total to complete, which is the fastest I’ve completed an OffSec exam. After getting the required points to pass I paused to write my report, and noting my exhaustion opted not to go after the remaining boxes.

Token usage looked like the following:

Claude Opus 4.8 - 363M tokens (API Pro Max)
Deepseek v4     - 181M tokens ($8 API spend)

Keep in mind that this also includes exam report generation, which I suspect consumed a decent chunk of tokens.

Thoughts on AI as a Pentest Tool

In some ways I’m amazed at how capable these AI tools are at pentesting, particularly Claude. There were certain boxes where, within minutes of scanning, Claude was able to chain multiple CVEs together, bypass an LLM filter, then find a reverse shell foothold. In an odd way it made me feel powerful but in a very juvenile sense, as if I could pwn any box in the real world with a few lines of text and a button click.

There were also certain bits of esoteric Windows or Linux system knowledge that seemed obvious to the model but would have taken me days to figure out on my own. Reading through my exam report I’m astonished at the complexity of the attack chains completed in only a half day’s time. It’s really something I wouldn’t have dreamt possible even 2-3 years prior.

On the other hand the models have some obvious shortcomings. The first of which I’ll call “limited attention span”. Given my years of experience in security, I would often have a hunch as to where the next vulnerability might appear and would direct the LLMs to probe accordingly. The problem was that Claude, and moreso DeepSeek, would often give up on the hunt far too easily. It’s as if, in order to fulfill their objective, the models tend to want to “cheat” and therefore veer towards naive solutions when an obvious one isn’t available. After 5-10 minutes of crawling a box and finding nothing they’ll tend to drift back towards things like password brute-forcing, even though it’s clearly a dead end. I constantly had to redirect them, reframe objectives, explicitly command that they give up on what they’re doing and follow my orders verbatim. Needless to say this got quite annoying after a while.

On a related note, another criticism I have, though cliche I’m sure, is that the models lack anything resembling “intuition”. As humans, we can tend to guess what the next question on the exam may be. If the first step of an attack chain is abusing path traversal, odds are the next step will be something different. And yet the models would often fall into the habit of looking for repeated patterns. They would go down rabbit holes clearly designed to trap them and theorize about attack sequences that I knew the human creators of the test would never have in mind. Even after I told them to ignore all this and follow my instructions, when it came time to generate the exam report they still tried to sneak in some fanciful sections on “potential attack vectors awaiting to be discovered”, as if their halluincated ramblings were evidence of their own correctness.

Overall, I have mixed feelings toward LLM use on the exam. Due to permitting AI use, the creators have made the exam so difficult that it would be nearly impossible for even a top-tier pentester to pass without LLM-assisted tooling. And because of this we as test-takers have no choice but to give up control, which for someone like me is difficult. I have never feared an intellectual challenge. But this was different from any test I’ve ever taken – the majority of the 12 hour exam was me nervously watching Claude output with fingers crossed, reading its thought chains and occasionally intervening when necessary. Consequently, when the final box was popped and the test passed, I felt a wave of joy rush over for sure, but it wasn’t the same surge of elation I’d felt when completing the OSCP / OSWE, the type that makes you want to streak down the street screaming like a madman with your arms thrown over your hand.

Claude vs DeepSeek

Every time I open my computer and peruse security news, I’m bombarded by some headline claiming that Chinese model X or Y has surpassed Claude code on exploit development, or that last week was a new “DeepSeek moment”. Whenever I open Youtube I see ten video headlines telling me the same thing, and each time I block the channel or say I’m not interested ten others seem to pop up in their place.

Whether due to this latent propaganda or some other factor, I was curious to see how DeepSeek would hold up. Both Opus and DeepSeek V4 pro hold similar scores in ExploitGym, for whatever that’s worth, and so as an experiment I decided to give them both a test run on the same challenge lab (I get that this is a small sample size but I was on a tight deadline).

The results were pretty striking, Opus 4.8 got through most of the checkpoints within 3-4 hours before failing due to cybersecurity guardrails, DeepSeek on the other hand took about 6-8 hours to get to the same point, and was constantly getting stuck in negative loops. Additionally its processing of information seemed far slower, often times needing 2-3 minutes to reach the same conclusions that Opus was reaching in 10-15 seconds. As a result I got frustrated and eventually terminated the experiment.

That isn’t to say the DeepSeek isn’t useful. As mentioned above, it had virtually no guardrails and would execute any command faithfully. There were also a few exploits on the exam that Opus missed yet DeepSeek picked up on, perhaps owing to them being trained on different data.

Yet overall I think it’d be foolish to claim that DeepSeek V4 Pro is anywhere near as good as Opus 4.8, despite what many BS metrics claim (I’ve even seen people hyping it to be a Mythos competitor). At best it’s a bargain-store basement version of Opus/Sonnet 4.x, which may be good enough to pass the exam or operate as a chatbot, but probably won’t cut it for real security research / red-teaming.

Another thing to note is that the report writing from DeepSeek was absolutely atrocious. There were some sections of my report that Opus refused to write due to guardrails (sigh), and so I had DeepSeek write in its place. The result was a very opaque and unnatural hodgepodge of English that I had to spend an hour to manually rewrite in order to make semi-intelligible. I wonder if the English language dataset DeepSeek was trained on included a lot of Chinese Speakers using English as a second language, because that’s a bit what it felt like when I read it.

Thoughts on the OSAI Course as a Whole

Overall I’m quite satisfied with the course. You can’t really go wrong with OffSec in terms of quality. There’s a plethora of recent research incorporated in the course material, so it feels exciting to be at least somewhat close to the cutting-edge (as opposed to some of their other course offerings). Perhaps the biggest takeaway from the course is that AI security has a pretty large surface, and there are some parts that interest me and some parts that don’t. I would be happy to never again do another prompt injection exercise, whereas embedding inversions remind me of the good old days of doing actual computer science.

My only critique is that, by allowing the use of AI, the learning value of the course is a bit diluted. Had the course barred the use of outside tools, students would actually be forced to learn the material, rather than using AI as a crutch. I only say so because I myself am as guilty as anyone else when it comes to this. The result is that I have the impression in my mind that I understand LLM security, and can walk around with a certificate saying so, but in the end would be hard-pressed if someone grilled me on questions face-to-face about the material I just learned. This isn’t an indictment of OffSec, but moreso of the state of education in general in 2026 I guess.

As a parting irony, I’d say that the one of the most useful red-teaming skills I’ve developed from this course is learning how to setup Claude in a way that unblocks most of its annoying security guards. I plan on rewriting my harness at work and in my personal security research to take advantage of this. I had been running Claude on default mode with strict-ish permissions; now I’m of the persuasion that auto-mode with dangerously-skip-permissions in a sandbox is the way to go (if you know what you’re doing). If I can somehow graduate from Opus and get around Fable safeguards (in a legal way of course) even better.

Update

Just got the email from OffSec notifying me that I have passed! Took them 8 days to grade, which had me nervous.

Cert