A recent blog post has been going around talking about the differences between competitions like CCDC (Collegiate Cyber Defense Competition) and CTFs (Capture the Flag). It’s a good read and I thought a lot of interesting points were brought up, and figured it could be beneficial to try to respond to them some place to encourage discussion (and besides, our blog needs any excuse it can get for new content). For what it’s worth, I didn’t watch Chris Eagle’s presentations, though I imagine I don’t agree with everything he said. Likewise, I don’t disagree with everything from Matt Weeks’ blog post. This post is mostly motivated by the fact that while a discussion on security competitions is going, it makes sense to talk about some related things I have been thinking about.

First off, as many of you probably know, the Plaid Parliament of Pwning has not played in purely defensive exercises. To those that may say “well aren’t you going to be biased towards capture the flag events?”, I answer “almost certainly”. I’m not going to say CCDC is worthless and CTFs are the epitome of security education, but my claim is that CTFs provide a much better platform and environment for players learning about a broad range of security topics. Further, I’m going to use the term CCDC when I suppose in general I mean “defensive security competition”. This probably means I will muddle some specific details.

Problem Solving

Fundamentally, CTFs are about problem solving. Although I know defensive competitions have many problems that their competitors have to deal with, I believe CTFs intrinsically allow for more “out of the box” thinking. Let’s start with rules. Most CTF competitions are pretty loose in this respect: maybe team size is limited, challengers aren’t allowed to share answers, and attacking game infrastructure is forbidden, but that’s about it. The nature of defensive competitions means they can’t be so simple. Although the rules for CCDC aren’t too bad, the rule book for the CyberPatriot competition is over one hundred pages long. There’s a reason for this: there are “easy solutions” that would break these competitions. What if you block malicious network blocks? set up a honey pot? rewrite services to be bug free? delete user accounts only the red team is using? proxy all traffic through an IPS? upgrade Windows XP boxes to Windows 7? run separate services in sandboxes?

Yes, not all of these options are feasible in the real world in every scenario: they may be expensive, or constrained by some other resources. But what is the lesson here? If you are put in a real world situation, do all these solutions become blocked just like in a competition? Of course not! You need to evaluate the merits and consequences of each one and decide for yourself. In most defense competitions, these decisions are made for you, and that limits your abilities to think out of the box. In many cases teams that succeed at CCDC do so because they manage to get as out of the box as possible. This ends up meaning they “game the game” rather than make good security practices. For example, maybe you aren’t allowed to delete users on a system, because in the real world that would restrict the abilities of legitimate users to access the system. Fine, change all their passwords to a 100 character hex string. The real world value of this isn’t any better, but it is more acceptable to do in CCDC.

In a Capture the Flag contest, just about anything goes. Over the years I’ve seen crazy hacks to get things working and working quickly: dumping memory by patching in a jump to MessageBox, reverse image searching to find plausible ECC bits for broken QR codes, forcing a core dump and using a file carver to extract files from obfuscated programs… The type of thinking and problem solving promoted by CTFs is fundamental to what it means to be a security researcher.

Feedback

When reading through some comments about the CCDC post, I came across someone complaining about the lack of transparency in CCDC. Although this may not be a universally held opinion, it doesn’t apply to Capture the Flag events. Let’s look closely at the procedure for earning points in both:

==========

Scoring system in CTF

  1. Solve problem
  2. Submit answer to score server
  3. Observe scoreboard adding points (or not)

==========

Scoring system in CCDC

  1. Maintain service uptime and fend off the Red Team
  2. Complete “injects” (business tasks)
  3. Write reports documenting injects and attacks
  4. Submit reports to organizers to get points,reduce deductions
  5. Wait until end of competition to see final standings

==========

Okay, so why is this important? First, scoring in a CTF is objective — there are no reports to be read and graded, and instant feedback to verify the system works correctly. What this means is CTF problem solving can be an iterative process: attempt a solution in a particular way, see the results, and modify your behavior accordingly. If you read about the importance of feedback in learning, you’ll notice a few things CTFs do right: feedback is immediate, feedback is individual (both on a problem and team basis), and feedback isn’t final (if you get a flag wrong, try, try again).

Let’s contrast this with a defense competition. Feedback is not immediate at all: sometimes a team standing is updated daily, and sometimes not until the end of the competition! Feedback is not granular: your team might get a final score at the end of the competition, but it’s not clear where your points came from. Even if points are broken down by advisories, SLA, and so forth, it’s not clear which actions are being re-enforced. Finally, feedback isn’t iterative. Although you may be able to fill out a form to request a regrade on a report, you can’t just submit a bunch of reports and use the best one.

One might argue: the real world doesn’t provide immediate feedback, isn’t objective, and isn’t always fair, so defense competitions are a better approximation of real scenarios! There is a lot of truth to this: CTFs aren’t depicting day-to-day scenarios of system administrators or work in a security operations center. But you know what? That’s kind of the whole point, and what makes CTFs fun.

Fun

I’ll admit outright, “fun” is clearly not universal. I know people out there thinks blocking exploits is far more fun than throwing them, but let’s be honest, this isn’t a popular opinion. Heck, even the NCCDC website talks about how much cooler the Red Team is than the contestants. Again, “well the real world isn’t always fun”. Fine. But if your goal is to encourage people to learn and try new things, is the best way to do it really to throw them into the middle of a hostile network, or would it be better to give them fun tasks that guide them through the learning process?

Jeopardy style CTFs are a lot of fun. They’re relaxing, you sit down and solve problems. Does this emulate high stress-real world scenarios in hostile networks? No, but that’s not a fun learning environment. DefconCTF is not a place for beginners. I am not sure how someone could argue that a teaching method that is less fun could ever be superior.

Of course, not all people like breaking things. That’s fine. Capture the Flags aren’t always about binary exploitation. There are plenty of forensics, programming, cryptography, and other skills that involve building things. The only skills that CCDC has which I have not seen in a CTF are writing reports and running updates. Looking for backdoors and malicious software, finding intrusions in log files, changing default settings and modifying web applications are all things that have shown up in CTFs (though I admit those aren’t the kind of tasks that I enjoy).

The “1%”

One of the points that struck me on the blog post regarding CCDC was the idea of appealing to the “1%”. Restated simply, the claim is that CCDC makes use of skills that 99% of security workers will use while CTFs really only teach 1% of relevant skills. The numbers aren’t really important here, but the sentiment is. Skills like writing reports, reviewing log files, and running updates are the sort of skills most “security workers” will use. Blech. What is that really saying? To get a job as a security worker I don’t need to understand ASLR bypass techniques, NTFS alternate data streams, or CBC Padding Oracle attacks? Yes, that is probably true! That writing a report is more valuable than understanding modern offensive security? Perhaps! But personally, I think that if CCDC claimed the reason it is a superior platform to CTFs is because it involves writing more reports and running Windows update, people would start losing interest. Why not just participate in a technical writing contest? Do we really need a contest which stresses the most mundane aspects of security?

It is probably worth mentioning that some CTFs also require contestants write up solutions. In one instance of RuCTF (2009), players could receive points for patches or writeups of vulnerabilities. Many CTFs with prizes or final rounds also require a writeup be submitted from teams that place. Some competitions also give separate prizes for highest points/first to complete challenges, and best writeup. Still, I think it is fair to say that writing reports is stressed much more in CCDC than CTF events, for better or for worse.

The fact of the matter is that students wanting to learn security are told they should participate in competitions like CCDC. And sure, security is heavily involved in these competitions, but not at the same level as in CTFs. If someone wants to learn about security, I think it’s pretty clear a CTF is a better avenue than a defensive contest. This is important to me because I often come across people who want to get jobs in security and so they play in CCDC. Although I think it’s wonderful for them to take that initiative, I can’t help but feel that their time would be much better spent playing a CTF.

One thing I have heard oft repeated which I whole-heartedly agree with is that in defense, you are always playing catch up. Unless you think as an attacker, you can only react to known vulnerabilities, and so you are always a step behind. The CCDC approach seems to be to focus on defense, and to succeed you will probably learn something about attacks; the CTF approach is that you should focus on attacks, as the defense comes naturally afterwards. I would posit that the top CTF players would be better suited to defending a network against advanced attackers than the top CCDC players for precisely this reason.

Aside from all this, I think everyone would be much safer if security professionals knew more about the 1% that they so rarely see. Maybe if the person reading the IDS logs knew more about writing polymorphic shellcode than what is in Metasploit, they’d be better prepared to prevent threats more advanced than someone using Shikata ga nai.

Conclusion

Both CCDC and CTFs address different goals. As far as I can tell, CCDC’s goal is to assess how well someone can administrate and document systems, while CTFs focus on teaching security. Both of these have their place. But telling fledgling security professionals the better way to learn is through CCDC seems dishonest, and forces them into a niche they might not enjoy. Although the skills taught in CTFs are not always used day-to-day, they cover a far broader range of security topics that are relevant to anyone working in the security field. As someone who tries to promote the value of CTFs, it’s important to me that those interested in security education recognize that CTFs are an incredibly powerful and should be encouraged where possible.

-tylerni7