Running a capture the flag event (for testers)

After a few workshops and presentations by security experts and fellow testers I have realised that security is everybody's responsibility. Testers can't just hope for developer expertise and wait for the yearly penetration test to tell us the state of our application security.

We are knowledgeable enough to at least mitigate the most common security risks through refinement and testing while we develop. To do this we have a lot of resources at our disposal, first and foremost those offered by the Open Web Application Security Project (OWASP).

OWASP has several projects of interest to testers. First and most well-known is the OWASP Top Ten, listing the ten most critical security risks. This document also includes a section on “What’s Next for Security Testers”, which is also interesting reading for non-security testers. It offers links to further reading, advice on test strategies, and ways to get started. One of the documents referenced is the OWASP Testing Guide which describes vulnerabilities and how to test for them. Besides its use in manual tests, the document can also be used to create automated security tests.

To make my fellow testers aware of these resources, and to lower the barrier to “scary” security testing, I organised a small capture the flag (CTF) event for my colleagues at Centric Test & Security Professionals. My aim was to have them find some common security holes so they could see how, with some simple tests – which in some cases can quite easily be automated – major security risks can be found, even by non-security testers like us.

To do this I used the OWASP Juice Shop application, developed to be “the most bug-free vulnerable application” by Björn Kimminich. This application, which can be run locally (using Docker, Node.JS, or Vagrant) or online for free, makes learning about security fun.

Because I used (and tweaked) some online guides on how to run your own CTF event, most notably this post by Josh Grossman and the book Pwning OWASP Juice Shop by Björn Kimminich, I would like to offer a run down on how I set up the CTF, what I used, what I learned, and tips if you want to do it yourself.

The set up
For the CTF event I prepared four components for the evening itself. I made a presentation of about 10 minutes, just going over what security testing is, the types we run into in our daily work, the laws (in the Netherlands) that mandate us to do something about security, and where to find resources to help us testers do this.

Documentation and support
I prepared a site with documentation about the evening, the presentation, and the guide to the Juice Shop. I used Hugo, a static site generator, to turn my markdown files into a site that the participants could use at their own pace during the CTF event. On this site I also placed much of the content of the Pwning OWASP Juice Shop book. I selected some of the introductory chapters, some of the instructions (except the running instructions), and all the challenges. I left out the solutions, but included all the hints, because security testing was completely new to my audience and I felt they needed some help to get going.

1. Screenshot of the Hugo site supporting the workshop. Image by orly.coloncapitalp.com (now defunct).

I did look into converting my site from Hugo to Gitbook so I could use a fork of Kimminich’s book repository to make my own, but I couldn’t get the legacy version of Gitbook (the one that still can be run locally) working on my machine, so I decided to stick with the (to me) more familiar Hugo site generator instead. Because I used Hugo I had to redo some of the internal links in the pages I got from Kimminich’s book, but with a quick find-and-replace action this was not a lot of work.

My Hugo site is stored in a repo on Gitlab (chosen over GitHub because, at the time, Gitlab had free private repos and GitHub did not). I connected this repo to Netlify, enabling automated publishing and free hosting of the site. Instructions on how to do this can be found on Netlify’s documentation site. I have found Netlify to be a great resource for workshop instruction sites like these (a form I always use for interactive workshops) because students can read the material at their own pace, and it remains available after a workshop ends.

2. Screenshot of the Netlify settings to build and deploy my Hugo site hosted on GitLab.

Making the Juice Shop available
I previously hosted a workshop similar to this one where I asked all participants to install Docker or Node.JS on their machines so they could run the Juice Shop locally. However, as expected, during the workshop we still spent a lot of time getting the Juice Shop running for everyone, and because the laptops varied in capacity, some participants couldn’t even run the Juice Shop because of storage limitations.

I had read about the Heroku option in Kimminich’s book, but was hesitant to use it. Sure, it said it was free, but I needed a dozen instances, and I was certain it wouldn’t be free for me. However, Grossman’s post about his CTF event convinced me to try it. It couldn’t have been easier (and it was indeed free).

To run the Juice Shop on Heroku you need to create an account, after which you can click the “Deploy to Heroku” button to deploy an instance of the Juice Shop to a Heroku dyno. As long as you promise not to execute any DDoS attacks, running and hacking the instance is free. I did find that to deploy more than five instances, Heroku wants credit card details, but after submitting those I was never charged, even when running (and using simultaneously) twelve instances.

3. Screenshot of the deploy process of the Juice Shop app on Heroku.

After deploying your Heroku app you need to go to the settings of the app and set the NODE_ENV configuration variable to ‘ctf’, and then restart your dyno. This enables the CTF mode of Juice Shop with features like unique challenge codes to register challenges solved, hiding the link back to the Juice Shop GitHub repo, and removing the hints from the score board.

4. Screenshot of the settings to adjust in Heroku to enable CTF mode in the Juice Shop app.

When the dyno is not used for a while it stops running. So, because I knew it would take a while for the dyno and app to start up again when I was ready to use the instance, just before the event I visited all my instances just to get that start up time out of the way.

The CTF platform
A CTF platform like CTFd enables challenge tracking and score keeping during the event. I found the CTF platform, how it works, and how it is used the most difficult part of setting up the event. It took me a while to get my head around the fact that the CTF platform is “just” another site available to all participants where they log in and register the challenges they find. The CTF platform has no link to the Juice Shop, and the Juice Shop has no link to the CTF platform. The Juice Shop offers a tool to create a “backup” of the CTF platform containing all challenge information needed, but this backup is also not linked to a specific instance of the Juice Shop.

From the documentation available, and from Grossman’s post, I got the idea that CTFd is a pretty ok platform to use, so I picked that to install. I don’t have my own publicly available server, so I tried to get a company server to host CTFd. However, the only one available was running an old version of Windows Server and CTFd needs Linux to run. Luckily Grossman ran CTFd on Amazon AWS, so I figured I’d give that a try.

It was my first time using AWS, so it took a couple of tries to figure everything out. I created an account, added payment details, and got started. Because I was a bit uncertain about the cost aspect, I tried to keep everything as small and light as possible. I chose a EC2 t2.micro instance running Ubuntu 18.04. I also set up SSH to this instance using the instructions found here, and using the instructions found on the Docker site and on Grossman’s site, I installed Docker and Docker Compose. I then cloned the CTFd repo on the instance. I updated the docker-compose.yml file by changing the ports section to 80:8000, mapping the docker container’s internal port 8000 to the host port 80. This means that the CTFd application will be reachable using just the IP of the instance (because later I will use my domain’s DNS settings to point a subdomain of my own to that IP).

5. Screenshot of updated CTFd docker-compose.yml to set ports to 80:8000.

After changing the docker-compose.yml I used Docker Compose to get CTFd running.

On my own computer (so not the Heroku dyno nor the AWS instance) I used the Juice Shop command line tool to create a “backup” of the CTFd settings needed to use CTFd for a CTF event using Juice Shop. I made all the hints free (because I was giving them away anyway in my support site).

Uploading the backup to my CTFd instance was a bit confusing. You have to make an admin account, upload the file, restart, and then create another admin account (because your first was overwritten by the backup file I guess). After doing all that I had the Juice Shop challenges loaded in my CTFd instance. I then customised the front page with our own logo and a link to the support site.

After everything was set up the way I wanted I stopped the instance on AWS. Because I was still uncertain of how much usage/uptime is free and when the meter starts running, I decided to only turn the instance on when I actually needed it (in the end my usage was below the limits so I ended up paying nothing). Turning the instance off and on again does change your IP, but I found that updating the IP in my DNS manager propagated quite quickly, so it was fine for my purposes. I did not lose any settings or customisation by turning the instance off, something I was a bit apprehensive about at first.

6. Screenshot of the CTFd scoreboard (after the event) and not on my subdomain.

I use Netlify to manage the DNS settings for my personal site, and I set up a subdomain pointing to the AWS IP. When I turned on my AWS instance again just before the event, I updated the IP in the DNS settings and found that it worked nearly instantaneously. I chose not to use TLS, both because setting up everything else had cost me a lot of time and effort already and because this sounded pretty difficult too. I realise that by omitting TLS, despite it being relatively simple due to the service of Let’s Encrypt and the EFF, I am doing exactly what OWASP is trying to prevent, and not making my app as secure as it could be. Mea culpa: next time I will surely try to figure out how to also set up TLS myself.

Organising the event
Before the event started we gathered a list of all participants. We created teams by pairing up testers based on technical ability: a tester with more technical knowledge was paired with one who had less experience with code, automation and security. The hope was that the first tester could share knowledge and experience with the latter. Inspired by the Juice Shop theme I created twelve Juice Shop instances, each with their own fruit-based name. I also created twelve similarly named accounts in CTFd. I made place cards with the team names on the front, and the relevant links to the Juice Shop and support site as well as the login details to their CTFd account on the back. Once we were certain who would attend, I also wrote the team member names on the back of the cards. I had created two extra teams as a backup in case an instance got corrupted or was otherwise not available.

Each participant brought their own laptop so it was just a matter of guiding them to the right seats and team partner and getting started. I had connected my own laptop to the beamer so I could project the scoreboard (automatically updated every 10 seconds by using the Chrome extension Super Auto Refresh Plus), one sheet with the relevant links, and a timer counting down the remaining time. I had another laptop with me for troubleshooting.

I set some ground rules before we started:
• No googling the answers or any Juice Shop related info
• No DDoS or extensive tool usage to attack the Juice Shop instance (because my credit card is linked, so be nice)
• No attacking CTFd or the other teams’ Juice Shops
• Start time is 18:30; end time is 20:30. Challenges can only be registered between these times (this is a setting in CTFd you can use)
• Work together, read the support docs well (especially the chapters “Walking the happy path” and “Hacking exercise rules”)
• Asking me questions is allowed.

I walked them through the first challenge of finding the score board, and then set them loose.

7. Capture the Flag event with colleagues from Centric Test & Security Professionals at Centric Gouda (Photo by Leo van Hamond).

Experience and lessons learned
When organising an event like this, especially one outside your comfort zone – like hacking is for us testers – the question is always, how well will it be received? I am happy to report that the participants all took it very seriously (you could hear a pin drop during the two hours of the challenge) and did their very best. Of course, some participants took to it better than others, but all reported that they found it a very enjoyable way to get to know security testing and ethical hacking a bit better.

I noticed several minor things during the event that I might do differently were I to do it again:
• The support site I set up was missing some internal links (notably those from the challenge overview to the challenge descriptions). This was because of the difference between Hugo and Gitbook and how they format those links. Luckily the fix was done pretty quickly thanks to my setup of GitLab and Netlify, but I should have tested these links better.
• Late arrivals found it hard to get started, so I needed to get them up to speed before they could begin. The next time I would make a dedicated page on the support site with the basic getting-started information.
• I had registered the teams in CTFd using just their fruity team name, which was also the name that was shown on the scoreboard. This wasn’t much fun as you couldn’t see who was beating you. During the challenge I changed the team names to include the members’ first names. Luckily CTFd didn’t kick out those who’d already logged in with the old team name.
• Some teams had a hard time getting started. Either they did not understand how to use CTFd, or they were solving challenges but not registering the codes, or they were unable to solve the challenges altogether. Each team should be able to register an easy 100 points from the demoed “Find the Score Board” challenge, so any team that failed to do that I helped. If a team did not have more points after an hour I started helping them with hints.
• The participants took the “No googling the Juice Shop” rule to mean “Don’t google at all”. This held them back because some of the terminology was totally new to them. I had to clarify this rule.

All in all, this was a very fun event to organise and run. I learned a lot, too, while organising the event, mainly about cloud services like Heroku and AWS that I had never used before. I have to admit, the pricing structure of these providers scares me (I like a fixed price), but I was pleasantly surprised to find that running this event with 18 participants cost me nothing in the end. I’m also happy that my colleagues took the event seriously and had fun that evening just trying a bit of hacking. I know that raising security awareness and enabling “ordinary” testers to do their own security tests will entail a lot more than just one CTF event, but at least we got started. In the future, I might organise more knowledge sharing based on the OWASP Top 10 to help testers with security testing.

Deze blog verscheen eerder op www.centric.eu/craft.