Assignment 3 Notes

In this assignment we put together all our knowledge into a semi-realistic Capture-the-flag scenario. You will be given a virtual machine (see OWL), and must capture the flag.

The Flag

There is only one flag in this assignment: obtain the root user’s password. There are two steps to this challenge:

  1. Execute a stack buffer overflow to run shellcode that will print the /etc/shadow file.
  2. Crack the root user’s password hash to obtain the flag (i.e., the root user’s password).


Assignment 3 is based on the Protostar/Stack 5 Capture the Flag challenge. This challenge brings to bear a great many concepts, and therefore you must prepare with some additional background preparation involving gdb, the stack, assembly code, program execution, and program memory.

The website offers several excellent videos walking you through these ideas:

  1. Start by watching video 0x0C, which is a walk through solution to the Stack 0 challenge. This will introduce you to the stack and its layout in the context of a CTF about using a buffer overflow to modify a variable.

  2. Next watch video 0x0D to gain additional insight into buffer overflow attacks in the context of the Stack 3 challenge, which demonstrates how a buffer overflow can be used to redirect program execution.

  3. Finally watch video 0x0E, which is a walk through of the Stack 5 challenge. Assignment 3 is based off this CTF, which demonstrates the attacker’s ability to use a stack buffer overflow to inject and execute shellcode. You may find you need to watch this video numerous times!

Phases of the stack buffer overflow exploit

There is an executable file smashme in the user’s home directory. Your job is to run this program and provide it with input to achieve two goals: (1) put a malicious program onto the stack (shellcode), and (2) use a stack buffer overflow to hijack the program’s execution flow, and get it to run this shellcode to gain access to the system’s shadow file.

This exploit involves several steps, which you should seek to understand and master before moving on to the next one. Anyone can get the finished exploit and run it. Your job is to understand something about why it works.

Here are the high-level phases of the buffer overflow exploit:

  1. Determine in gdb how many characters of input are needed before you can take control of the return address
  2. Determine in gdb the appropriate address on the stack to send the program execution to
  3. In gdb, demonstrate trivial code execution in the stack by getting the sigint interrupt to run
  4. Implement a Nop slide to overcome the address difference between running it in gdb, and running in the command line
  5. Add in your shellcode, run from the command line to print the /etc/shadow file.

Password Cracking

Once you have the shadow file, you can use your favorite password cracking utility to recover the root user’s password!

Hint: The root user chose a password in this list of the top 1000 most common passwords.

You may use whatever shellcode you wish to obtain the shadow file, and the website provides numerous specimens. However, we recommend the following shellcode which executes /bin/cat /etc/shadow.

For convenience you can copy it in hex form here:


Tips and differences from Stack 5

As is pointed out in the 0x0E video starting at 6:38, if you copy and recompile the program, crucial addresses in the stack may be different due to different environment variables.

For example, if you analyze the program in gdb using virtual box’s terminal window, you will find you get different stack addresses than if you log into the VM over ssh, because the two different sessions will have different environment variables, which will make the stack bottom start at different addresses.

My recommendation, therefore, is you only interact with the VM over ssh, that way you can have multiple sessions open simultaneously, with environment variables of similar length.

Also, when the author implements the Nop slide starting at 8:18, he uses a 30 byte offset, and 100 byte Nop slide. If you keep getting an illegal instruction error, you may want to try increasing these values.

How to complete the assignment

  1. Download the Assignment 3 VM in OWL under Resources->ECE9609-Assignment-3.ova
  2. Complete the assignment steps to recover the root user’s password
  3. Give your solutions in OWL under Tests/Quzzes -> Assignment 3