___......__             _
                                          _.-'           ~-_       _.=o---_
                  --=====-.-.-_----------~   .--.       _   -.__.-~ ( __====>
                                '''--...__  (    \ \\\ { )       _.-~
                                          =_ ~_  \\-~~~//~~~~-=-~
   _____________  ___  ____ ___  ___       |-=-~_ \\   \\
  / __/ ___/ __/ / _ \/ __// _ \/ _ \      |_/   =. )   ~}
 / _// /__/ _/   \_, / _ \/ // /\_, /      |}      ||
/___/\___/___/  /___/\___/\___//___/      //       ||
                                        _//        ||
                                     '='~'          \\_   
______ _______ _______ _____  ______ __   _ _______ _______ __   _ ______   /|
|_____| |______ |______   |   |  ____ | \  | |  |  | |______ | \  |   |    /_|_ 
|     | ______| ______| __|__ |_____| |  \_| |  |  | |______ |  \_|   |      |  

Assignment 4


In this assignment you will complete a Capture-the-flag (CTF) style challenge. You will be presented with a Linux virtual machine and will execute a stack buffer overflow to gain access to the system’s /etc/shadow file which can be used to recover the root user’s password.

How the assignment is marked

In order to get full marks on this assignment it is not sufficient merely to capture the flag. Similarly it is not acceptable to copy/paste a solution found on the web. You can use existing solutions to help you build your understanding. But in order to get full marks, you will need to provide some evidence that you actually attempted to think and work through the problem. Give us a window into your thinking. Convince us you understand these ideas by telling us about your journey.

What to do

Some examples of how you can convince us you thought through the problem could include sharing details of:

  • Your thought process (e.g., “I noticed something unusual in the code, so I…”)
  • Things you didn’t know (e.g., “I had to look up how netcat works”)
  • Things you tried that didn’t work (e.g., “The documentation mentioned the ‘-x’ flag, but it kept giving an error, so I…”)
  • The lead up to the moment where things finally made sense (e.g., “…then I realized, no, it had to be … so I changed it and then it worked!”)
  • Include all commands you used to capture the flag

What NOT to do

  • Say “I couldn’t figure it out” and not write anything else
  • Submit the writeup of another person, whether another student, or someone online (duh)
  • Use text and images you didn’t write/create yourself (unless you properly quote and cite it)
  • Only give the flag and no other window into your thought process
  • Only say “I watched the video” or “I followed the tutorial” and not elaborate further.

The Flag

There is only one flag in this assignment: Obtain the root user’s password. There are two steps to this challenge:

  1. Execute a stack buffer overflow to run shellcode to gain access to the /etc/shadow file.
  2. Crack the root user’s password hash to obtain the flag (i.e., the root user’s password).

VM Ground Rules

The purpose of the assignment is to simulate a real, remotely accessed computer system and questions are meant to be solved in this spirit. To that end, you will not receive credit for accessing the flags by attacking the virtual machine itself e.g., by recovering the flags from forensics on the .ova file.

Note for Mac Users with M1/M2

As with assignment 3, be sure to download (and unzip) the .utm file and use UTM in emulation mode.


Assignment 4 is based on the Protostar/Stack 5 Capture the Flag challenge. This challenge combines and extends the concepts we’ve been learning, so you may need to prepare with some additional background preparation involving gdb, the stack, assembly code, program execution, and program memory.

  1. Start by watching video 0x0C, which is a walk through solution to the Stack 0 challenge. This will introduce you to the stack and its layout in the context of a CTF about using a buffer overflow to modify a variable.

  2. Next watch video 0x0D to gain additional insight into buffer overflow attacks in the context of the Stack 3 challenge, which demonstrates how a buffer overflow can be used to redirect program execution.

  3. Finally watch video 0x0E, which is a walk through of the Stack 5 challenge. Assignment 4 is based off this CTF, which demonstrates the attacker’s ability to use a stack buffer overflow to inject and execute shellcode. You may find you need to watch this video numerous times!

Phases of the stack buffer overflow exploit

There is an executable file smashme in the user’s home directory. Your job is to run this program and provide it with input to achieve two goals: (1) put a malicious program onto the stack (shellcode), and (2) use a stack buffer overflow to hijack the program’s execution flow, and get it to run this shellcode to gain access to the system’s shadow file.

This exploit involves several steps, which you should seek to understand and master before moving on to the next one. Anyone can get the finished exploit and run it. Your job is to understand something about why it works.

Here are the high-level phases of the buffer overflow exploit:

  1. Determine in gdb how many characters of input are needed before you can take control of the return address
  2. Determine in gdb the appropriate address on the stack to send the program execution to
  3. In gdb, demonstrate trivial code execution in the stack by getting the sigint interrupt to run
  4. Implement a Nop slide to overcome the address difference between running it in gdb, and running in the command line
  5. Add in your shellcode, run from the command line to print the /etc/shadow file.

Password Cracking

Once you have the shadow file, you can use your favorite password cracking utility to recover the root user’s password!

Hint: The root user chose a password in this list of the top 10,000 most common passwords.

You may use whatever shellcode you wish to obtain the shadow file, and the website shell-storm.org provides numerous specimens. We recommend the following shellcode which executes chmod 777 /etc/shadow.

Differences between Assignment 4 and the Stack 5 challenge

As pointed out in the 0x0E video starting at 6:38, if you copy and recompile the program, crucial addresses in the stack may be different due to different environment variables.

For example, if you analyze the program in gdb using virtual box’s terminal window, you will find you get different stack addresses than if you log into the VM over ssh, because the two different sessions will have different environment variables, which will make the stack bottom start at different addresses.

My recommendation, therefore, is you only interact with the VM over ssh, that way you can have multiple sessions open simultaneously, with environment variables of similar length.

Also, when the author implements the Nop slide starting at 8:18, he uses a 30 byte offset, and 100 byte Nop slide. If you keep getting an illegal instruction error, you may want to try increasing these values.

Keep in Mind

Here’s a few things that will be helpful to keep in mind:

  • The stack grows downwards in memory (higher addresses to lower addresses)
  • Strings write upwards in memory (lower addresses to higher addresses)
  • Code execution (i.e., the instruction pointer) moves upwards in memory (lower addresses to higher addresses)
  • gdb typically prints address from low to high, which means visually it’s reversed: the low address appear higher on the screen and the higher address appear lower on the screen. This is opposite of how we draw the memory map in the lectures (with high addresses on top, and low addresses on the bottom). This can be a BIG source of confusion, so focus on the numeric direction of addresses

How to complete the assignment

  1. Download the ECE 9609 Assignment 4 VM.ova virtual machine (or M1/M2 users: ECE 9609 Assignment 4 VM.utm.zip) in OWL -> Resources
  2. Login as user: user with password: user
  3. Execute a stack buffer overflow to gain access the /etc/shadow file and recover the root user’s password. The root user’s password is the flag.
  4. Submit a PDF of your writeup in OWL-> ECE9609-> Assignments-> Assignment 4