This is the eleventh assignment from my Masters Advanced Network Security Course which has never been published anywhere and I, as the author and copyright holder, license this assignment customized CC-BY-SA where anyone can share, copy, republish, and sell on condition to state my name as the author and notify that the original and open version available here.
A buffer overflow contains the word buffer which is a temporary data storage area and overflow where too much data is inserted it overflows. Buffer overflow can be define as an event where the data size that was input exceeds the size of the buffer and the last part of the data overwrite the other section of the memory [1]. Think of it as a tea spills out of the cup if we put too much tea in it, and same when the data is too much that it will spill out.
Different from a tea being spilled buffer overflow is a data that may overwrites other parts of the memory and maybe read and executed by the machine. The spilled code could change, damage, add, or delete the user's data, even worst may contain an program to execute a remote connection to a malicious person. In July 2000 a buffer overflow vulnerability was found in Microsoft Outlook. No need for a virus attachment, just receiving an email with an exceeded header size added with a payload already open a session for the attacker. These makes buffer overflow as a famous security attack. This attack began when C supplied the framework and poor programming was practice [1].
On Figure 1 shows memory layout of Linux process. A process is a program in execution, and an executable program contains a set of binary instructions to be executed on the processor. They could be read only data like printf, global or static data that last through out the process, brk pointer that keeps track of malloced memory, or local function that cleans up after it is run [2].
Figure 1. Linux Memory Layout
A process image on Figure 1 shows that it started with the program's code and data on the first 2 blocks after the unused block. Following is the runtime heap created at runtime by malloc, then goes to memory mapped region of shared libraries. On the top is user's stack whenever a function call is made it is used. On Figure 2 shows an illustration of stack region, whenever a function call is made the stack pointer pushes the parameters from right to left, then the return address, and the frame pointer between local and return address [2].
Figure 2. Stack Region Illustration
Back to buffer overflow if the data inserted is larger than allocated buffer, it will overflow to other memory address. A simple C program on Figure 3 allocates A 8 of strings, and B is an unssigned short of 2015 value. If we put too much string in A it will overflow and here B changes value [3]. On Figure 4 is the assembly link of the code, we see that the value $2015 is allocated on 2 out of 16 of the address, and to overflow 2015 value we should at least input 14 characters.
Figure 3. Simple Buffer Overflow Demonstration in C programming
Figure 4. Assembly Link of Program on Figure 3
On Figure 5 is another demonstration. It's a simple login program, even though a wrong password is given, but root privilege was also given [4]. Figure 6 is one of the famous vulnerabilities ms08-67 even among beginners. It's char() buffer overflow vulnerability on unpatched Windows XP where the attacker can open a remote session, in other words hijacked the system. The real codes for starters is quite complicated but metasploit provides a ready to use application and all we need to do is specify the target and exploit [5].
Figure 5. Simple Login Buffer Overflow Demonstration in C programming
Figure 6. Hijack Windows XP Using Metasploit
From [2] we got some countermeasures:
Buffer overflow overwrites other part of the memory. The simplest case is variable B if put too much data can flood to the next variable A. A buffer overflow on login program may grant access to the attacker, worst case it can grant remote session as of the ms08-67 vulnerability. To cope with buffer overflow is by well written code for example avoid dangerous functions as gets().
