Reducing the Risk of Solaris Buffer Overflow Attacks

Roamer - 12/20/00

Buffer Overflow attacks are when arbitrary code is executed on the stack. This code is usually executed with the rights and privileges of root user. This paper deals with UNIX buffer overflow attacks. Jeff Thompson of Argus claims that "a buffer overflow attack places new programmatic code on a process' stack and causes the process to execute this code when returning from a function. This code is executed with the same rights as the running process, allowing the attacker to force the program to perform nefarious attacks." The intruder attacks setuid 0 programs and any daemons running as root.

One of the most common ways an attacker can gain unauthorized access is by executing a remote buffer overflow attack. There are many scripts available which will allow a relatively inexperienced attacker to accomplish this. Buffer overflow problems can almost alwways be attributed to poor coding on a particular application.

Luckily for security minded system administrators, most buffer overflow attacks can be prevented. I say most, because a knowledgeable attacker can find a way to get around buffer overflow countermeasures. With Solaris 2.6 and higher, the /etc/system file can be modified to disable stack execution. By default, stack execution is enabled and the system administrator must manually disable it. To do this, add the follwoing two lines to /etc/system:

set noexec_user_stack=1
set noexec_user_stack_log=1

If you are making this modification on a production system, I highly recommend that the 2nd line (set noexec_user_stack_log=1) be added by itself for a period of a week or two. This will write an entry to syslog every time that code is executed on the stack. The reason I say this, is that there are some applications out there which require stack execution by design. If your system is running an application that falls into this category, by monitoring the log files, you will determine this and thus avoid inadvertently shutting down a legitimate application that is required for your activities. In conclusion, while it is impossible to completely prevent an attacker from utilizing a buffer overflow attack, by modifying the /etc/system file on you Solaris 2.6 or higher box, you can thwart the attempts of many if not most would be attackers.