Can be shown: not possible to be precise in most general case
Can detect all such programs if willing to accept false positives
Can constrain case enough to locate specific malicious logic
Some defenses
Type checking (data vs. instructions)
Limiting rights (sandboxing)
Limiting sharing
Preventing or detecting changes to files
Prevent code from acting beyond specification (proof carrying code)
Static signature checking
Behavioral analysis
Check statistical characteristics of programs
Vulnerability models
PA model
RISOS
NRL
Aslam
Example Flaws
fingerd buffer overflow
xterm race condition
RISOS
Goal: Aid managers, others in understanding security issues in OSes, and work required to make them more secure
Incomplete parameter validation—failing to check that a parameter used as an array index is in the range of the array;
Inconsistent parameter validation—if a routine allowing shared access to files accepts blanks in a file name, but no other file manipulation routine (such as a routine to revoke shared access) will accept them;
Implicit sharing of privileged/confidential data—sending information by modulating the load average of the system;
Asynchronous validation/Inadequate serialization—checking a file for access permission and opening it non-atomically, thereby allowing another process to change the binding of the name to the data between the check and the open;
Inadequate identification/authentication/authorization—running a system program identified only by name, and having a different program with the same name executed;
Violable prohibition/limit—being able to manipulate data outside one’s protection domain; and
Exploitable logic error—preventing a program from opening a critical file, causing the program to execute an error routine that gives the user unauthorized rights.