Reference:

Johannes Kinder, Stefan Katzenbeisser, Christian Schallhart, and Helmut Veith. Detecting malicious code by model checking. In GI SIG SIDAR Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA'05), volume 3548 of Lecture Notes in Computer Science, pages 174–187. Springer, July 2005.

Abstract:

The ease of compiling malicious code from source code in higher programming languages has increased the volatility of malicious programs: The first appearance of a new worm in the wild is usually followed by modified versions in quick succession. As demonstrated by Christodorescu and Jha, however, classical detection software relies on static patterns, and is easily outsmarted. In this paper, we present a flexible method to detect malicious code patterns in executables by model checking. While model checking was originally developed to verify the correctness of systems against specifications, we argue that it lends itself equally well to the specification of malicious code patterns. To this end, we introduce the specification language CTPL (Computation Tree Predicate Logic) which extends the well-known logic CTL, and describe an efficient model checking algorithm. Our practical experiments demonstrate that we are able to detect a large number of worm variants with a single specification.

Suggested BibTeX entry:

@inproceedings{mcodedimva05,
    author = {Johannes Kinder and Stefan Katzenbeisser and Christian Schallhart and Helmut Veith},
    booktitle = {GI SIG SIDAR Conference on Detection of Intrusions and Malware \& Vulnerability Assessment (DIMVA'05)},
    month = {July},
    pages = {174--187},
    publisher = {Springer},
    series = {Lecture Notes in Computer Science},
    title = {Detecting Malicious Code by Model Checking},
    volume = {3548},
    year = {2005}
}