Analysis of the Control Flow Graph to Detect Attacks at the Application by Verification its System Calls
Modern applications are often subjected to various attacks. The problem of identifying buffer overflow attacks on applications has been and remains one of the urgent tasks of information security. The aim of this work is to develop an algorithm for detecting attacks on an application by checking its system calls. Seeing this, existing approaches to detecting a buffer overflow attack are analyzed. Given their shortcomings, an algorithm is proposed for detecting attacks on the application by checking the system calls it makes. Such an application protection method involves detecting attacks that lead to code injections, that is, to changes in the software control flow. To build this algorithm, the control flow graph and the process of making a system call were analyzed. Based on the obtained data, an algorithm for constructing the behavior model of the software with vulnurability is proposed, in which, when passing through each of the control graph vertices, the system call made by the function is extracted and recorded in a special structure that will be used to protect the analyzed application. This approach allows you to protect a system with potentially vulnerable software from code injection attacks.