gpt3_security_vulnerability_scanner - GPT-3 found hundreds of security vulnerabilities i

项目说明

Experimenting with GPT-3 for Detecting Security in Code (this was the first real LLM cybersecurity eval!)

Summary: GPT-3 found 213 security in this git repository. In comparison, one of the better commercial tools on the market (from a reputable cybersecurity company) only found 99 issues, although their tool provides context in a more structured format. After manually re a random sample of 60 / 213 of the detected by GPT-3, only 4 were false positives. Both tools had many false negatives.
The full text of this README is a as a Medium article here.

Introduction

In recent years, the field of artificial intelligence and machine learning has seen tremendous growth and has opened up a new realm of possibilities. One such field that has been gaining attention is AI-based code analysis, specifically the use of AI models to detect security in code. In this experiment, we used OpenAI's GPT-3 to find security in a code repository containing 129 files.

How it Works

The of GPT-3 I used (text-da) has a context window of 4000 tokens, which is roughly 3000 english words. This means it can process at most a few hundred lines of code per request. Unfortunately, GPT-3’s current architecture can’t handle a whole repo at once.

To get around this, I had to scan all of the files with GPT-3 separately. This means GPT-3 might ha finding security that are the result of multiple files of code interacting, unless the import/exports are clear enough to make a guess as to what those functions do without needing to specifically see the code. This ended up often being the case, particularly when the source code was using common libraries like express.js, Flask, the Python standard library, the C standard library, etc. It’s likely that GPT-3 has many of the most common libraries either partially memorized, fully memorized, or encoded in some other way. In the case of the code analyzed in this article, GPT-3 had enough prior knowledge about the imported libraries that it was able to accurately detect security without needing to inspect any of the imported library code.

To be fair to GPT-3, I suspect that many of the existing commercial scanners don’t actually inspect imported library code when doing static analysis — so this is not that different from how some of the tools on the market already work.

The code that was analyzed

Each folder in the repository is named after a type of security and contains files with example code containing one or more . Some of these files contain trivial code, but many are fairly realistic code snippets you might come across in a production code base (note: they are still snippets though, and therefore lack the context of a larger codebase). The README.md file in each folder of the repository contains GPT-3’s analysis of the security for all of the files in that folder.

Let’s take a look at some examples to see how GPT-3 did!

Example 1 (trivial)

Here is a simple C program that is to a Format String Attack:

And here is GPT-3's analyis (which is correct):

Example 2 (less trivial)

Let's try this with a less trivial program in a higher le, like this C# program containing a Log Forging . I' seen similar (although less trivial) code in open-source projects and production code bases:

GPT-3's output:

GPT-3's e is correct - there is a log injection in this code.

Example 3 (non-trivial)

The following C program reads and manipulates an image. It contains numerous security , including Out Of Bounds reads and writes:

GPT-3's output:

GPT-3 is correct for 4 out of 5 of these. 1 is a false positive as the value of fopen() is checked in the line if(fp == NULL). Also, 4 is worded in a way that is misleading - it is correct that there are two uninitialized memory access bugs on the lines of code that it is referencing, but it mentions the wrong names.

Example 4

This last Ja is my fa because, unfortunately, I didn't see anything wrong with this code when I first read it. To me, it looked completely innocuous and *familiar* - I've seen similar code in the wild but never had any second thoughts.

GPT-3's output:

GPT-3 is correct again.

The Ultimate "Shift-left" in Cybersecurity

Shifting-left is the process of applying security practices earlier in the de process - like testing for security bugs before deploying your code to production.

AI code-analysis could represent the ultimate shift-left in cybersecurity by applying best security practices and educating de on cybersecurity as they are *writing and committing code*. This could truly foster a culture of strong cybersecurity inside an organization and e the next big hack, but...

Admittedly, the technology is not perfect yet and there are several things it missed that an experienced cybersecurity professional doing an audit would not ha. Still, it's shockingly good for being a general-purpose large language model and it's far cheaper than the alternatives.

GPT-4 doesn't currently have a release date, but I'm sure these large language models will continue to march forward as they gain more and more capabilities.

The False Positive

After manually re around 60 out of 213 of the detected by GPT-3, one of the false positives I found that was notable was on line 13 of this Ja program:

GPT-3's output:

GPT-3 is correct about the first two , but the third is a false positive - obj.username is escaped, but GPT-3 is saying that it is not.

Results

The results of the experiment showed that GPT-3 was able to detect security in 85 out of 129 files scanned. This is quite impressive!

The script summarize_results.py generates a summary of GPT-3's results:

Comparison to Commercial Offerings

To round out this experiment, I compared the results of GPT-3 with a commercially a code scanner, Snyk Code, which is made by Snyk - a company which I think makes excellent security products. After running this repo through Snyk Code, it found 99 security compared to the 213 found by GPT-3.

!Snyk's results

One contributing factor is that Snyk Code only supports certain programming languages, and was only able to scan around 103 files compared to the 129 files scanned by GPT-3.

Final Notes

If you're interested in seeing this experiment become a full product, express interest through this super short Google Form.

The code snippets in this repo were taken from snoopysecurity/, which is a fantastic resource. I tried to remove any comments embedded in the code snippets that hinted at what security were contained in that snippet. This required me to remo containing links to blog posts and articles that these example snippets were gathered from. Any attributions present in the original repo can be found in the attributions.md file.

项目信息

开发语言：PHP

Star 数：604

技术要点

该项目属于安全软件方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕GPT-3 found hundreds of security vulnerabilities in this repo - (this was the first real LLM cybersecurity eval!)展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。