Dramatically Improving Triage Tools with Graphing
One of the key problems facing open source communities today is the tsunami of security reports. Even worse, it's nearly impossible to separate genuine security reports from slop reports that are both flooding security reporting channels.
Some projects have taken drastic actions, like blocking all reports that look like they were assembled by AI, or creating allowlists for people who can report issues. Some maintainers have silently stopped reading security reports entirely.
All of these run the risk of genuine reports coming from high quality research going unread and vulnerabilities being left open.
Triaging with AI is hard. Doing it responsibly is even harder.
It is very hard to measure the success of AI triage tools, as it requires finding a novel vulnerability, running tools against that novel vulnerability, and then having a senior researcher review the results to determine if the triage tools worked as designed.
There's also a large difference between using tools that are working on a known format, and tools that are supposed to take generic reports of varying quality from many different sources. We see massive variance in the contents of reports and the variable nature of LLMs also makes research less deterministic than many of us are comfortable with.
Working toward more reliability with triage is where everyone is spending their research time.
Triaging without LLMs is hard.
Many projects do not have a public threat model. It can only be inferred from what a researcher can see on the Git. We have to check the security.md, we have to check for CVEs for the same class of bug, we have to check issues and PRs for rejections surrounding similar issues. Sometimes the documentation is separate from the Git, and we have to go to these places to glean some information about what is and isn't a valid bug. It's a lot of work for a researcher outside of the project to find and distill all of this data, and make decisions about whether the bug they've found is valid or not. Senior researchers often do not have enough information to make judgements about this and have to resort to asking the maintainers or security group questions to validate or refute their findings.
LLMs have challenges on top of the human process, namely they have token limits and context windows to deal with, which further limits their ability to find, digest, and infer whether a vulnerability is real or not.
Why Graphs?
Graphs not only create a format for dataflow and processing that is easier to manage for LLMs, which reduces token burn, but it critically also integrates documentation and community data into graphs, which gives the LLM a low-cost way to find the WHY of design choices, and a gives it access to a lot more context around the scope of the project.
This is crucial for triage as our internal data shows patterns of flaws that come up with reports from the public:
The vulnerability simply doesn't exist.
The vulnerability is actually in a dependency.
The vulnerability is outside of the project's threat-model.
The vulnerability is a duplicate of another report.
The vulnerability is a bug, but has no security impact.
The vulnerability requires excessive privileges.
The vulnerability requires misuse.
The vulnerability requires debug options.
The Setup
We have been using Graphify in our testing. A full graph with graphify pulls in all kinds of extra context into the graph including documentation, config files, PDFs, images, etc. This should allow our LLM based tooling to take in all of this extra context and make sounder judgement calls on whether a vulnerability report is valid or not.
We graphed every project with Opus5 before doing triage with advanced unreleased models using an experimental skill (to be open sourced soon.)
The Process
We created a single-prompt skill for efficacy testing, and will break this down into stages for context/token burn savings in the near future as we refine the skill for public release.
This skill does the following checks as seven gates:
GATE1 - Threat Model - Check all documentation in the graph for mentions of threat model, scope, trusted input, or exclusions. Check if the entry point and sink are plausible targets that are not excluded.
GATE2 - Dependencies - Ensure that this bug report does not exist in a dependency.
GATE3 - Duplicates - Check that the bug report is not a publicly-known duplicate using CVE records and recent changelogs.
GATE4 - Privileges - Check if the bug requires excessive privileges to execute.
GATE5 - Reachability - Check if the bug is reachable by an attacker. Trace the entry point to the sink.
GATE6 - Execution - If a POC/POV is provided, after making extensive checks to ensure that the POC is local execution only, spin up a minimal VM environment to test it.
If any gate 1-6 fails, the report fails to meet the criteria for a valid bug report. If the report contains accuracy errors that can be corrected, report that to the terminal so that the user can decide on actions. (correct and rerun, correct and pass, fail)
GATE7 - Re-Rate - Generate an updated CVSS 3.1 score based on this analysis.
Early Results
Our tooling that includes full graphs with Graphify generated by CPU and using Fable 5.1 where LLM analysis of the docs is required, more than halved the rate of false positives that slip through our initial tooling. This is because the "WHY" of how things are designed is more apparent, even when an overt threat model doesn't exist. Many exclusions can be implied from community data just by reading closed issues and rejected merge requests, further, you can infer what classes of bugs a project accepts by looking at past accepted vulnerabilities.
Including graphs alongside source reduces headaches for researchers and maintainers alike.
Refining
We are now looking into aggregate statistics on whether Sonnet 5 or local models can handle the graphing portion of the work without adversely affecting performance. This will reduce the token burn on the graphing stage which is already low.
Additionally, we are building early-outs in the tooling so that it will stop the process if ANY of the gates in the sequence fail, reducing token burn and average processing time.
And we are working on optimizing the text of the reports themselves before an LLM processes them. Removing all of the extraneous prose from public reports as a first stage dramatically reduces token burn and can be done with lower-end models.