Everything is Broken and I'm Always Tired

Dramatically Improving Triage Tools with Graphing

2026-09-10T00:00:00+00:00

One of the key problems facing open source communities today is the tsunami of security reports. Even worse, it's nearly impossible to separate genuine security reports from slop reports that are both flooding security reporting channels.

Some projects have taken drastic actions, like blocking all reports that look like they were assembled by AI, or creating allowlists for people who can report issues. Some maintainers have silently stopped reading security reports entirely.

All of these run the risk of genuine reports coming from high quality research going unread and vulnerabilities being left open.

Triaging with AI is hard. Doing it responsibly is even harder.

It is very hard to measure the success of AI triage tools, as it requires finding a novel vulnerability, running tools against that novel vulnerability, and then having a senior researcher review the results to determine if the triage tools worked as designed.

There's also a large difference between using tools that are working on a known format, and tools that are supposed to take generic reports of varying quality from many different sources. We see massive variance in the contents of reports and the variable nature of LLMs also makes research less deterministic than many of us are comfortable with.

Working toward more reliability with triage is where everyone is spending their research time.

Triaging without LLMs is hard.

Many projects do not have a public threat model. It can only be inferred from what a researcher can see on the Git. We have to check the security.md, we have to check for CVEs for the same class of bug, we have to check issues and PRs for rejections surrounding similar issues. Sometimes the documentation is separate from the Git, and we have to go to these places to glean some information about what is and isn't a valid bug. It's a lot of work for a researcher outside of the project to find and distill all of this data, and make decisions about whether the bug they've found is valid or not. Senior researchers often do not have enough information to make judgements about this and have to resort to asking the maintainers or security group questions to validate or refute their findings.

LLMs have challenges on top of the human process, namely they have token limits and context windows to deal with, which further limits their ability to find, digest, and infer whether a vulnerability is real or not.

Why Graphs?

Graphs not only create a format for dataflow and processing that is easier to manage for LLMs, which reduces token burn, but it critically also integrates documentation and community data into graphs, which gives the LLM a low-cost way to find the WHY of design choices, and a gives it access to a lot more context around the scope of the project.

This is crucial for triage as our internal data shows patterns of flaws that come up with reports from the public:

The vulnerability simply doesn't exist.

The vulnerability is actually in a dependency.

The vulnerability is outside of the project's threat-model.

The vulnerability is a duplicate of another report.

The vulnerability is a bug, but has no security impact.

The vulnerability requires excessive privileges.

The vulnerability requires misuse.

The vulnerability requires debug options.

The Setup

We have been using Graphify in our testing. A full graph with graphify pulls in all kinds of extra context into the graph including documentation, config files, PDFs, images, etc. This should allow our LLM based tooling to take in all of this extra context and make sounder judgement calls on whether a vulnerability report is valid or not.

We graphed every project with Opus5 before doing triage with advanced unreleased models using an experimental skill (to be open sourced soon.)

The Process

We created a single-prompt skill for efficacy testing, and will break this down into stages for context/token burn savings in the near future as we refine the skill for public release.

This skill does the following checks as seven gates:

GATE1 - Threat Model - Check all documentation in the graph for mentions of threat model, scope, trusted input, or exclusions. Check if the entry point and sink are plausible targets that are not excluded.

GATE2 - Dependencies - Ensure that this bug report does not exist in a dependency.

GATE3 - Duplicates - Check that the bug report is not a publicly-known duplicate using CVE records and recent changelogs.

GATE4 - Privileges - Check if the bug requires excessive privileges to execute.

GATE5 - Reachability - Check if the bug is reachable by an attacker. Trace the entry point to the sink.

GATE6 - Execution - If a POC/POV is provided, after making extensive checks to ensure that the POC is local execution only, spin up a minimal VM environment to test it.

If any gate 1-6 fails, the report fails to meet the criteria for a valid bug report. If the report contains accuracy errors that can be corrected, report that to the terminal so that the user can decide on actions. (correct and rerun, correct and pass, fail)

GATE7 - Re-Rate - Generate an updated CVSS 3.1 score based on this analysis.

Early Results

Our tooling that includes full graphs with Graphify generated by CPU and using Fable 5.1 where LLM analysis of the docs is required, more than halved the rate of false positives that slip through our initial tooling. This is because the "WHY" of how things are designed is more apparent, even when an overt threat model doesn't exist. Many exclusions can be implied from community data just by reading closed issues and rejected merge requests, further, you can infer what classes of bugs a project accepts by looking at past accepted vulnerabilities.

Including graphs alongside source reduces headaches for researchers and maintainers alike.

Refining

We are now looking into aggregate statistics on whether Sonnet 5 or local models can handle the graphing portion of the work without adversely affecting performance. This will reduce the token burn on the graphing stage which is already low.

Additionally, we are building early-outs in the tooling so that it will stop the process if ANY of the gates in the sequence fail, reducing token burn and average processing time.

And we are working on optimizing the text of the reports themselves before an LLM processes them. Removing all of the extraneous prose from public reports as a first stage dramatically reduces token burn and can be done with lower-end models.

Open Source Cybersecurity in 2026 is a Mess. Maintainers Need Help. Now.

2026-09-08T00:00:00+00:00

Let's talk about the open source landscape and how AI is the cause of (and solution to?) all of life's problems.

I'm not going to go into the hype cycle and value of LLMs and how they affect software security. People who maintain projects are well aware of the ongoing slow train-wreck that is happening now. I don't think that is up for debate anymore. Mythos is real. GPT Astra is real. Custom AI tooling is also real (in some cases. There's bad actors selling snake-oil out there, but that's not new to cybersecurity.) Advanced harnesses and prompts can enhance the cyber capabilities of smaller models. Now that we have that statement out of the way...

The real problem isn't that we have new powerful tools to find vulnerabilities in software, it's deeper than that. There are systemic problems in open source development, community building, and sustainability that have led us to where we are today.

The Symptom - Maintainers are getting buried in reports of both low and high quality.

Often just a glorified inbox, maintainers receive security reports from the public, validate that a security problem is a real issue, and write a fix. In early 2025 this was not an issue. Depending on the size and complexity of your project, your time addressing issues was a sizable but not unmanageable part of software development.

Today, we have maintainers receiving security reports at an unprecedented rate. This overwhelming pile is part nonsense and part reality, and it's extremely had to extricate the two. LLM based reports can be coming from a random person with no software engineering experience who used a public tool to find a potential issue and report it with the confidence of a professional with 20 years of experience. Or, conversely, it could be the product of a state of the art LLM-based tooling run, that was hand-validated by senior engineers. The reports look the same until you spend time with them, and it's far far too many reports for maintainers to spend time on each valid report, let alone taking the extra time to eliminate all of the slop.

The Real Problem

The open collaboration on software with open source code is the load-bearing pillar that has created trillions of dollars in value for the world. But the Achilles Heel of open source has always been that maintainers have limited time. From the small solo-maintained project to corporate-run open source programs with entire dedicated teams, there is a limit on the amount of work that can be done building, improving and maintaining a project.

This bandwidth is dynamic and assigned to tasks based on the maintainers preferences and what the project needs to continue to succeed.

Some time is spent on security, but that depends entirely on the maintainer, their level of knowledge about writing secure code, and how much time they want to spend making sure a typo or misunderstanding an API doesn't bring down the whole ecosystem.

The value derived from open source has always been the business case. That's not to say that free software isn't itself valuable. But when we see the giant $8,000,000,000,000 valuations of open source, they are talking about economic activity that otherwise wouldn't be enjoyed if we didn't have it.

This connects directly back to maintainers, communities, and governance. The value generated by these projects drives reinvestment into those projects at abysmal rates, with a few exceptional projects that have been deemed valuable enough for support. This core issue has created an environment where projects are chronically underfunded. They can't afford commercial security tools, they can't afford the cloud hosting for fuzzing. They can't even get easy access to free security tools to review their code. Many projects today don't have security testing in their CI pipelines. Not because they don't want it, but because it is too complex to manage alone. You switch on some tools, and get overwhelmed with the reports and understanding what is a real issue and what is a false positive. (Sound familiar?)

This chronic under-investment in open source security has left the worlds codebases in a state that doesn't quite fall apart, rather than well supported and reinforced foundations that we can (should?) build the worlds economy on top of. The world has tacitly chosen a bridge made of sticks and glue over stainless steel beams and welded rivets.

Security focused LLM tooling has only allowed us to examine things at a new scale. This has happened before with SAST, and again with fuzzing, but this time the scale is orders of magnitude larger, because we can analyze billions of lines of code with electricity and GPUs instead of decades of knowledge and dedicated research time. The rot has always been there, we've just invented a better nose.

The Event Horizon has been Crossed. We Can't Go Back.

I do not say this lightly. There are millions of bugs that need to be found, properly triaged, and fixed before an open source Mythos lands and attackers have the same capabilities as defenders, and we don't know how long we have until that happens. We can't rely on maintainers to do this alone. It's going to take engineers, community building, money, good-faith dialogue with maintainers, strategy, and policy to prevent disaster. We can't wait for someone to save us or go back to the old ways. Things have changed, and we need to adapt, quickly.

How to Help

Companies, Foundations, Governments:

Build programs to get security engineers on this problem as quickly as possible. Don't just assign them to scan code and dump even more problems on open source maintainers. Make sure that you're offering verified security bugs, shaping your reports to sensible, succinct and readable, and real solutions alongside your security reports. (Do not AI-generate fixes without verification by a human expert. It's a bad idea.)

Reach out to your communities. If you don't know who your communities are, check your Software Bill of Materials for your software. If you don't have an SBOM, build one. Everyone on that list needs your help. Find out how bad their backlogs are, and how much help you can provide, and what the maintainers actually need from you.

Invest in the foundations that support the software you use. These foundations directly help keep the lights on, and help allocate security resources to projects and keep them afloat.

Researchers:

Don't spam maintainers with bad reports. They should be succinct, describe the affected function, how it's supposed to work, why there's a bug in that code that causes unexpected behavior, and a POC that shows that the problem is real and exploitable. Check their documentation to ensure that your findings fit into their threat model. If they say that all inputs are trusted and that they don't care about stack overflows or denial of service issues for their library, respect that and do not submit an issue.

When providing vulnerability reports, offer at least some remediation steps. If the fix is complex, help the maintainer with making non-breaking changes. Don't AI generate fixes unless they are extremely simple checks or one-line corrections, and even then hand check to make sure that what you're submitting as a fix makes sense.

Don't put too much pressure on maintainers for immediate fixes, especially if you don't pay them in any way.

Maintainers:

Be more vocal about how bad the situation is. If you are buried in security reports, ask trusted organizations for help or advice. They can either help you directly or point you to helpful resources that can help you manage the influx.

Help researchers give you better reports. Include your threat model in your documentation (many of the best LLMs/harnesses look in security.md first for validating issues). If you get invalid reports from someone, make sure there is an easily accessible public place for an LLM to read about your rejection and why. If your documentation is off of your git, be sure to include this specific documentation inside of your git. This is particularly big in 2026 as LLMs often get rate limited when trying to pull docs from other sources, and then will continue to give you bad reports. Additionally, be mindful that LLMs have limited context windows, so if you want to prevent a large volume of bad reports, you can't rely on researchers looking in PRs or Issues for clues. The right docs in the right place will reduce the spam.

Volunteer for security scanning if you have time for it. This will optimize your time and energy. Significantly hardening your codebase will dramatically reduce the rate of incoming reports. Multiple organizations either already have built or are building programs to make that as easy as possible. Reach out to the foundations that work in your ecosystem to see what scanning programs they have. Failing that, there's also people at the OpenSSF, Alpha-Omega, OSTIF, OpenAI's Patch the Planet initiative, and more.