• JohnEdwa@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    5
    ·
    15 hours ago

    Yes, a 55% false positive rate technically means more often than not, it isn’t actually a valid vulnerability and it is wasting your time. But if you have 1000 boxes to go through while looking for a certain item, someone with a 50% false positive rate could identify two boxes for you, and it would be in one of them.

    So unless that number is “AI claims there’s a vulnerability on every other line of code you wrote”, which I kinda doubt, and is actually based on made bug/vulnerability reports, then isn’t is extremely helpful even if it’s only right half the time?

    • Aeder@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      3 hours ago

      The numbers are based on what happened to curl, since they are transparent about their results, and the ones that weren’t false positives were underwhelming low severity vulnerabilities.

      Of the 34 reports published so far, 19 have been marked as not a vulnerability, 6 have been confirmed as low severity, and 9 are still awaiting assessment. That puts the false positive rate at 55.9% so far, and only 17.6% of findings have been confirmed, all of them low severity. That’s a lot of work dumped on open source maintainers for a small handful of low-impact bugs, and when a maintainer has to pace reports just to protect his team, that says plenty about the burden.

      This also isn’t new for curl. Back in May, Daniel wrote that five Mythos findings boiled down to a single confirmed vulnerability.

      Here’s the report: https://www.vulncheck.com/blog/anthropic-glasswing-november-update

    • MrScottyTay@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      5
      ·
      14 hours ago

      Getting what I’ve seen even the actual vulnerabilities are the sort of stuff that is very very very hard to pull off, like it’s identified a .2ms window where if you input something specific, you break something and gain access. Like race conditions that even though they theoretically exist, they’re actually impossible to execute