• kkj@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    10
    ·
    11 days ago

    “Cleaning up” seems a little generous. It’s finding them, but someone still has to patch them.

    • The_Decryptor@aussie.zone
      link
      fedilink
      English
      arrow-up
      3
      ·
      10 days ago

      “Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.

      They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.

      • kkj@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        10 days ago

        A 95% false positive rate isn’t actually that big a deal if you have a good way to test the claims. I wonder if you could set it up CtF-style, where you don’t get notified until the LLM can write to a specific file on the target box.

    • P03 Locke@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 days ago

      Most vulnerability patches are a few lines of code. Finding it is the hard part.

      Even then, the AI session that found it probably already has a solution to fix it.

      • kkj@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        4
        ·
        11 days ago

        In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.

        • P03 Locke@lemmy.dbzer0.com
          link
          fedilink
          English
          arrow-up
          2
          ·
          10 days ago

          I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.

          • kkj@lemmy.dbzer0.com
            link
            fedilink
            English
            arrow-up
            1
            ·
            10 days ago

            Do you have a guide on which baseline rules are good? I use Claude Opus 5.5 at work and while it’s definitely more helpful than not having it, it also does some incredibly baffling things that a human would realize immediately are unproductive.

            • P03 Locke@lemmy.dbzer0.com
              link
              fedilink
              English
              arrow-up
              1
              ·
              9 days ago

              “Baseline” is subjective, so I’d recommend just start a global rules file and steer it away from the behaviors you don’t like. Don’t be afraid of quizzing the agent itself on why it did that thing, and what’s a good bit of language to add to the rules file to prevent it from happening.

              Also, for specific tasks, skill files are really useful, and LLMs can write the whole thing, based on what you did the hard way in a session. Same thing for triggering a skill file and then ask it if there are any useful improvements to make to the skills file in the future, at the end of the session.