Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind?

Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on.

But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?

    • mattburkedev@programming.dev
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases.

      Every enterprise I’ve consulted for that had code coverage requirements was full of elaborate mock-heavy tests with a single Assert.NotNull at the end. Basically just testing that you wrote the right mocks!

      • MagicShel@programming.dev
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        That’s exactly the sort of shit tests mutation testing is designed to address. Believe me it sucks when sonar requires 90% pit test pass rate. Sometimes the tests can get extremely elaborate. Which should be a red flag for design (not necessarily bad code).

        Anyway I love what pit testing does. I hate being required to do it, but it’s a good thing.

    • xthexder@l.sw0.com
      link
      fedilink
      English
      arrow-up
      0
      ·
      1 year ago

      I’d never heard of mutation testing before either, and it seems really interesting. It reminds me of fuzzing, except for the code instead of the input. Maybe a little impractical for some codebases with long build times though. Still, I’ll have to give it a try for a future project. It looks like there’s several tools for mutation testing C/C++.

      The most useful tests I write are generally regression tests. Every time I find a bug, I’ll replicate it in a test case, then fix the bug. I think this is just basic Test-Driven-Development practice, but it’s very useful to verify that your tests actually fail when they should. Mutation/Pit testing seems like it addresses that nicely.

      • Sleepkever@lemm.ee
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        We are running the above pi tests with an extra (Gradle based) build plugin so that it only runs mutations for the changed lines in that pull request. That drastically reduces runtime and still ensures that new code is covered to the mutation test level we want. Maybe something similar can be done for C or C++ projects.

        • xthexder@l.sw0.com
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 year ago

          I’m currently working on a C++ project that takes about 10 minutes to do a clean build (Plus another 5 minutes in CI to actually run the tests). Incremental builds are set up, and work quite well, but any header changes can easily result in a 5 minute incremental build.

          As much as I’d like to try, I don’t see mutation testing being worthwhile for this project outside of maybe a few isolated modules that could be tested independently. It’s a highly interconnected codebase, and I’ve personally reviewed (or written) every test, so I already know they’re of fairly high quality, but it would be nice to be able to measure.

    • canpolat@programming.dev
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      1 year ago

      I think this is a good rule-of-thumb in general. But I think the best way to decide on the correct coverage is to go through uncovered code and make a conscious decision about it. In some classes it may be OK to have 30%, in others one wants to go all the way up to 100%. That’s why I’m against having a coverage percentage as a build/deployment gate.

      • raspberry_confetti@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Bingo, exactly this. I said 80 because that’s typically what I see our projects get to after writing actually useful tests. But if your coverage is 80% and it’s all just tests verifying that a constant is still set to whatever value, then yeah, thats a useless metric.

  • snowe@programming.devM
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    Mutation testing. Someone else mentioned it as PIT testing, but its actual name is mutation testing. It accomplished exactly what you’re looking for here.

  • UFO@programming.dev
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    I’d like to see state space coverage instead of line coverage. That, at least, catches silly “100%” cases.

    I don’t know of a tool that provides this metric. I don’t even think such a thing could be made for most languages. still, useful to think about when reviewing code.

  • cgtjsiwy@programming.dev
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Different applications require different tests, so no measure is going to please everyone. If you’re making embedded devices for an airplane, the buyer might ask you to provide a formal proof that the program works. In contrast, web apps tend to simply use end users as testers, since it’s cheaper.

  • SorteKanin@feddit.dk
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?

    Formally? No, this is basically impossible by Rice’s Theorem. There is not even a guarantee that if you have 100% test coverage, the program is good (the tests could be flawed).

    This is just a natural limitation of turing completeness. You can’t decide these properties while also having full computational power. In order to decide such things, you need a less powerful mode of computation (something not turing complete) that can be analyzed more thoroughly and with more guarantees.

  • fades@beehaw.org
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    1 year ago

    So true lol. Mgmt just announced a directive at my work last week that code must have 95-100% coverage.

    Meanwhile they hire contractors from india that write the dumbest, most useless tests possible. I’ve worked with many great Indian devs but the contractors we use today all seem like a step down in quality. More work for me I guess

    • phoneymouse@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      It’s always fun to hear management pushing code coverage. It’s a fairly useless metric. It’s easy to get coverage without actually testing anything. I’ve seen unit tests that consist simply of starting the whole program and running it without asserting anything or checking outputs.

  • Sibbo@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Maybe fraction between money spent on writing code versus money spent on testing code?