• jordanlund@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    arrow-down
    29
    ·
    3 months ago

    Blame the dev who pressed “Deploy” without vetifying the config file wasn’t full of 0’s or testing it in Sandbox first.

    • floofloof@lemmy.ca
      link
      fedilink
      English
      arrow-up
      40
      arrow-down
      1
      ·
      3 months ago

      If the company makes it possible for an individual developer to do this, it’s the company’s fault.

      • sugar_in_your_tea@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        7
        ·
        edit-2
        3 months ago

        Exactly. All of our code requires two reviews (one from a lead if it’s to a shared environment), and deploying to production also requires approval of 3 people:

        • project manager
        • product owner
        • quality assurance

        And it gets jointly verified immediately after deploy by QA and customer support/product owner. If we want an exception to our deploy rules (low QA pass rate, deploy within business hours, someone important is on leave, etc), we need the director to sign off.

        We have <100 people total on the development org, probably closer to 50. We’re a relatively large company, but a relatively small tech team within a non-tech company (we manufacture stuff, and the SW is to support customers w/ our stuff).

        I can’t imagine we’re too far outside the norms as far as big org deployments work. So that means that several people saw this change and decided it was fine. Or at least that’s what should happen with a multi-billion dollar company (much larger than ours).

        • Prox@lemmy.world
          link
          fedilink
          English
          arrow-up
          2
          ·
          3 months ago

          Are “product” (PM, PO) and “engineering” (people who write the code) one and the same where you work? Or are they separate factions?

          • sugar_in_your_tea@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            6
            ·
            edit-2
            3 months ago

            No, separate groups. We basically have four separate, less-technical groups that are all involved in some way with the process of releasing stuff, and they all have their own motivations and whatnot:

            • PM - evaluated on consistency of releases, and keeping costs in line with expectations
            • PO - evaluated on delivering features customers want, and engagement with those features
            • QA - evaluated on bugs in production vs caught before release
            • support - evaluated on time to resolve customer complaints
            • devs - evaluated on reliability of estimates and consistency of work

            PM, PO, and QA are involved in feature releases, PM, QA, and support are involved in hotfixes. Each tests in a staging environment before signing off, and tests again just after deploy.

            It seems to work pretty well, and as a lead dev, I only need to interact with those groups at release and planning time. If I do my job properly, they’re all happy and releases are smooth (and they usually are). Each group has caught important issues, so I don’t think the redundancy is waste. The only overlap we have is our support lead has started contributing code changes (they cross-trained to FE dev), so they have another support member fill in when there’s a conflict of interest.

            My industry has a pretty high cost for bad releases, since a high severity bug could cost customers millions per day, kind of like CrowdStrike, so I must assume they have a similar process for releases.

    • aodhsishaj@lemmy.world
      link
      fedilink
      English
      arrow-up
      23
      ·
      3 months ago

      That’s not how any of this worked. Also not how working in a large team that develops for thousands of clients works. It wasn’t just one dev that fucked up here.

      Crowd Strike Falcon uses a signed boot driver. They don’t want to wait for MS to get around to signing a driver if there’s a zero day they’re trying to patch. So they have an empty driver with null pointers to the meat of a real boot driver. If you fat finger a reg key, that file only containing the 9C character, points to another null pointer in a different file and you end up getting a non bootable system as the whole driver is now empty.

      If you don’t understand what I just said here’s some folk that spent good time and effort to explain it.

      https://www.youtube.com/watch?v=pCxvyIx922A&t=312s

      https://www.youtube.com/watch?v=wAzEJxOo1ts