• 👍Maximum Derek👍@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    80
    ·
    edit-2
    1 year ago

    I once set an S3 lifecycle setting that accidentally affected 3 years worth of logs to Glacier. The next morning I woke up to a billing alert and an AWS bill with an extra $250k in charges (our normal run rate was $30k/month at the time). Basically I spent my entire add annual cloud budget for the year overnight.

    Thankfully after an email to our account rep and a bunch of back and forth I was able to get the charges reduced to $4,300.

      • aard@kyu.de
        link
        fedilink
        arrow-up
        14
        ·
        1 year ago

        The problem is having a competent team to manage your infrastructure. You can do a lot with a handful of people - but you need competences spanning a lot of areas, and finding that is pretty hard.

        If you can get a competent team the only advantage cloud still has is the ability to quickly scale up and down - but if there might be a need for that it’d still be better to go hybrid, most on your own hardware, and just the prepared ability to quickly bring up cloud workers if needed. The cost savings of properly doing it yourself are so huge that it still might be cheaper to just have some pre-provisioned standby hardware for that, though.

      • 👍Maximum Derek👍@discuss.tchncs.de
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 year ago

        If I never have to buzz into another colo and stand in the exhaust of hundreds of servers again, it’s worth every single penny. If I never have to plan for capacity weeks to years in advance again, its worth every penny.

      • zacher_glachl@lemmy.world
        link
        fedilink
        arrow-up
        5
        ·
        edit-2
        1 year ago

        Depends on your needs. If you expect to grow fast and unpredictably, or have extreme burst workloads (at my company it fluctuates between requiring ~10 cpus to ~50,000, and between 0 GPUs and dozens) or if you need several complex types of services and no people at hand who can manage them, it can be way cheaper. If you just need a few servers, a tape backup and a database, actual hardware has always been cheaper.

      • sylver_dragon@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        It depends on the workload. Some workloads do well on other people’s computers, some are better on your own computers. One size does not fit all.

    • railsdev@programming.dev
      link
      fedilink
      arrow-up
      4
      ·
      1 year ago

      I made this mistake but honestly? AWS is the most confusing clusterfuck of all time. I can’t stand it and refuse to use it for personal projects.

      For me the problem came down to four conflicting sources on AWS regarding tiers and then another problem with the SDK. The SDK didn’t match the tiers at all so “archive” meant Glacier for some reason. 👎

    • catastrophicblues@lemmy.ca
      link
      fedilink
      arrow-up
      3
      ·
      1 year ago

      Yeah luckily Amazon is good about mess-ups that are one-time like this. Was the cost because you were pushing to, or retrieving from Glacier?

      • 👍Maximum Derek👍@discuss.tchncs.de
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        Deleting from. We move logstores and I added an ageout policy for anything over 1 day, to “easily” empty a bucket overnight. I forgot that I had been cycling stuff to glacier after 6 months, and there were 3 years of logs in there.