"Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they’re not getting charged for this.
It’s currently our policy to not shut down free sites during traffic spikes that doesn’t match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn’t come through in the initial support reply."
This was posted 4 days ago in hackernews.
And that’s all he posted. I think he responed to one of two comments and then ignored everything else. I really dislike that ‘CEO here’, ‘I’m important, listen to me’ means that always ends up on HN. Then they disappear like their mere response to the post is enough. After all, they’re very important people.
If he thought ‘damaged control done’ he was sadly mistaken.
I host my site on Netlify. I’m moving. If they think that it’s acceptable to bill somebody $104k for a small site, at all, without it tripping some alarm for a human to look at before it goes out, then they’re doing it wrong. Something that says ‘Month 1 bill = $20; Month 2 bill = $104,000’ could be a problem isn’t difficult to do. And that they have ‘done this often’ (my words) highlights it’s a problem.
There are many bullshit hosting companies out there I can use who don’t do this sort of thing. Why is Netlify special.
You don’t think the person who’s ultimately responsible for a company’s policies is important in a discussion of those policies? There’s nothing arrogant about knowing you’re the one at the center of a news story.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
Not everything is an algorithm or ratios. Hopefully you know that.
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
See above.
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
Theres a solvable issue when your site doesn’t deploy. And then theres repeated issues that cause you to send life changing / threatening bills to customers.
If you think they should be treated the same, let me know when you become CEO so I can avoid your business as well.
Good luck with your career though.
Not everything is an algorithm or ratios. Hopefully you know that.
This is just straight up wrong. Literally everything theoretically could be extrapolated into an “algorithm or ratio” complex enough to consider every factor, and perfectly simulate the real-world outcome.
You obviously had an issue with the statement as-made by the CEO. The question you’re dodging is essentially “how different would it have to be for you to not have the issue you’re having right now?”
Unless, of course, you’d just be angry no matter what the guy said. If this is the case, it would easily explain why you’re throwing out non-answers and superfluous catty dialogue.
Not everything is an algorithm or ratios. Hopefully you know that.
This is just straight up wrong.
It’s not but you do you.
You obviously had an issue with the statement as-made by the CEO.
Go back and read what I wrote. I never wrote that.
The question you’re dodging is essentially “how different would it have to be for you to not have the issue you’re having right now?”
Read what I wrote. You’ll find your answer in there.
Unless, of course, you’d just be angry no matter what. If this is the case, it would easily explain why you’re throwing out non-answers and superfluous catty dialogue.
If you don’t read what I’m saying, me repeating it again, isn’t going to help that.
I find it astonishing that Netlify had no safety mechanism in place to prevent this.
Saddling customers with unbounded liability is irresponsible; arguably negligent.
Definitely negligent, I still remember the young adult who killed himself when he thought his Robinhood account was negative nearly 3 quarters of a million dollars.
Makes you wonder how many customers were wrongly charged some other less insane amount, and no one noticed because it wasn’t jaw dropping.
“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I’ve currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
One user on Hacker News with the alias ‘bobfunk’ introduced himself as the Netlify CEO and assured users that the bill would be forgiven. Cybernews was unable to verify the CEO’s identity independently. However, many previous posts from the same user and his bio support the claim of him being Matt Biilmann, the founder of Netlify.
In another twist, the DDoS attack version of the story is being ruled out
“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
“We’ve confirmed that the user was notified multiple times about the additional bandwidth that was being consumed on their site, but given their lack of response to these notifications, we believe that we should revisit and improve the messaging and urgency that’s being communicated.”
I’m kind of impressed by the amount of research they did to figure out why this guy’s bill was so high, then immediately offered a resolution, and then immediately offered another avenue if the resolution wasn’t good enough. Shout out to the customer service rep.
“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
I’m confused, what is this supposed to mean? Some sort of non-distributed DOS attack? How would working with the customer help there? If they’re susceptible to a denial of service, isn’t that entirely an internal problem?
I am too. Is the agreement to charge per mb downloaded? Do they not have some sort of "turn it off if I hit this max?* feature?
I usually avoid hosting solutions like this just because of this shit. I wanna know how much I’ll owe before the month starts even. Anything else feels like gambling.
They are saying that it wasn’t a ddos at all but organic use. The user was notified but did nothing. So they think their notifying stuff isn’t good enough.
Sorry, but what exactly is a “sustained download event” supposed to be? It sounds like they’re describing some sort of DOS-like attack that isn’t a DDOS, where a user manages to force the server to serve up way more data over a sustained period of time than would be reasonable for downloading a single MP3 for normal use.
But maybe that’s not what they mean. It’s very unclear.
Basically, it was a giant uptick in use that was likely made by human beings instead of a DDoS botnet, and they’re still investigating where it came from
I use Netlify to host my frontend projects and portfolio. Does anyone have a way to prevent something like this?
I recommend hosting your projects on Cloudflare Pages, as it is a free service provider to the best of my knowledge.
