TL;DR:
We've been on the Cloudflare Business plan ($250/month) for years. They suddenly contacted us and asked us to either pay them $120k up front for one year of Enterprise within 24 hours or they would take down all of our domains. While this escalated up our business we had 3 sales calls with them, trying to figure out what was happening and how to reach a reasonable contract in a week. When we told them we were also in talks with Fastly, they suddenly "purged" all our domains, causing huge downtime in our core business, sleepless nights migrating away from CF, irreparable loss in customer trust and weeks of ongoing downtime in our internal systems.
Backstory
I'm a SysOps engineer at a fairly large online casino. (I think this article is relevant regardless of whether you think that in general casinos are ethical or not, I’m just mentioning it for context). We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
Now admittedly, $250 is probably fairly low for the amount of traffic we were pushing through Cloudflare. We mainly use CF for the CDN (caching all our static content) and DDOS protection, for which it works pretty well. It’s easy to use and you don’t usually have to think about it much.
I had read a few articles on Hacker News about how at some point Cloudflare contacts you, asking you aggressively to move to "Enterprise" with custom pricing. But I wasn't expecting it to go this horribly.
April 19, 2024
In April, we received this email from Cloudflare:
This sounds like there's some issue with our website. We scheduled a call with their “Business Development” department. Turns out the meeting was with their Sales team, and they didn't have any “serious issues” to report at all. They asked us whether we would like to consider Enterprise. We politely declined, a bit confused as to the tone of the email.
May 3, 2024
Two weeks later, we received another email:
Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way, including by removing any affected secondary domains from Cloudflare.
We sent them info about our domains and tried to get more information from them about the issue and who from our team we should get involved, but they refused to give us anything apart from a date for a call.
May 7, 2024
So we scheduled another call, now with their "Trust and Safety" team. But it turns out, we were actually talking to Sales again.
They said they could offer us an amazing contract for $10k per month with all kinds of great features. We tried figuring out how exactly this was related to the TOS problem and how to resolve the situation. We asked them which domains were affected by their “rotation” concerns. They didn't give us an answer. We asked which of the Enterprise features we actually had to get.
They would not offer us anything apart from a full deal for $10k per month, which would magically resolve the issue. They were not interested in any other resolution.
They said we had 24h to sign their contract because they had to “get back to Trust & Safety". We asked to pay monthly. They said we need to sign a yearly commitment and pay the full year up front. It felt like extortion. Pay us $120k until tomorrow or we destroy your business.
After the call, they sent us this:
Note the email implies a monthly payment might be possible. When we asked for clarification, we were told we must pay the full year upfront.
We do not need most of those features they mention. I understand asking us to do BYOIP to remove their liability for our domains, but the rest is all things we don’t need or are purely “nice to have”.
We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
But still, they didn't care about any other resolutions to the issue and refused to give us any other contract options. Finding numbers online is difficult, but if you squint your eye a bit (compare with this post: https://news.ycombinator.com/item?id=29333160 and this post https://news.ycombinator.com/item?id=31336515 ) 80TB of traffic might have a reasonable price of $150-$2000 per month. Note that 80TB is the number they tried to sell us, I don’t know if it is accurate since they removed all our access to historical analytics.
During this time we also looked into alternatives and set up a test domain and call with Fastly, since they seemed to be a reasonable competition.
May 16, 2024
In another call, trying to negotiate a reasonable contract, our CEO told Cloudflare Sales we were also talking with a competitor. I would have thought this is obvious, who wouldn't look for alternatives when getting slapped with a $120k invoice? But a few hours after the call, this happened:
Cloudflare had suddenly deleted all of our domains. All of our DNS records, caching setup, rate limits, whitelists, gone. Our public website, our incoming emails (including support emails from our customers) and our internal infrastructure, our authentication configuration on Cloudflare Access, down.
They also sent us this email:
The email says “this [...] does not impact current services”, so we frantically wrote them a support ticket but got no response. So we called in our SysOps team and started migrating our main site to Fastly. We had the basics after a few hours, but even then, a "NS" DNS entry change apparently takes a pretty arbitrary time to propagate everywhere, from 1h to 48 hours. We’re still recovering from the aftermath.
At some point, Cloudflare responded to our ticket with this:
“Trust and Safety” never reached out to us, and our account remains locked.
My tips for when Cloudflare reaches out to you
First of all, congrats! Your business must have become pretty successful. How exactly did CF decide to “ask” you to switch to Enterprise?
Maybe...
...you hit 10TB of traffic per month
...their lava lamps went into a specific astral alignment
...a sales rep realized that they haven't hit their quarterly quota yet
In the end, who knows? Cloudflare has absolutely no information on when they will force you into custom billing, but when they start "urgently" needing to talk to you you're probably not going to get out until you have a juicy custom contract with them. There's a reason why they have no public information anywhere on traffic limits or Enterprise pricing. Their Sales team will use anything (like having multiple domains) as fuel to force your whole account to Enterprise , no matter if it is fixable in a simple way.
The price they give you is going to be purely based on what they think you might pay, not on any measurable metric or feature set.
We tried asking how the price is going to be affected if we have less traffic, but they refused to say anything except 80TB is included (we have a large amount of callback traffic that uses IP whitelists and thus doesn’t actually need to go through a CDN, we just never spent time optimizing it since unlimited traffic was included).
We tried saying that we don't need any number of the 14 features that are included, they said all those amazing features are included whether we want them or not.
Numbers found on Hacker News threads (links above) suggest that the prices vary by at least one order of magnitude for the same services.
We tried saying our different domains (like internal ones) don't all need Enterprise, they said the whole account is Enterprise.
If they think you are flakey (maybe if you have alternatives?), they will give you an unreasonable deadline and force you into paying the year up front.
They will use any excuse as a reason for why you suddenly "need" enterprise, even if you're happy with the feature set of Business.
We're not the only ones that got their business threatened by CF’s aggressive sales tactics:
Just because you're paying $250/month, don't expect any kind of courtesy or (non-sale) responses to support mails. If you want CF to respond to you outside of Sales, the only way is apparently to give them negative press.
Be ready to move away from Cloudflare within 24 hours.
Never register domains directly on CF. If you do this and they block you, I have no idea how you can get your domain back in a reasonable time frame. Luckily we only had our NS pointed to CF and thus could move away with ~3-24h of downtime for most users.
Don't use any custom caching rules on CF. CF by default ignores most / all standard HTTP cache headers except for an arbitrary set of extensions and encourages you to create custom rules on CF. Instead, set CF to "Cache: Always" (this does not actually mean always) and "Respect Origin Headers". That way the rules will work for other caching proxies.
Don't use any proprietary Cloudflare products like Zero Access or Workers. We heavily used Zero Access for authentication in internal products, and now we have to rebuild all this infrastructure from scratch with massive downtimes. Only use their technology where it is compatible with third-party standards.
Make backups of your configuration on Cloudflare. It's an unexpectedly large pain to recreate all those configurations, including various sending email services (SPF, DKIM, …), site verification DNS entries, ip lists, rate limiting rules, etc.
Make sure you understand the impact of CF’s business model on you: Either you’re leeching off Cloudflare (customers on the free/business plan), or Cloudflare is leeching off you (intransparent Enterprise pricing). There is no in-between, and at some point the time comes to switch.
And: Consider whether you need Cloudflare at all.
CF only managed (for us) pretty large DDOS attacks. If you have some a bit more vulnerable attack surfaces (for example, an uncached unauthenticated API request that eats up 100ms of CPU time, and can thus use up your cores with just 10-100 requests per second), Cloudflare is not even going to detect it. Especially since all semi-professional “DDOS attack as a service” groups seem specifically specialized for Cloudflare-backed services, including “Under Attack Mode” workarounds etc.
Subscribe to Robin’s Substack
My personal Substack
This is a classic case of someone demonstrating themselves as being victimized in an effort to get others to sympathize with them. All the while, creating artificial ill will towards a vendor.
Imagine the horror of a company trying to run a sustainable business model where they return a profit???!!!
I read this and I see someone that's portraying their role in the situation as "we've done nothing wrong and they want to make us pay $120K to continue doing business." This requires peeling back the layers of the onion to see where the fault truly lies.
Since you’re openly sharing domains/emails of who you spoke with at the vendor, surely you could share the domains you using for your business.
If it’s a casino, it should be something we could go look at and become a customer of, right?
I mean...who doesn’t love a little online gambling in the middle of the night, right?
Why not come clean with the details of what they observed you doing to level the playing field?
This following statement is utter BS and IMHO, discredits anything else you've shared:
“When we told them we were also in talks with Fastly, they suddenly "purged" all our domains, causing huge downtime in our core business, sleepless nights migrating away from CF, irreparable loss in customer trust and weeks of ongoing downtime in our internal systems.”
Clearly your talking to Fastly had nothing to with your domains being purged.
Your domains were purged because you were in violation of terms of service. Not because you were talking to a competitor.
"Your account and domains were brought to our attention following intelligence of your account being involved in domain rotation activities, namely, activities to evade or otherwise circumvent blocks being placed on you by a third party."
In other words, you allegedly knew there were attempts by third parties to place the Cloudflare owned IPs associated with your account on block lists. Cloudflare detected said alleged activities carried out by your organization to circumvent them from being added to block lists.
And of course, this is all being done with IP addresses that belong to Cloudflare - not to you.
Anyone that understands how Cloudflare works knows their IP address space is shared across all of their customers. I would hope they would care a lot about the reputation of their IP address space.
Any actions that put their IP addresses at risk subsequently puts their other customers at risk.
Had you been using BYOIP all along, this probably would not have even been an issue and you probably would still be on their platform.
But BYOIP is only available to customers on an Enterprise plan so it isn't cheap.
I guess it's a calculated risk on your part. What is the cost to your organization if it was blocked vs. the cost to your organization for services that provide you with the ability to do what you need with your own addresses?
The email from support on 05/03/2024 informed you that you had 48 hours to provide them with what they requested or discontinue the activities:
"Usage of Cloudflare services for this purpose is strictly prohibited, and we would request you provide information as to what your account and domains are being used for within the next 48 hours. Note that your account may be terminated should you fail to respond, or otherwise react to this notice."
Based on what support said, they would have purged your domains on May 5th, had they followed what they said they were going to do.
The log you shared show your domains were purged on 05/16/2024 - 13 days after the day they reached out to you.
They were actually very generous seeing as how they provided an additional 11 days to get things under control and to move you to a plan that was more in line with your actual utilization and requirements.
They kept up with their commitment until they determined you were in violation of the terms of service.
Once you violate terms of service, it doesn’t matter who the provider is, the provider has every right to shut you down.
This is all too typical. Most people do not realize how much bandwidth, infrastructure, colocation facilities, R&D, support, etc. cost. Even on a Business plan for $250/month I would have to think they were losing money on your account.
Anyone can spend time going through the Cloudflare subreddit and read of the horrors of how they treated someone on a Free ($0)/Pro ($25)/Business ($250) plan.
Pricing is not based sheerly on the amount of bandwidth consumed or data transferred. There is a wide range of factors that influence the price.
It would be interesting to see what services Fastly required you to sign up for. Or how long you last on Fastly should you end up violating their TOS.
Hopefully your risk management team has a contingency plan in place in the event that you get booted from Fastly as well.
I don't think any of us want to see you go additional sleepless nights!
Hi, we've had the exact same thing happen to us, the BYOIP, the trust and safety team but it was sales, we can talk and discuss about how to NOT use cloudflare?