Arrgh! :( Just this week I needed to do some emergency rearranging of personal infrastructure, and moved one nameserver to Linode Fremont, while the other nameserver is in Linode Dallas. I was fully aware that I needed a more diverse long-term solution, but I figured "What are the chances that both Fremont and Dallas will go down in the next week or two?"
Sorry, folks. I hate that my dangerous flirting with Murphy's Law is hurting other people.
For whatever it's worth, since it was a DHCP problem, I was able to manually configure the static IPs and get the hosts back up. The last incoming connections logged before the outage were around ~ 6:24:00 am MDT (about one hour ago). I guess that's when my server needed to renew the DHCP lease and couldn't.
The number of reported issues coming through has exploded since Akamai. Over the last week alone, almost every day a new networking issue for London is in my inbox. However - none of the recent ones have actually affected any one of my Linodes, even though it's possible since it falls under the service and datacenter - perhaps the change we are seeing is more transparency?
I'm not saying no one is affected... as others in this thread clearly are, and I was affected by the datacenter wide outage in London earlier in the year - but it's possible that being more upfront in notifications of issues could change the perception of what before would have been considered a collection of isolated incidents.</speculation>
[edit]
I generally appreciate transparency, but when it comes to varying degrees of service disruption maybe there is a balance to be struck... right now it feels like the signal to noise ratio is low in terms of it's affect across all users even within an area.
Right now I am pretty much ignoring the Linode issues filling my inbox, if a serious one came through I'd probably miss it.
Well, it appears to be fixed now (for my websites at least), so I'm not as worried about Akamai as I was a couple hours ago. If it's another decade before I have an outage, I'm satisfied. On the other hand, if this happens again any time soon, it will be time to start evaluating options, I guess.
I've also been with them nearly twenty years (!?!), and they definitely have had problems throughout that entire time. perhaps the difference was that you'd get caker@ and co on IRC explaining what was up.
I moved off Linode 5 years ago due to outages and issues. I don't have any experience post-akamai to compare but it was definitely never a reliable service for me in the past.
I searched my emails and it appears it was the Fremont site. It was about 7 years ago I moved off Linode. Initially I went to EC2 but then I just replaced most of my VPS needs with other specific services. FWIW, EC2 was more reliable IME, but obviously the Linode UX was way better.
My Newark node was down for over six hours, this is how I fixed it. Uptime on the VPS had been over a year before this (don't remember why it was rebooted a year ago). Like others here said, I did not greet the news of Akamai buying them with joy, and things like this starting seem to bare my initial impressions out.
this.... replaces the use of DHCP with a static file rewriter on boot, OK, why is this recommended in general, should we expect DHCP to have ongoing issues that their "network helper" would not ?
Yeah, we’ve only reallocated an IP address once (in the past week, as it happens), but that was an exception. Our servers are redeployable and backed up, but they’re not really ephemeral. So they just have their IP “forever”. I think the oldest virtual server we’ve had was deleted when it was almost 3 years old.
They were just bought by Akamai so unfortunately things like this are bound to happen if they are trying to integrate with Akamai the way they'd need to successfully keep growing. I'm not talking just software integration but process and people integration as well. We'll see I guess!
The Akamai acquisition has had me kind of stressed to be honest. Here in Oceania there aren't a whole lot of established "cloud" hosts to pick from that host locally.
Linode is fantastic and I have loved them for 10+ years. Digital Ocean doesn't host here, nearest is Singapore. Vultr does but I have had poor experiences with them. The other options are AWS and Google which are potentially unbounded expenses which is hard to sell to clients that already see $12/m (aud) as a cost sink, no matter what the server+service will save them in seat time.
There are of course local hosts but they're either oversold cpanel hosts or comparatively expensive (3-10x the price for the same hardware).
I guess it just feeds from my existential dread of being a small fish where if any one of these upstream providers decide "oh, no we flagged that account because of [black box reason], sorry!", you can be out a lively hood with no real recourse.
Guess it just kinda sucks to be at what feels like the end of the long tail of infrastructure.
Unrelated to the Linode thing, but in 15 years building and selling SaaS products, I've encountered a number of customers who nitpick prices or try to haggle. I'm not talking about non-profits with micro budgets, rather very profitable businesses.
Nowadays, whenever someone goes there, I just change my usual lighthearted tone to a serious one and say "Yeah, our prices are firm, if that's outside of your budget it's better we just move on". They 180 real fast. Every. Single. Time. Then they never mention prices again.
A caveat with them is that they're really trying to be price competitive, and can sometimes go overboard (like design their own custom datacenters and servers with watercooling, and skipping some things like fire suppression systems, and using cheap material such as plywood). Having a backup with a different provider, somewhere else, is always a good idea.
If you ever plan on colocating a 1U box somewhere, count me in for nibbling a bit off the end of it for $20/mo :) (I'm in Sydney, and seriously, seriously want my latency back)
I did a bunch of digging a while back and it looks like the "simply cannot go any lower" baseline is $250/mo.
I don't think that will happen with Linode (but I certainly could be wrong). When Rackspace bought Slicehost, people didn't really know what the future was going to look like. At this point, we've seen AWS/GCP/Azure and Akamai wants to have their own "run your apps with us" system.
Unlike when Rackspace bought Slicehost, Linode probably isn't even the leader in its niche. That's probably DigitalOcean. Akamai probably figured out which of the two it could buy and DigitalOcean probably would have cost $4+ billion while Linode only cost $900 million. When Rackspace bought Slicehost, there were really two alternatives: Linode and AWS. AWS came with a lot of complexity and Linode was small.
If you're looking for a new Linode to pop up, you don't have to wait. Vultr, DigitalOcean, and tons more already exist and have for years. It's not 2010 where you don't have loads of options in this space. Go with Hetzner or OVH Cloud and get really cheap prices (I've found Hetzner Cloud to be really nice). There are so many places that will give you a VPS today. Heck, AWS will give you a Linode-like experience with Lightsail, right?
That's why I think Akamai wants to use Linode as a way to build something that will let them start getting into the AWS/GCP/Azure space. It's somewhat where Linode was already heading, albeit slowly. Linode started offering hosted Kubernetes, object storage, block storage, hosted databases, and load balancers. How long before they might have added a queuing system, an app-runner so you don't even need to think about boxes/load balancers, etc.
For the low price of $900M, they could jump-start that process at Akamai. Rackspace thought that the future was their high-touch, high-cost, managed services. Akamai probably wants to use Linode to get into the AWS space and with a separate brand, they can do it without as much risk to their mainline business. There are too many alternatives for Akamai to be deluded enough to think they can just ruin Linode and not like that $900M on fire. That wasn't the case when Rackspace bought Slicehost.
Akamai looks at Cloudflare's valuation with envy (and some contempt because Akamai has the larger edge network). Definitely agree Linode was purchased as a growth opportunity.
DigitalOcean doesn't have much Asian presence, Linode's backup doesn't support non default file system like zfs, DO / Linode don't have HDD block storage like Vultr does (which I think is a hidden gem you can't realize until you go add one as it's not mentioned in the pricing page) that can be a very nice option to throw backup data in at 40GB/$.
Vultr's CPU isn't as good as the other 2 by the look of sysbench, even for high frequency types.
Those 3 vendors only have 2 or 3 backup slots which can be kind of unreliable if you didn't notice data loss over a weekend and weekly backup can be too far apart.
Hetzner is almost perfect if they have a bit more location. Very cheap, very high performing, daily backup with plenty of slots but no HDD block storage.
Sometimes Amazon Lightsail can be a decent option as they have 7 daily backup slots, free bandwidth is comparable to the others and prices aren't bad except their CPU suck.
Switch to Pair - https://www.pair.com/ - they are older than Linode and have a great reputation. (PS: However, I am not sure if they have been acquired by another company now. The "old" Pair had a sterling reputation, and I can vouch for that as a former customer.)
Just pointing out that price comparison between two product or service needs to be done with similar yardsticks. A product or service being costlier doesn't automatically mean that it costs you more as it can save your business money elsewhere.
Pair is absolutely not price-competitive. Pair's "Starter I" plan is $30/mo, for 1 vCPU, 2 GB RAM, 20 GB storage, and 500 GB transfer. Linode's most similar plan ("Linode 2 GB") is $10/mo, and includes 2 TB transfer.
Does look like they have hiked their prices. But it also depends on how you are making the comparison. For example, just a cursory look at both pricing shows me some major differences - Pair is offering "fully managed" VPS hosting which means the hosting providers employees will help you do most of the common hosting tasks (and provide better customer support) unlike "unmanaged" hosting where you have to do everything yourself with the tools provided. Is the Linode plan you cited also similar or for "unmanaged" hosting (which is what Linode was famous for)? Note that Linode (Akamia) also makes a distinction between "shared" vCPU vs "dedicated" vCPU and prices it differently. Don't know how Pair looks at that. The point is that the pricing variation can be justified by the services being offered. I admire and respect Akamai and also like Linode a lot. But I also recognize that Akamai is big B2B company offering enterprise services and small or independent developers will always be an after thought for them.
No, I don't. I am vocal about brands I admire or dislike. One of the ways I was taught to evaluate web hosting companies was to check how long they have been in the business (both the company and the management) as web hosting is an easy business to get into (even today, any Tom, Dick and Harry can start a web hosting business with next to nothing investment by becoming a reseller). Pair hosting is one of the oldest web hosting companies I know, and they have / had a reputation for high server uptimes (I still remember one of their employee had once posted in some forum on how one of their FreeBSD run server had an uptime of more than 2+ years - yeah, bragging about server uptimes was once a thing too) and great customer support. (I also have fond memories of Linode, Slicehost, Rackspace etc. before they all gave up and were gobbled up by someone, and web hosting became all this "cloud" bullshit.)
Tl;dr: No. Brand recognition / recall is a powerful marketing tool.
Some old businesses are still running because they've been able to stay relevant over the long term. And some old businesses are still running because they've been relying on name recognition and customers who can't be bothered to move.
Unfortunately, I think Pair is in the latter category.
Businesses that don't change with times do die a slow death. I have no idea how you've come to the conclusion though that Pair is in that category - would you like to clarify?
For Linode Ubuntu 22.04 installations, at least, the default seems to be DHCP, even if the IP address is fixed. And I certainly wouldn't have thought anything about this, since even when using fixed IP addresses, I'm in the habit of using DHCP to provide a central source of IP address truth for my networks, to ease the pain of the inevitable renumbering.
That being said, maybe check if you have `dhclient`, `dhcpcd`, `systemd-resolved`, or (less likely) `dnsmasq` or `bind` running (htop / ps aux).
If you do, your instance may fall over after its DHCP lease expires and it then presumably fails to acquire a new one and goes all 169.254.x.x on you.
The possible counterpoint / information-vacuum bit here is that I don't know what the DHCP lease time is, so the expiry may already have happened.
IOW, apparently this started 4h ago, and so the last leases were presumably issued 4h ago, and maybe the lease length is 1h, so everything's already as dead as it's going to get, so maybe if you're not already dead, you're fine.
We run some Windows instances, which were DHCP out of laziness. <facepalm> [edit: It's a PITA to fire up a Windows instance on Linode. Once working, switching out of DHCP is near the bottom of the todo list.]
I would expect the opposite. Doesn't DHCP make more sense, for example, if the user creates a snapshot and restores it with a new IP? I don't use Lenode, so I can't judge, but I would expect DHCP to be default.
Default created Debian based linodes use a static IP. How they assign you the IP is they have an initial network boot script that injects that static IP into your config.
I learned this because I wanted to test image recovery of deleted linode. When I restored it, I got a different IP address from linode (the static IP assigned to me), and I couldn't connect to it. Using the web console I found the IP address was hard coded to the old one still.
Just checked my tiny Nanode 1 GB image (Fremont) running Debian and it's indeed running dhclient. Still have access, but I need the instance this weekend for a virtual event. Kind of sucky timing for a potential outage, Linode.
I gave up. I submitted an inquiry on their sales form for dedicated and got no response, I contacted their support and got "we'll let you know." They've at least added nvme storage for instances, but in at least their Dallas data center the iops suck. In Newark it's a bit better.
If you've got a bunch of servers at a linode data center and want bare metal ones, I ended up just looking at what other hosts are peered with their nearest exchange. Our servers are in Dallas and Vultr, Equinix via IBM Cloud, and Servermania all had peering to them through a local exchange at infomart or nearby. We ended up moving database servers to those bare metal services and connecting over wan to our linode ones. Latency less than 5ms.
Linode also contacted us to ask why our monthly usage halved. We told them, they offered no alternatives.
Limestone networks (http://lstn.net) offers great bare metal services. I used to work there 7-8 years ago and they are a no frills, no bs company. They will give you a good deal also :)
A bare metal data point, since I just went through the process of finding an alternative to Packet.net since they were acquired by Equinix : try Hivelocity. My experience with them has been so far so good.
I think "Outage" as stated is a bit over-the-top. We run significant operations in most of these DCs, and we have seen zero impact. YMMV of course, but I think it's a big disingenuous to say this is a full outage.
For what it's worth I have around 25 Linodes active, mostly in Newark but a few in Atlanta and Dallas as well.
Of those I had two in Newark which actually had their leases expire before we were aware of what was going on and then found three more (two Atlanta, one Newark) once we knew what to look for which were failing to renew. All five of those machines already had network helper enabled as Linode suggests but were unable to get a lease even with manual operation of the DHCP client until they were rebooted.
Linode London has been experiencing "greater than normal" intermittent packet loss to Akamai network which they use to peer with other player such as Cloudflare in LDN for nearly two days - before they peered with CF via Linx if I remember correctly. It isn't just isolated to UDP customers. Currently it's fairly stable so for most it's hard to notice but certainly uncomfortable how far they want to drag this problem on - see status.linode.com for how grim the LDN situation is - their support said we'll tell you in a few days what it was once we fixed it. i.e. it's not the rock solid of the last 7 years and their support are largely deaf to how unacceptable this is based on previous standards.
to whomever is at the top of the automation repo commit log, i wish you a safe and speedy recovery. Enjoy this friday, that it may succor a distant memory of fridays past when the latest build didnt break production.
Amusingly, I am trying to log in to the dashboard to reboot my nodes, but it requires a verification code which was sent to my email, which is hosted on Linode.
Consider signing up to a service like MXGuardian that your MX records can point to, and that then forward email to your server. This will give you an “emergency inbox” that you can use in cases like this (might also help with spam depending on how you are currently dealing with that).
My sites in Newark are up. The situation is not as alarming as the submission title would suggest. They've updated the status page to indicate that it is limited to an issue with DHCP.
I had 6 servers out of about 50 go down, all in Dallas/Atlanta even though most servers are in Newark. IPv6 connectivity seemed to remain up for all of them, though.
That’s too bad. Which other vps hosts are folks using these days besides aws/azure/gc?
If anyones thinking of switching to digitalocean, I’m currently switching linode from digitalocean who had an unforgivable, catastrophic loss of vm.
Digitalocean also prevebts customers from downloading disk image backups offsite for any kind of adult DR. Linode lets you download backup images right from the admin console.
For what it’s worth, I’ve been a Linode customer for about 12 years and have never had a problem. This current issue does not even affect me as my boxes are on static IPs. My setup consists of a number of Linode VPSes, a single Hetzner Box in Germany, and an S3 glacier bucket (the only Amazon service I use) for backups. It’s probably not optimal, especially for the business case, but it works for all my hobby and side projects.
Not leaving them because of how good their tech support is or was to help me early on in my career, but they can’t be a point of failure either.
I’ll just get to see how easily a hybrid cloud might be orchestrated like yours. Things are drastically easier to learn to setup and maintain like you have said compared to 5 or 10 years ago.
I was thinking on moving to Hetzner (from Digital Ocean) - now that they have/will have a data center in Virginia....but i have heard iffy things about hetzner (some people are happy, others not so much). I should clarify, i'm only thinking on moving due to costs (not because i have encountered any ops or support issues). The stuff that i host is not as essential, and could do with slightly lower costs. ;-)
Sorry, folks. I hate that my dangerous flirting with Murphy's Law is hurting other people.