New

In Singapore, we enjoy very good broadband at quite a reasonable cost. Even on residential plans.

5 to 6 years back, I used to routinely push backups to the GCP object storage and their Singapore AZ - and still do to a small extent. Although I normally throttle that, when lifting all limits, I would see sustained numbers in the range of 950Mbps. Back then, my fiber plan was 1Gbps so this was basically line speed!

Cross-border traffic is a different story.

In Singapore, more than in other countries I have been to or lived in (except China, which is a very special case, of course), your bandwidth on cross-border traffic when on residential plans can very quickly fall off a cliff. My favorite example, just take a 40 minutes ferry ride to Batam. On a 100 megabits fiber plan, which Confirms. The 100 megabits are there when doing a speed test to the nearest ISP server. Trying to connect back to my residential to my servers who are on a residential plan with Singtel in Singapore. Only shows a download speed of 3 to 5 megabits, sometimes a bit more if you are very, very lucky. In a nutshell, at least on Singtel, Peering is very poor the moment you get out of Singapore.

I still have servers in France, for instance, at my parents’ house that I use for storing backups. And they do not see such issue on ISP. I haven’t seen such dramatic issue with any of the ISPs they’ve used over the years, whether free or SFR. None of them exhibit such a dramatic loss of bandwidth the moment you need to do international peering.

The problem here, most likely, is that Singtel on residential plan doesn’t pay too much attention or doesn’t allocate that much of budget to having good international peering for you. They probably peer with the big guys, hosting the Netflix, Cloudflare, AWS, and so on. But of course, a smaller Indonesian ISP or Vietnamese ISP, they don’t care about it. And the problem there is then, is that when you travel or if you’re hosting services on those landlines, of course you should not host anything critical on a residential line, but nonetheless, the moment you go international or you’re outside the country, then the speed is terrible. Latency also becomes quite bad.

So what to do then? Over the years I’ve looked at a couple of tricks and there are two nowadays that came on top, and work well for me.

They’re both essentially the same thing. You need to ride on the private backbone or good peering agreements of another provider. And usually that’s a hyperscaler or a major CDN or network provider. How do you do that? The first solution, which is actually my personal favorite, but maybe not fantastic from a strictly privacy preserving perspective, is to leverage on Cloudflare. Cloudflare Tunnels and or the Cloudflare VPN service they know called Cloudflare Zero Trust.

The second is to get a small VPS at one of the hyperscalers or a data center that has good peering agreements and route your traffic through that VPS. Traditionally, I used to do that with a combination of WireGuard tunnel and traffic or NGX proxies, plus a small dose of split tunneling. DNS records pointing to my local server went online and then public DNS records pointing to the VPS when outside - on the public internet or aboard. More recently, another solution has appeared, I would say, over the past year. And this solution is really very solid. I’ve been very happy about it. This solution is called Pengolin. And it basically automates the entire process of deploying the tunnels and then proxying the traffic for you. But it also adds some very nice fairy dust on top, which includes monitoring tunnels and traffic, whether they are for not, automating the distribution of secrets, keys, tunnel ports and also uptime monitoring for all of those resources. It is also multi-site and multi-organizational. It’s published under AGPL3 if I’m not mistaken and they have a very generous Fossil Yard license which allows you to use all the enterprise features as a private user, so for personal use, or as a small business owner. The enterprise plan, which they do have, is also priced very competitively. When I see what they charge against some of the other solutions available out there, it’s very reasonable. out there is very reasonable. Not only that, but they are not building in zero trust components into the platform. So you don’t just, if you want to, you don’t just expose the resources externally, but you now have the possibility to make them accessible only through VPN. So basically you proxy the resources, a bit similar to what you would do with Telscale, you know, an advert of the solution, but then you also make them available only tousers that are running the VPN client. So essentially, it’s also a very, very comprehensive solution, very comprehensive platform. Of course, the problem of such platform is, as you put, ever-increasing amount of trust in them, the compromise of a single component can basically become even more catastrophic. And that is the whole problem with solutions that generally describe themselves as zero trust. And I’m not the only one making that point. You have similar communication from cybersecurity agencies such as the ANSSI from France who highlight this critical point. So keep in mind, especially if you’re handling enterprise networks or sensitive networks, you don’t want to put all your eggs into the same basket. Zero trust solution is not a silver bullet. You still need to have strong network principles laid out. This is quite easy and works very well, but of course you need to be careful about where egress charges. A lot of the service providers do charge substantial egress charges. One exception to that that works really well is OCI, aka Oracle Cloud. OCI not only has very generous features that allow you to set up a pretty robust infrastructure to put all of this in place, an infrastructure that meets all IMH and enterprise compliance guidelines without spending a single cent. But also, they have quite a generous English free chair. To be precise, you get up to 10 terabyte of free traffic. In a nutshell, what I have laid out at Oracle, all within Freechair, encompasses Bastion Service to administer the resources:

  • VPS with quite generous specs
  • the equivalent of a VPC, to segregate those instances.
  • equivalent of ACLs and security group to harden, you know, restrict network traffic
  • etc.