Self-hosted S3

Disclaimer

First of all, don't do this by default; use AWS S3 if any:

• You are not a hardware (server) expert.
• You plan to have a production service or data backup (your photos are entered in this requirement).
• Your company rules or data requirements don't force you to be on-premise data centres.
• You don't live in the middle of nowhere, and internet access is just fine enough.

Now, with this clarified, If your plan is to learn, have some fun, want to get used to S3 API and ways of working without expending money, or you really need to have your own S3 forcibly, then I hope this guide helps you on the right path.

Starting point

So, let's start working. As you may know, MANY systems allow you to use S3 as their backend. This means automatic backups, "disk" mounted as a network from S3, webs provided directly from S3, CMS, and even some databases! I don't know, the list is vast.

On the other hand, traffic entering the cloud is free, but traffic leaving is expensive.

This brings an issue if you plan to use it from outside of AWS as your backend. You'll see a (not) funny bill EOM.

Also, S3 is exceptionally cheap for low volumes, but if you plan to do a backup of all your YouTube raw creator videos, you have an issue, well, two issues.

• Typically, you're talking quickly about TB of information so that S3 will ask for a funny bill each month.
• The second, and not minor, is relatively easy right now to have 10G or even 40G at home to transfer TB in a matter of few minutes, but transfer 8Tb at the speed of internet connections, even assuming that you have 1G, takes around 18 hours non-stop. Compared with 26min with 40G.

And allow me to repeat myself so we don't forget the expected usage:

• You want a backup that can be restored
• Worst, you plan to work directly on those files

Let's continue with the YouTube video creator. Let's call him ElMoreno. ElMoreno has 40TB of videos that he wants to keep safe and, at the same time, use as a repository for future videos. ElMoreno must fulfil the 3-2-1 rule (more info here) as part of his plan. But he doesn't know everything we'll discuss, so he hires a temporary system engineer with experience, Carlos.

The Knot

Carlos has an unlimited budget, so he will mount a decent solution for ElMoreno. It is not the best, surely not a tier IV class, but it is enough for him to do this in a safe way. Carlos, first of all, ask:

• Do you mind if we use the AWS S3 cloud service to store your data?
• How much is it going to cost me?
• Around $150/month or $1800/year just to have a third copy outside of your normal environment and just to retrieve in case of disaster recovery
• Ok, I allow it

Great, from the 3-2-1, he solved the 1 and the 2. Now he needs the 3 number to achieve it. Okay, he decides then to buy a server with the following characteristics and accessories:

• 12 LFF (Large Format Factor or 3.5") bays for HDDs (ideally SAS)
• 2Gb writing cache with battery (raid controller)
• 2 NVMe drives for live cache (2TB each)
• 12 SAS 10K (this is the RPM) 12TB
• 2x40G SFP+ network card
• 2xTransceivers Fiber Optic multimode LC-LC
• 20 meters of OM4 Fiber Optic with LC-LC connector
• CPU and RAM are quite irrelevant here, but let's use some Epic AMD and 128Gib DD4 as a reference

Now that he has the server in front of him, he creates two hardware RAID 6, with a total of 48TB of data each. And when he enters into the SO (Fedora 37, for example), he mounts both into `/srv/data-a` and `/srv-data-b` using XFS as a file system type.

Then, he launched two nodes of Minio in Docker, using data-a and data-b as data folders of both. He now goes to AWS and creates a bucket in S3 called data-c, with some data retention policies and as default storage "Glacier".

Lastly, he enters into the Minio console, configures both as a cluster and some tiering and gives a KEY and SECRET to ElMoreno.

The details

Did he fulfil the request? Well, the short answer is yes. After an oversimplified version of the steps, what you have is a cluster that automatically generates a 3-2-1 copy of all the files and can be mounted as a single hard drive from Windows/Linux/MacOS and can be accessed at 40Gbps or 5Gbytes per second (an entire BlueRay copied every 5 seconds).

What is happening on the inside? MinIO is an open-source software which duplicates the core functionality of AWS S3 and the APIs. So, all the software that can talk to S3 can talk to MinIO transparently.

The second thing is that due to the cluster situation (and the tiering config), EVERY file you upload to any of the clusters will be replicated on the other AND, due to the tiering, if it stays there untouched for enough time, he is going to upload a copy to the AWS S3 in the concept of Glacier storage.

Gracier is the proper method to store massive amounts of data in an extremely cheap way, assuming one critical detail: you won't have access to it, not just often, rarely or never. Restoring is quite expensive, but you use it because you need it to continue your service.

So, it's the ideal backup to have it as a second medium outside your normal space.

But what about the real-time usage?

Yeah, we didn't talk about that but then is when the magic of the NVMe that we talk about is being used to accelerate working and really be able to use that 40Gpbs line.

If you configure them properly, the SO (Linux) will use it to copy info from your Hard drives and keep it there to be extremely fast to access, based on the usage you do. So it's also like S3 when moving stuff from Glacier to Standard but without the retrieval cost. Most modern solutions of NAS allow you to configure the behaviour, amount of space available for this, etc.

From the outside, you never see those NVMe as a separate folder; it is just like having the files in memory but without the need to have 4TB of RAM. Then, when ElMoreno accesses his local S3 using s3fs he will see a 48TB drive. It's extremely fast sometimes (like using the local one or better), and some others a bit slow for a small period of time.

The videos he used or uploaded last week for his last video are there, so he can use them to take pieces, cut them, and see how the video collage will look.

Who uses stuff like this?

In my previous jobs, working for companies from the TV environment, it was very common to create daily news content. Those teams have dedicated access to servers and machines with configurations like this. When you move to companies that size, the solutions are using specialized hardware, pushing the boundary to use 100G, and typically also private software.

But imagine that, with a budget of $6K, you can have an incredible solution at home that allows you to have all your data safe and ready to be used and consumed. We're not discussing using this for your personal photos, but we can create equivalent solutions for that scope. There, you don't need NVMe, 40G or 48TB, so the budget can be easily cut by 6 times.

Disclaimer again

One of my key points was, "You are not a hardware (server) expert". Mounting something like this production-ready is not an easy task. Requires effort, time and a lot of knowledge. Clustering usually doesn't work the first time, you use more than one Server (and VMs don't count), you need to know about networks and fibre (it's not a big deal but...) and lastly, you need to know about AWS S3, his security protocols, data lifecycle and tiering cost.

Feasible? Well, for sure, I did it in my own house as an over-engineering project to keep all my kids/wife/family photos and videos safe without any cost to them. But, to reach the correct smoothness is far from simple, especially when you compete with a "one-button-click" to upload everything to Google Photos or Apple iCloud.

But if you're concerned enough about cyber security, breaches, leaks, and you have the knowledge and passion to do it. I totally recommend it.

PD: By the way, from the 3-2-1, the one is my own Glacier, but that is for another day!