OMG - What have I done?

OMG - What have I done?

I break my email server and I lost all my email! Where email put "database" or "webpage" or "queues", the point is, do Backups and do them using your head, not just some simple policy from the internet.

First, classify your data, you can do it by answering some questions:

  • How much critical is my information?
  • What happens if I lost it?
  • What happens if someone stole it?
  • How much time can I survive/keep working without access to it?
  • How much is the opportunity cost during that time?
  • How much time can I lost in case of total failure?

These questions will help you to reach the answer to two critical values RTO and RPO. RTO = Recovery Time Objective, RPO = Recovery Point Objective. But let's talk in "normal language", RTO is basically how much time can pass between Failure and complete recovery. And RPO is how much information did you lost after complete the recovery.

And yes, many bosses will say, instantaneously both! And yes, despite what you think, is possible but "It ain't cheap bro..." and let me tell you something, the cost grows exponentially the closer to zero you get. And if you think about it, it makes sense. Recovery in zero time and with zero lost means that you have ALL your information and systems (at least) duplicated and perfectly synchronized.

But you'll understand even better with an example, if the stock market of New York fails for just one single minute, around $210M[ref] won't be exchanged, so someone will pay the bill and you don't want to be that guy. On the other hand, if you lost the buying list for your normal grocery shop and need to use the one from the last month, probably you'll be pretty much OK. Maybe your partner will be a bit mad because they expected something but that's all.

As you can see, there are many parameters to be taken into consideration, the speed of change, the volatility of the data (restore the stocks from 1 year ago is pretty much useless) but there is one important, I'd even say critical, parameters, the reproducibility of the data.

If you normally watch Netflix and you spent a few hours choosing the videos to be downloaded for your flight and you factory-reset your phone by error, you'll be mad, but you can always recover all the information with enought time, but, imagine that yesterday you took the most beautiful photo of your just-born baby and the same happend. No matter what you do, you cannot restore that photo.

That factor, many people don't consider it, and that's why emails are important to take into account when choosing what stuff requires backup, your personal photos deserve a proper backup. And again, proper means the optimum election for the type of data. You don't need a full backup of your phone, only the pictures and to have a duplicated copy outside of your device, you don't care about RTO or RPO, but you care about the data itself.

In business, in your company, in your team, in your projects, the same can be applied. Some data doesn't care about the RTO meanwhile the RPO can be fullfilled.

But, it doesn't matter, you'll see tutorials, guidelines from AWS, your boss will insists, the CTO from the corner company, internal recomendations, you'll lose data. And it's unaboidable, the perfect answer to my first questions doesnt exists, because not everyone answer equally. What is critical for you, is not for <insert someone who takes decisions>, until they suffer it in their bones.

So my personal recomendation, find time to always have some backup, maybe it's not perfect but some, and stay prepared to use it, do some recoveries to verify that your RTO and RPO are correct. The day that something happen, and it will happen, you use that email about "no, don't dedicate time for the backup, later, now we need to <whatever>" and come with, "luckily... we did some backup and you're not that much screwed!" and after that, you'll get your budget to do it properly.

Except if you work for a bank, they take this very very seriously and you'll struggle to fullfil all the requirements and also pay the bill. For ther rest, please, take my advice for the sake and future of your daughter's graduation day.