Hardware Failure: The most stressful thing for a team lead
by Brian Reisdorf, on Nov 11, 2019 9:00:00 AM
I'm here in this nice tranquil place to talk about the thing that stresses out every post production supervisor or post production manager and that is a major hardware failure.
One of those things that we will all encounter and it's awful when it happens. There's a few things that we can do to prepare ourselves and come out on top when
something major goes down.
The first thing is take the time to make a plan which might sound silly.
Take a day or month or day a quarter and go through and understand what can fail in your infrastructure and where your choke points are. Collect that information and have a bit of a plan.
When something happens you know how to act because you are the captain of your production ship. Your editors and your producers are going to come to you when something goes down and if you don't have a plan they're going to know and that can be a problem, which I'll explain in a little bit
The second thing is along the lines of of creating a plan, go through and take a look at your infrastructure. It's good to understand you know where things could go bad that would stop you cold.
If you have a storage system with it's not redundant or you know, if you recover things from archive or LTO or Cloud systems on a regular basis hardware failure can absolutely or a major system failure rather can absolutely be in the form of a major cloud service going down and suddenly you don't have access to a method of organization or data that you normally have access to or even stock footage in some cases. All of that can affect your production time. So have a plan! Have a backup and understand what those elements are that can go down that can cause a problem.
If you have hardware that can potentially cause a problem absolutely spend about an hour to get the model numbers, model names and put it on a piece of paper. Get the support numbers for the companies that support it. Get it on a piece of paper so that you can go straight to that or you can email it out to the people that can.
Just having that information at hand is awesome. Even if an IT team is supporting you, you are probably going to be the driver of those fixes so having that information to hand them outright is going to move the needle forward a lot faster.
The third thing is make sure you communicate your plan as it happens. We work with editors and producers and all kinds of tech-savvy people who work on deadlines. They got things to do and if they're not hearing about how things are going, they tend to try to solve things on their own or try to and that can involve external hard drives, data duplication in ways that you weren't expecting, you know files being transferred in securely, missing files, there's only new stuff that are offline if something happens to storage.
There's a lot of things that can go sideways. So make sure you communicate make sure everybody on your team knows what's going on so they know what to expect. Otherwise, they're going to do something themselves and that can be a bigger problem than the actual hardware failure.
So keep that in mind!
Definitely make a plan. Get ahead of the game on this thing. It is one of those things that you know, we hope that we'll never have to utilize. But, if we do, we're happy that we spent the time to do it ahead of time.