S3 Ltd

De-Duplication - Don't be Afraid

11/05/09 | in: De-Duplication

I was chatting to a potential customer the other day, looking at the pros and cons of de-duplication methods and it’s clear to me that most people still view de-duplication with a bit of trepidation.

While the mechanics of de-duplication are complex, it’s important to understand that de-duplication should be considered as just another process running on the storage system. We trust the storage system to write the data to disk as ones or zeros, that’s rocket science, so surely we can trust the system to get the de-duplication right.

10 years ago the majority of customers would only trust a full data backup regime. It took some time for them to capitalize on the benefits of incremental backups and now synthetic full backups (creating a full data set from the original full backup while layering on the changes contained within the incrementals) are common place. De-duplication capitalizes on this idea but takes it much further.

Products are either inline, using a smart file system to store the data or they are post process, de-duplicating the data once it has been written to disk. Both methods have different benefits and most vendors will tell you theirs is the best but the real decision should be based on your objectives, environment and budget.

What impact does this have on our DR environment?

Disk based backup devices always give a good boost to restore performance and with some careful consideration the DR process can be much easier with greatly improved flexibility.

In short, don’t get too wrapped up in the technology just concentrate on the benefits the new technologies can bring and embrace them for what they are – progress.