new backup architecture snapdiff v3

The challenges of NAS backup

With the explosive growth of data, backing up NAS systems is increasingly challenging and complex for enterprises around the world.

 

Historically, NAS data was backed up and restored with the Network Data Management Protocol (NDMP).  However, NDMP architecture is inherently inefficient and outdated.

 

With NDMP, backups are typically performed as periodic full backups with intervening incremental backups. To restore backed up data to a specific point in time, NDMP needs to restore the last full backup before the restore point and then apply data from all the incremental backups between the last full backup and the restore point. As the number of incremental backups increases, the ability to restore becomes increasingly challenging, complex, and time consuming. As a result, NDMP backup operations require regular full backups, which increase the customer’s backup window. These inherent inefficiencies make NDMP difficult to scale to meet the backup needs of large enterprises.

 

NDMP was originally designed to work with tape-based backup media. Tape systems are labor intensive to manage and difficult to scale. Many enterprises are adopting cloud storage to meet their backup needs. Instead of maintaining dedicated tape subsystems or purpose-built backup appliances, customers are moving to general-purpose public cloud object storage and private cloud infrastructure as the technology of choice for storing their backups. Object storage offers ease of scaling to any capacity and ease of access from any location, as well as the flexibility of a “pay-as-you-grow” model.

A new backup architecture

To help customers solve the NAS backup problem and adopt the latest technology, NetApp is introducing a new SnapDiff v3 backup architecture to efficiently back up NAS data to both on-premises and cloud object storage across the data fabric. SnapDiff v3 architecture is highly efficient and cloud friendly for backing up NAS systems.

 

NetApp® ONTAP® customers have long used NetApp Snapshot™ and SnapMirror® technologies to create efficient Snapshot copies to transfer between ONTAP systems. The new SnapDiff v3 architecture leverages the same unique underlying ONTAP capability to efficiently identify changed data between ONTAP Snapshot copies and then transfer it to a backup target.

 

Two key enabling technologies make up the SnapDiff v3 architecture:

  • SnapDiff, a differencing engine that generates a list of changed files between two Snapshot copies.
  • SnapMirror Cloud, an extension of SnapMirror that transfers Snapshot copies from an ONTAP system to an object storage target.

Together, SnapDiff and SnapMirror Cloud technologies optimize the backup process by copying only the changed data between Snapshot copies of an ONTAP volume to a destination object storage, such as a cloud.

 

This architecture offers the following benefits:

  • Forever incremental backup. After the initial full backup, no more full backups are required.
  • Block-level incremental. If one block of a file is changed, only that block is backed up.
  • Storage efficiency preservation. Deduplication and compression in ONTAP are preserved so that backups are transferred and stored most efficiently.
  • Granular file recovery. Files can be recovered individually from object storage.
  • Online backup data access. Backup data in the object storage can be accessed directly for analytics and other use cases.

RESTful APIs for these NetApp technology components (collectively, the SnapDiff v3 APIs) are made available to third-party backup partners to incorporate into their backup solutions.

SnapDiff v3 backup architecture
SnapDiff v3 backup architecture.

Backup workflow

The following is a typical backup workflow using SnapDiff v3 architecture:

  1. Identify and configure an object storage (for example, a bucket created in cloud storage or NetApp StorageGRID®) where backup data will be sent.
  2. Create a SnapMirror policy for the object storage.
  3. Select an ONTAP volume to be protected, then create a SnapMirror relationship between the ONTAP volume and the object storage target by using the SnapMirror policy.
  4. Start the data transfer—an initial full baseline backup followed by incremental transfers.
  5. Run SnapDiff along with each transfer to generate a list of files that have changed since the previous backup and update the backup catalog.

Restore workflow 

Backup data in object storage can be restored back to ONTAP or accessed directly in a number of different ways. The following is a workflow to restore individual files using SnapDiff v3 architecture:

  1. Select a list of files from the previous SnapDiff output to be restored.
  2. Start the data transfer from the object storage to an ONTAP volume to restore those individual files. 

Summary

SnapDiff v3 architecture addresses the key challenges that companies face with NDMP backups. With SnapDiff v3, ONTAP users can improve backup operations and reduce backup windows by using  a combination of SnapDiff and SnapMirror Cloud to efficiently identify and transfer changed data to object storage on the premises and in the cloud.

 

NetApp is working with selected partners to develop solutions based on the new backup architecture. For more information about the SnapDiff v3 API program and to participate in the SnapDiff v3 API Technology Preview, please contact your NetApp representative.

 

Ling Zheng

Ling Zheng is a Technical Director at NetApp. In his current role, he is responsible for driving the data protection strategy and innovation including integration with NetApp data protection partners. Previously he led the development of a number of storage technologies including file system, virtualization, storage efficiency. At the start of his NetApp career, he was the original technical lead responsible for the creation and delivery of NetApp Deduplication feature in Data ONTAP.

Add comment