Data is growing.

It’s no secret. For certain scale-out workloads, we’ve left behind the days when 100TB in a single volume was enough space to accommodate a single file system. Files and datasets are getting larger. For instance, think about the sheer amount of data that’s needed to keep something like a photo or video repository running. Or a global GPS data structure. Or an electronic design automation environment designing the latest computer chipset. Or seismic data for analyzing oil and gas locations.


Environments like these require massive amounts of capacity, with billions of files in some cases. Scale-out NAS devices are the best way to approach these use cases because of their flexibility. However, it’s important to be able to scale the existing architecture in a simple and efficient manner.


For a while, storage systems like NetApp® ONTAP® had a single construct to handle these workloads —the flexible volume (NetApp FlexVol® volumes).


FlexVol volumes are great, but…


For most use cases, FlexVol volumes are perfect. They’re large enough (up to 100TB) and can handle enough files (up to 2 billion). For NAS workloads, they can do just about anything. But when you increase the number of metadata operations in a file system, you start to see issues with flexible volumes. The volumes serialize these operations and don’t use all possible CPU threads for the operations. I think of it like a traffic jam due to lane closures; when a lane is closed, everyone has to merge, causing slowdowns.


When all lanes are open, traffic is free to move normally and concurrently.


Also, because a FlexVol volume is tied directly to a physical aggregate and node, NAS operations are also tied to that single aggregate or node. If you have a 10-node cluster, each with multiple aggregates, you might not be getting the most bang for your buck.


That’s where NetApp FlexGroup comes in.


FlexGroup is designed to solve multiple issues in large-scale NAS workloads.


  • Capacity. Scales to multiple petabytes
  • High file counts. Hundreds of billions of files
  • Performance. Parallelized operations in NAS workloads, across CPUs, nodes, aggregates, and constituent member FlexVol volumes
  • Simplicity of deployment. Simple-to-use GUI in System Manager allows fast provisioning of massive capacity
  • Load balancing. Use all of your cluster resources for a single namespace

With FlexGroup volumes, NAS workloads can take advantage of every resource available in a cluster. Even with a single node cluster, a FlexGroup volume can balance workloads across multiple FlexVol constituents and aggregates.


How does a FlexGroup volume work at a high level?


FlexGroup volumes essentially take the already awesome concept of a FlexVol volume and enhance it by stitching together multiple FlexVol member constituents into a single namespace that acts like a single FlexVol volume to clients and storage administrators.


A FlexGroup volume would look roughly like this from an ONTAP perspective:


Files are not striped, but instead are placed systematically into individual FlexVol member volumes that work together under a single access point. This concept is similar in function to a multiple FlexVol volume configuration, where volumes are junctioned together to simulate a large bucket.


However, multiple FlexVol volume configurations add complexity via junctions, export policies, and manual decisions for volume placement across cluster nodes. It’s also necessary to redesign applications to point to a file system structure that is defined by the storage rather than by the application.


To a NAS client, a FlexGroup volume would look like a single bucket of storage.


When a client creates a file in a FlexGroup volume, ONTAP decides which member FlexVol volume is the best possible container for that write. This decision is based on a number of factors such as capacity across members, throughput, last accessed, and so on — basically doing all the hard work for you. The idea is to keep the members as balanced as possible without hurting performance predictability at all, and in fact increasing performance in some workloads.


The creates can arrive on any node in the cluster. When the request arrives to the cluster, if ONTAP chooses a member volume that’s different than where the request arrived, a hardlink is created within ONTAP (remote or local, depending on the request). The create is then passed on to the designated member volume. All of this is transparent to clients.


Reads and writes after a file is created operate much like they already do in ONTAP FlexVol volumes. The system tells the client where the file is located and points the client to that particular member volume. Therefore you would see much better gains with initial file ingest versus reads and writes after the files have already been placed.


Why is this better?


When NAS operations can be allocated across multiple FlexVol volumes, the issue of serialization in the system doesn’t arise. Instead, the workload is spread across multiple file systems (FlexVol volumes) joined together (the FlexGroup volume). And unlike NetApp Infinite Volumes, there is no concept of a single FlexVol volume to handle metadata operations — every member volume in a FlexGroup volume is eligible to process metadata operations. As a result, FlexGroup volumes perform better than Infinite Volumes in most cases.


What are the best workloads for a NetApp FlexGroup volume?


A FlexGroup volume works best with workloads that are heavy on ingest (a high level of new data creation), heavily concurrent, and evenly distributed among subdirectories.


For example:

  • Electronic design automation
  • Log file repositories
  • Software build/test environments, such as Git
  • Seismic and oil and gas data
  • Media asset or HIPAA archives
  • File-streaming workflows
  • Unstructured NAS data, such as home directories
  • Big data (Hadoop with the NetApp NFS connector)


For information about best practices and workloads on a FlexGroup volume, check out TR-4571: FlexGroup Best Practices and Implementation Guide.


What is the potential performance boost?


In EDA workload tests, we have seen between 2x and 6x performance gains for NetApp FlexGroup volumes versus FlexVol volumes. In addition, performance can be scaled along with capacity as new cluster nodes are added to a cluster. Although the gains are not entirely 1:1, they are large enough to make performance scaling in ONTAP a reality in large-scale file system deployments.


The following graphs compare an AFF8080 4-node cluster and an A700 2-node cluster. The test done was a GIT compile of the gcc library. The metric used was “time to completion.” The completion times show that more nodes = better performance with this particular workload due to the increase in available resources.


Blue line: Single FlexVol volume


Orange line: Single FlexGroup volume, all local to a single node


Green line: Single FlexGroup volume, spanning all nodes


In addition to enhanced completion times for workload tests, we’re also seeing higher IOPS and throughput in a NetApp FlexGroup volume versus a single FlexVol volume. In these same tests, the following graphs show the results for the A700.


FlexVol volume versus FlexGroup volume — maximum throughput trends under increasing workload.


FlexVol volume versus FlexGroup volume — maximum throughput trends under increasing workload, detailed.


FlexVol volume versus FlexGroup volume — maximum average total IOPS.


Where can I find out more about NetApp FlexGroup volumes?


NetApp FlexGroup volumes are available in ONTAP 9.1 and later. For technical reports, best practices, and information about what is and is not supported, check out these resources:


ICYMI: NetApp Took the Top Spot in Scale-out NAS

Last week, ONTAP completed a clean sweep of IT Brand Pulse’s 2017 Networking and Scale-out Storage Brand Leaders Awards in the Scale-out File Storage categories.

Email us at to find out how FlexGroup volumes are helping to further enhance our industry-leading scale-out storage message

Justin Parisi

Justin Parisi is an ONTAP veteran, with over 10 years of experience at NetApp. He is co-host of the NetApp Tech ONTAP Podcast, Technical Advisor with the NetApp A-Team and works as a Technical Marketing Engineer at NetApp, focusing on FlexGroup volumes and all things NAS-related.