NetApp Active IQ provides customers and partners with actionable intelligence on their NetApp environment via a dashboard that summarizes performance, availability, capacity forecasting, health summary, case histories, upgrade recommendations, and more. Every week the system generates about 100TB of data and 225 million files―and it is growing!

 

As the team responsible for IT operations and fulfilling storage requirements for these rapidly growing, almost insatiable data sets, we struggled on two fronts. As the Active IQ data lake grew, IT constantly teetered on exceeding SLA targets set with the internal business team for application processing. It was nerve wracking.

 

Moreover, we continually hit capacity limitations of the assigned NFS volumes. Every 2-3 weeks, new volumes had to be established with redirects. This drove 24.2 hours of change activity each month as the Command Center dealt with the frequent alerts from exceeded thresholds, the storage team established new volumes, and the application developers updated over 200 servers with the new information. It was a reactive, manual hot mess.

NetApp ONTAP FlexGroup Volumes

To improve the Active IQ data ingestion challenge and it’s growing data lake, we implemented NetApp ONTAP FlexGroup Volumes as it has the capacity to scale up to 20 PB of storage and 400 billion files. The FlexGroup technology allowed us to present a single, scalable storage volume while delivering a 15-20% reduction in overall data processing time from the application side.  We have seen a 2x improvement in input/output operations per second (IOPS) performance, 10% more throughput, and lower total average latency.  Today we are easily meeting SLA targets with ample headroom.

 

By implementing FlexGroup, we have simplified operations and removed the tedious manual activities associated with volume changes that happened every 2-3 weeks to once every two years (based on projected data growth). This is because FlexGroup can span multiple nodes and grow capacity non-disruptively, while providing a single namespace. Today when we run out of space, we can add more nodes/constituent volumes to the same FlexGroup volume(s), transparent to the app.  We also get to leverage all of the efficiencies of ONTAP like deduplication, compaction, and compression.

 

The introduction of FlexGroup has come at the right time as the frequency of volume change activity increases for NetApp’s growing Active IQ data lake. We are pleased with how FlexGroup blends near-infinite capacity with predictable, low-latency performance for our metadata-heavy Active IQ workloads.

 

The NetApp-on-NetApp blog series features advice from subject matter experts from NetApp IT who share their real-world experiences using NetApp’s industry-leading data management solutions to support business goals. Visit www.NetAppIT.com to learn more.

Faisal Salam

Faisal Salam is a Senior Storage Engineer in NetApp’s corporate IT team and is a member of the NetApp Customer-1 team, which acts as the first adopter of NetApp solutions and services. Faisal supports software-defined storage solutions for enterprise data management and has more than 10 years of experience.

Ramesh Singh

Ramesh Singh manages Enterprise Platform & Application Operations for NetApp IT. His team is responsible for the architectural design, development, and deployment of cost effective, sustainable technical and application solutions to meet business requirements. Ramesh has 13 years of IT experience

Arullaldivakar Jayapalan

Arullaldivakar (Arul) Jayapalan is the IT System Architect for NetApp’s AutoSupport and Active IQ Big Data platform. Arul architected and implemented the 140 node Hadoop cluster with the use of Cloudera manager and various Hadoop eco system technologies. He has worked with this Big Data platforms for the past five years.