At NetApp we use Splunk® Enterprise software to index, collect, monitor and analyze large amounts of real-time data. This technology acts like a great big correlation engine digesting machine data for a wide variety of applications inside NetApp.

Flexible, Scalable Infrastructure

With Splunk churning so much data, we needed fast, high performing storage. We selected NetApp E-Series storage system because it is resilient, built for high performance, and provides flexible storage configurations. It gives us the ability to easily expand storage as our requirements grow.

 

When deploying Splunk, most IT shops follow the product’s general guidelines and run the indexers on physical hardware and directly attach storage to it. Instead, we used an alternative approach: virtual machines (VMs) for compute and E-Series storage (via iSCSI) in our Splunk infrastructure to achieve the best flexibility and scalability.

10X Performance Improvement

When running Splunk in our virtual environment on NetApp E-series, we have achieved a 10x indexing performance improvement and a 3x search performance improvement by:

  • Switching to high performance iSCSI; and
  • Allocating Hot & Warm index buckets on SSD for two weeks of data. After two weeks, the data goes to Cold index buckets on low cost disks.

We currently index one billion events per day without data being queued for indexing. Getting great performance at the lowest possible cost is transformational.

 

We predict that our Splunk indexer data would continue to grow exponentially. Being able to distribute the work of search requests and data indexing using our E-Series technology allows NetApp IT to scale and to be ready for the future.

 

The NetApp-on-NetApp blog series features advice from subject matter experts from NetApp IT who share their real-world experiences using NetApp’s industry-leading data management solutions to support business goals. Want to learn more about the program? Visit www.NetAppIT.com.

Han Jo

Han Jo is a senior IT engineer, specializing in automation and monitoring tools at NetApp.