Oil and Gas companies are seeking ways to improve business efficiency while facing a number of challenges including the volatile pricing of oil, evolving energy policies, competition from new sources of energy, and ongoing operational management costs and inefficiencies. Those companies wanting to improve their efficiency must apply new technologies and processes that will capture and transform data into actionable insight, in order to optimize exploration, drilling, production, refining and delivery. With the right technology and solutions, you can move beyond traditional real-time monitoring to real-time prediction and agile responses.
Explosion of the data being collected
Big Data and analytics may be new to some industries, but the oil and gas industry has long dealt with large quantities of data to make technical and business decisions. In their quest to learn what lies below the surface and how to bring it out, energy companies have, for many years, invested in seismic software, visualization tools and other digital technologies. The amount of data generated by oil and gas operations is now starting to explode as real-time information from sensors is being collected at a rate of 4 milliseconds.
The speed of this data gathering is pushing the size of oil and gas Big Data into the exabyte range and larger as companies handle more data sets. The rise of pervasive computing devices-afford-able sensors that collect and transmit data-as well as new analytic tools and advanced storage capabilities are opening more possibilities every year. Oil producers can capture more detailed data in real time at lower costs and from previously inaccessible areas, to improve oil-field and plant performance. However, data is not coming only from exploration, today, oil and gas companies analyse data from a variety of sources:
- Data from machinery sensors during exploration, drilling, production, refining and delivery
- Machine-generated data (IT system logs, security systems, sensors…)
- Enterprise data from operational systems
- Social Media
- Historical oil & gas exploration, delivery, and pricing data
Low latency storage for their Splunk analytics
Recent surveys show that companies with better analytic capabilities were twice as likely to be in the top quartile of financial performance in their industry and five times more likely to make decisions faster competitors.
This is the reason why a leading Italian oil and gas industry contractor, decided to invest in a brand new infrastructure based on NetApp EF-Series EF560 storage, in order to fulfill their requirement for a low latency storage solution for their Splunk analytics applications, allowing the organization to run faster analysis of the company-wide machine-generated data by a factor of 5 versus traditional commodity servers with internal drives. This translates into better efficiencies, shorter decision making path and costs optimization.
Splunk is operational intelligence software that enables customers to monitor, report, and analyze live streaming and historical machine-generated data. Splunk helps users distil, sift, and understand this machine data to improve service levels, reduce IT operations costs, mitigate security risks, enable compliance, and create new product and service offerings. As the use of Splunk for operational intelligence grows from a pilot program to full deployment in your organization, its operational integrity becomes critical. Splunk deserves a storage infrastructure that will make sure of optimal and consistent performance at minimal maintenance and expense. The E-Series storage system provides improved performance, data availability, scalability, data protection, and single-interface storage management compared to Splunk workloads running on commodity servers with internal drives.
The EF560 all-flash storage solution combines robust, full-featured storage management software, a bullet-proof array chassis, and the most recent solid-state disk innovations to provide superior technological and business value. The NetApp® EF560 and E-Series utilize the same chassis, which is used in thousands of installations that demand high-performance, dense, cost-effective storage. Together, these storage systems have a proven record of five nines reliability across millions of systems deployed.
For many operations, the most compelling reason to power your Splunk environment with NetApp storage is the performance advantage you will realize versus commodity servers with internal drives. Recent testing closely simulating real-world Splunk indexing and data searches showed conclusively that operations have much to gain from this storage approach. While indexing performance was similar between NetApp and internal drives, searching was significantly faster, on average 69% faster. Stream searches were more than twice as fast. And this is on average; recent NetApp installations have seen 12x search run-time improvements.
Key Benefits of E-Series vs Commodity servers with internal drives in Splunk environments
- Increase search performance by 69% versus commodity servers with internal disks*
- Improve reliability with enterprise storage building blocks
- Optimize performance and capacity buckets for Splunk’s hot, warm, cold, and frozen data tiers
- Realize better performance than internal disk drives during data rebuilds
- Reduce storage requirements and maintain availability with fewer copies of data
- Scale compute and storage independently to better match application workload
- Enjoy single-interface management across storage environment
- Encrypt data at rest
Learn more on what Function1 has to say on NetApp E-Series for Splunk Enterprise: Splunk Enterprise, as deployed on NetApp’s E-Series Storage Systems, represents a cost-effective solution for indexing, collecting, monitoring, and analyzing machine data for a wide variety of applications. The Splunk workload, which includes indexing, searching, and monitoring, performs exceptionally well in an E-Series storage system envirenvironment. The E-Series storage system provides improved performance, data availability, scalability, data protection, and many other advantages compared to Splunk workloads running with commodity servers with internal drives.
*Results when compared with commodity servers using internal disk drives based upon third-party lab report and NetApp internal testing.
Written by Marco Pozzoni and Christian Lorentz