If you're in customer service, sales management, financial analytics or software development, having to contend with poor application performance isn't just annoying - it's a serious inhibitor to productivity. Ironically, despite the enormous processing speed and advanced capabilities of robust application server platforms, often times these systems have to wait to process and deliver data from underlying disk-based storage. And this leaves servers with disappointingly low utilization rates, a correspondingly low return on investment, and can have a ripple effect on key segments of an organization's workforce sitting idle while waiting for applications to respond.

Enterprises are starting to learn the hard way that inefficient application servers can lead to unnecessary application wait times, killing productivity for business processes reliant on application results.

To boost utilization, IT administrators often deploy virtualization strategies that help consolidate under-utilized platforms. However, while this approach can reduce hardware expenditures, it does not solve the underlying problem of optimizing server CPU utilization rates that rely on fast access to storage.

Top Symptoms of Storage Bottlenecks

The complexities of modern data centers can make identifying any bottleneck tricky. Pinpointing storage I/O issues can be even more challenging due to the shared nature of network storage. But there are a few easily recognizable signs that often appear when an underlying storage I/O constraint is present. Recurring symptoms of I/O bottlenecks generally fall into three categories:

Lagging performance and hotspots
Poor utilization
Environmental considerations

Lagging Performance and Hotspots
Application brownouts: When applications cannot get access to data quickly enough, they suffer from brownouts and extended delays which impacts overall productivity.
Long response times / high latency:Application servers and storage systems suffering from long response times and high latency are often indicative of a storage constraint.

Delayed responses and long latency indicate I/O constrained clients.

Disk Hotspots: When frequently requested data resides on a single disk spindle or a small set of spindles, particularly if multiple application servers require access to that data, the disks become heavily used to the point where they become a "hotspot". More specifically, the amount of data being requested from that disk or disk group outweighs the capability of the disks to respond. Disks that frequently become hotspots are more failure-prone.

Storage system hotspots: Similar to disk hotspots, entire storage systems can become hotspots when the data requests outweigh the system's capabilities for data delivery.

Poor Utilization

Low application server CPU utilization: Low application server CPU utilization is often caused by long waits for I/O requests. This can be particularly severe in cases where the application requires frequent and random data requests from conventional storage systems. The delay in receiving I/O responses lowers CPU utilization and overall system efficiency.
High storage system CPU utilization: When storage systems are under heavy loads, CPU utilization is high and often results in the inability to access full system functionality. For example, some systems have difficulty keeping up with snapshot and backup/recovery applications under high CPU loads. Alternatively, they cannot deliver the full visibility to real time statistics, further inhibiting the troubleshooting process.
Inefficient Memory Utilization: Data center managers frequently deploy memory to alleviate performance problems. Often, memory is deployed on individual application servers which falsely assumes that workloads are uniformly balanced. Similarly, deploying the maximum memory configuration for each storage subsystem generally results in poor memory utilization due to the shifting nature of disk and subsystem hotspots.

Environmental Considerations

High amount of concurrent client access: When lots of clients try to simultaneously access a single storage system, that system can get bogged down due to disk contention, extending response times.
Server virtualization / consolidation without storage change: Consolidating servers usually means an increased amount of activity to a single storage system. Implementing server virtualization results in more virtual machines (VMs) accessing the same storage system with an increased chance of contention.
Frequent data migrations or reconfigurations for performance: Excessive data movement, either migrating to a new system or reconfiguring an existing one, are typically symptoms of insufficient performance. Taking a closer look at the reasons prompting the migration or reconfiguration, administrators often realize that much of this work is to deliver a consistent and predictable amount of storage performance.

Granted, some data migrations take place due to capacity constraints, but with the introduction of terabyte drives and clustered file systems, this seems even less likely compared to the need to make active data readily available and accessible.

Solving Bottlenecks with Centralized Storage Caching

Scalable caching appliances address this "application waiting game" problem by revving up idle, data-hungry CPUs through centralized caching. By caching frequently accessed data in memory as opposed to slower, mechanical disk, data from high-speed, high-capacity, memory achieves vastly superior I/O access times. This produces up to 10-50X I/O performance improvements, increased CPU utilization rates, and accelerated applications.

During peak loads, scalable caching appliances deliver instant data access to thousands of application servers, eliminating application brownouts and sustaining service level agreements.

Scalable caching appliances deploy seamlessly in the data center, complementing existing network attached storage configurations. No change to the application servers or underlying storage is required as the appliances simply attach to the network as a shared resource, enabling acceleration for multiple applications simultaneously.

Basic deployment of scalable caching appliances

Advantages of a Network-centric Approach

Understandably there are several methods data center managers use to increase storage performance. The most common is to increase the number of disk spindles. This approach has a perceived low cost in that individual disk spindles are relatively inexpensive. But over time, the cost of deploying excess disk spindles for performance with low capacity-utilization rates adds up in management, power, space and cooling.

The other common approach is to add cache memory on individual application servers or subsystems. But this is akin to the days of jamming more drives into individual servers before the arrival of network storage. Deploying memory resources on individual end-nodes (such as servers and storage systems) focuses those resources on single systems as opposed to making them available to the data center as a whole.

New choices with scalable caching appliances deployed in the network deliver several advantages while improving storage performance.

Scale and Manage Performance Independently from Storage Systems

Historically, storage systems have combined features for I/O persistence with I/O performance. Given the amount of work that must take place on the persistence front such as snapshots, backup, replication, mirroring, provisioning, recovery and more, it can be overwhelming to a single system to then demand the utmost in I/O performance.

Separating these functions and allowing each to scale independently brings tremendous flexibility to data center and storage managers. No longer do systems need to be prematurely retired due to performance limitations. And the availability of large-capacity drives means that the utilization of those spindles can be pushed higher given that performance is maintained with the presence of a caching appliance.

Remove Storage Hotspots Across Multiple Systems

Storage hotspots can be unpredictable and move between different storage systems. This is exacerbated by large enterprise application deployments that may have different data stores across different systems.

A centrally located caching appliance that accelerates multiple storage systems can manage and prevent hotspots across all of those systems. This is not possible when caching resources are deployed on a single end-node.

Multiple Caches Can Support a Single Data Source

Many data sources that focus on real time data presentation are more easily maintained as a single instance. By deploying caching in the network, multiple caches can support a single data source. This removes the needs for unnecessary copies and excess infrastructure while [providing performance in more than one location] - an alternative phrasing could be "optimizing performance across multiple locations".

For example, a customer might deploy one caching appliance in a primary data center, then another caching appliance in a secondary data center that points back to the source data set. The secondary site still enjoys the same performance as the primary data center when the cache is fulfilling cache-hits.

Shared Caching Resource Across Applications and Clients

Deploying a caching resource can add significant flexibility for data centers that support constantly growing and increasingly complex applications. As workloads shift and new applications are deployed, the caching resource can be assigned accordingly. Further, a network-centric caching approach can serve thousands of concurrent clients, as well as provide options for accelerated data access through the appliance, or direct access to the storage if required.

Dynamically Adjust Cached content

Decoupling caching resources from storage systems sets the stage for dynamically cached content across applications and storage systems. For example, an application that moves from one data set to another will automatically keep the most actively used data to remain in cache. This is far more efficient than statically assigned cache resources on a per-storage system basis.

Network-based caching appliances dynamically cache currently accessed content

Performance Visibility Across Applications, Clients, Storage

Diagnosing and troubleshooting system performance across hundreds or thousands of application servers and dozens of NFS file servers is a big deal.

That's because NAS devices present statistics on a per device basis. And application servers present statistics on a per server basis. Consequently, it's difficult to create an aggregated view when evaluating these end points.

Scalable caching appliances help customers see the forest as well as the trees. By placing a centralized resource in the network that can serve data from any storage system to any application server, the caching appliance is uniquely positioned to deliver aggregate data center statistics. This provides an invaluable resource to administrators trying to understand, manage, and improve their overall system performance.

Centralized caching provides complete visibility to I/O activity and performance

Conclusion

Several forces continue to put pressure on data center storage systems. The increase in server virtualization means more virtual machines are accessing a single storage system. And the continued onslaught of new applications increases an already heavy demand on storage systems to deliver fast and predictable I/O responses.

Data center and storage managers should keep an eye out for the I/O bottleneck symptoms outlined in this article. From there, it is important to pick an architecture that delivers the most cost advantages and operational flexibility to improve performance. This includes:

Locating performance resources in the network to serve accelerated I/O to any application on any client
Retaining the existing persistent storage investment to maximize returns
Efficiently deploying memory resources that can scale with the active data set across multiple storage systems
Adopting approaches that cache data dynamically and provide visibility to multiple clients and storage systems to understand, manage, and improve performance. ENS