Chevron has two U.S. data centers - one in Houston, Texas, and the other 1600 miles away in San Ramon, California. These data centers are the centralized service hubs for the business units located in their respective areas.

IT is looked after by a Chevron unit known as Information Technology Company (ITC). ITC provides IT services for the shared infrastructure for the corporation - which includes the desktop, file storage, network, facilities, application hosting, productivity tools (e.g. PDA's, Blackberry). Its user base is 50,000-plus Chevron employees in 186 countries. ITC is responsible for around 4,000 Compaq, HP, and Sun Solaris servers worldwide running mainly UNIX, Windows, and Linux - 50,000 HP desktops and IBM laptops all running on a Cisco-based network. Each data center has its own metropolitan area network (MAN).

"In Chevron, IT is not allowed to drive the business," says Tony Jurgens, storage technologist for ITC. "Instead, our customers - the various business units within the company - give us their objectives and we have to translate our IT actions into the realization of these business requirements."

In terms of storage, each data center has a mix of SAN, DAS and NAS residing within them from a range of vendors such as NetApp, Hitachi and EMC. The main SAN elements are:

• HDS 9960, 9980 and 9570
• Brocade 3800, 2800 and 2900
• Emulex HBA
  
  

Over time, however, storage and backup costs continued to rise. ITC diagnosed the problem-providing high end services for all data when much of that data was stagnant i.e. unstructured data existing in spreadsheets, PowerPoint presentations, word docs, jpegs, etc., that has not been accessed in over a year. Out of 32TB of data in the Houston/San Ramon environment, a total of 16TB turned out to be dormant.

"Storage and backup were way too expensive and time consuming," says Jurgens. "We had about 50 percent stagnant data and that was really slowing us down. Plus we were adding an average of 2 TB of data per day to our systems worldwide."

Having so much data in its production systems caused problems. In terms of recovery, for example, it took about 8 to10 hours to recover 200 to 250 GB's of information. Further, the company didn't have any formal disaster recovery processes in place.

ITC also recorded storage growth rates of 50 to 100 percent annually - well above their projections of 30 percent annual growth. ITC had to request additional capital and explain the higher than forecasted storage growth numbers. As a result, management demanded better storage management and reduced IT costs.

Jurgens analyzed storage more closely. He broke it down to three key areas of storage management:

Capacity Planning to plan and forecast for added capacity
Information protection

Data Management - to decide what data should reside in which media and be available how rapidly.

"We needed one strategy to unify all three areas," says Jurgens. "That's why we opted for tiered storage."

The company decided to engage in a strategy to move dormant information to lower cost storage. In this Tiered Storage environment, near line storage hardware is fed stagnant data. The data mover migrates the data automatically from primary to secondary storage.

To implement tiers of service for its user community, ITC investigated newer technologies such as disk-to-disk, snapshot, etc. in order to integrate into the existing backup services. For now, the tiers at Chevron, as laid out by Jurgens, are as follows:

Tier 1 - HDS TagmaStor to host the databases, data warehouses and for online transactions processing (OLTP)

Tier 2 - HDS 9200/9570 and some NAS for active data such as file servers, seismic files, Excel spreadsheets and email

Tier 3 - ATA disk and tape for static data such as reports, images and nearline archiving

Tier 4 - offline tape for inactive data required for legal retention.

The data mover employed to shift inactive data from Tier 2 to Tier 3 is Legato Disk Extender 2000. The moved data will not be retrieved from secondary storage unless changed. Thus when a user accesses an inactive file to look at the contents, it is read from the lower cost storage device instead of being recalled to primary storage.

NetApp R200 is being used in Tier 3 for nearline storage. This storage is harnessed for a wide variety of applications. First and for most, it addresses Chevron's growing volume of unstructured data that hasn't been accessed in over a year. ITC projections indicate that this project will have an Initial Rate of Return (IRR) of 66% over 3 years.

The NetApp R200 has several functions:

• Oracle RMAN (Recovery Manager) to make disk backups of the Oracle database. Oracle RMAN offers the ability to back up the database to any media - whether it is tape or disk. The focus of the Oracle RMAN to disk backup project is to have the RMAN tool back up the databases to disk. Once this is on disk, Legato Networker backs up these files as normal flat files with no concern to database intervention.
  
  
• Application Hosting Disaster Recovery for mission critical apps. This involves backing up databases to disk and then snap mirroring to a reciprocating data center (between Houston and San Ramon and vice versa). This project uses the NetApp R200 as the storage device for mission critical backups. The Snap Mirror software that comes with the NetApp copies the data to the reciprocating data center.
  
  
• Exchange Email documents. This project stored email documents on the R200- again this addressed the cost of the storage. ITC needs approximately 5-6 TB between Houston and San Ramon. This project is still in the proto-type phase and is still evaluating various software components.
  
  

"By implementing a tiered architecture, ITC has achieved a 49% reduction in the storage rate charged to its business units," says Jurgens.

Prototyping hurdles

ITC conducted extensive prototyping on this project. One hurdle it had to overcome was that Legato Disk Extender 2000's software failback didn't work in an active-active Windows cluster. As a result, the supplier had to rewrite the DLL with help from Microsoft. From a migration standpoint, however, the software works as advertised. The result of this is that it makes the archiving process transparent to the end user i.e. archived documents look like documents in the production system. According to tests, the response time penalty for data retrieval is negligible.

ITC also had to do several equipment migrations to improve performance and reliability. For example, it switched from HDS 9200 to HDS 9570. The goal was to improve the reliability of the aging Tier Two storage. This project was complete in July 2004. Since then ITC has seen no unplanned outages. Another migration was from a Troika HBA to an Emulex HBA.

"We switched as our old vendor was purchased by JNI and the hardware was no longer supported," says Jurgens.

It terms of storage management software, Chevron has opted for HP OpenView Storage Area Management (OVSAM). The company is leveraging its ability to provide capacity planning reports, trending analysis, and asset reporting. This allows ITC to provide more accurate budget forecasting. Currently, ITC is upgrading to OVSAM 3.2 to address issues with previous versions per a recommendation from HP.

Another project dealt with backups for NAS. This involved the implementation of LAN-less backup for the Net App F9xx and R2xx devices. This has already been implemented for the F9xx devices with an overall improvement in backup speeds. Using LTO1 tape drives, ITC has achieved 60GB/hour in backup and recovery speeds. When LTO2 is implemented, it is projected to double the throughput.

ITC plans to continue the upgrade of its storage environment by implementing disk-to-disk backup, initially for email. Jurgens explains the vital necessity of this move.

"It took us three days to recover an email system using tape backups," he says. "We need a more immediate method of recovery."

In particular, the company is using Zantas to archive Exchange documents to lower-cost storage.

Moving forward

ITC is reviewing the current backup infrastructure design which was based on 5-year old technology. The organization needs to first understand the impact of integration of new technologies such as snapshot technology, disk to disk backups, electronic vaulting. ITC is also reviewing the potential of putting its Oracle databases on Tier 2 NAS versus Tier 1/2 SAN.

Another important project on the immediate horizon is aimed at Windows App Servers using centralized storage. Analysts are reviewing the potential impact of server virtualization and how this will integrate with the overall storage infrastructure.

Based on discussions with Oracle, hierarchical storage management may be built into the Oracle product set in the next year or so. Chevron is looking at how it can leverage this technology.