By Sarah Burnett, senior analyst

(11.17.09) As we approach the end of the decade, the data warehousing market stands out as one that has become much more competitive and dynamic than before thanks to developments such as the low-cost data warehouse appliance.

The trend for analytical databases has further increased choice and performance options. These and other more recent innovations are set to keep the data warehousing market dynamic for the foreseeable future.

Data warehouses are not the easiest technology to deploy. Many organizations would attest to experiencing difficulties in controlling the cost, risk, and complexity of the data warehousing process while attempting to realize or maintain the significant business value it can deliver.

The need to reduce costs of data warehouses and simplify their deployment gave rise to a new breed of appliances in the early noughties. These combine all the elements of a data warehouse (database, server, and storage) into a single pre-configured package of commodity hardware with very fast connectivity.

The different components of the appliance are optimized to make the most of their capabilities, making these dedicated solutions very fast. The Massively Parallel Processing (MPP) architecture on commodity servers and the open-source operating system and database deliver low-cost scalable data warehousing capability with entry points as low as $10,000 per terabyte of data.

These types of features have helped propel young data warehouse appliance companies such as Netezza into the ranks of direct competitor to established vendors such as Microsoft and Teradata. In response, a couple of years ago Microsoft acquired appliance start-up Datallegro, and Teradata has added a number of low-cost appliances to its product range.

Another development is the growing number of analytical databases. To Sybase IQ and Sand, the oldest of the bunch, we can add a number of new ones, such as Vertica, Infobright, and ParAccel.

Analytical databases are designed from the ground up for analytical applications and are natively optimized for querying. They are column-based with automatic indexing and reduced read and disk I/O requirements to speed up query performance, something that is needed to process growing volumes of data.

These developments are in addition to the various waves of consolidation in the past decade among BI and data-integration vendors that played a key part in the development of more integrated end-to-end solutions, from data integration through to query and reporting in BI. Examples include IBM's acquisition of Ascential, SAP's takeover of BusinessObjects that had earlier acquired Acta, and Hyprion and Brio being subsumed into Oracle.

The convergence of BI with search technologies and the unifying realms of structured and unstructured data promise to bring more innovation to data warehousing.

Already there is an emerging trend for MapReduce-oriented capabilities in data warehouses (e.g. in Aster Data) to increase the degree of parallel processing in data warehouses and to process repetitive activities in parallel with the potential to process non-tabular data.

As such, the next decade promises to be another dynamic one for data warehousing. ENS