Data Warehousing has grown popular, so popular that 18 percent of organizations say that their warehouses are doubling in size every year. The rapid growth of data often means that organizations are staying busy tuning their systems to maintain the new volumes of data while keeping the system performant. Data warehousing is important — companies typically use data warehousing as a key component oftheir business intelligence solution.
To keep pace with the exploding amounts of data being managed, many organizations are turning to next-generation software and hardware solutions.
On the hardware side, data warehousing appliances are being sold as specialized hardware solutions specifically tuned for data warehousing. InformationWeek identified some of the many players in the warehousing appliance business. They include IBM, Teradata, DATAllegro, Dataupia, Greenplum, HP, Kognitio, Netezza, ParAccel, Sybase and Vertica. While Oracle isn’t in the hardware business, they’re marketing software for data warehousing that’s tuned to some of the most popular warehousing appliances.
Software is evolving along with hardware. One concept that is gaining popularity is the column-based datastore. Normal relational databases are row-based. Row-based processing is good when many transactions occur and there are many system writes. Column-based processing is proving itself to be very effective when a limited number of columns of the data is being processed — and this is often the case in queries run against Data Warehouses.
Column-based databases are able to better apply compression to the data than row-based systems. Also, since entire rows of data are not being returned in the result sets, very often column-based queries out-perform row-based systems.
Telecommunications, banking, finance and companies from many other sectors are now using or investigating how column-based computing can help them. For example, the IRS converted their 158 terabyte system from row-based to column-based and found that their query times dropped dramatically.
