Skip to main content

White Papers

Data Mining: Extending the Information Warehouse Framework

Abstract

Competitive business pressures and a desire to leverage existing information technology investments have led many firms to explore the benefits of data mining technology. This technology is designed to help businesses discover hidden patterns in their data -- patterns that can help them understand the purchasing behaviour of their key customers, detect likely credit card or insurance fraud, predict probable changes in financial markets, etc...

These pages explore data mining, its potential benefits to users, and IBM's activies in this area. It also explains how data mining activities can be integrated within an existing user environment, including those activities that already make use of data warehousing.


Contents

  1. Abstract
  2. Introduction
  3. The Business Mandate
  4. Potential Industry Application Areas
  5. Data mining: Verfication vs. Discovery
  6. Enabling Technologies
  7. Data Mining and the Information Warehouse framework
  8. The IBM solution
  9. Customer Validation Examples
  10. Summary

Introduction

Many firms have invested heavily in information technology to help them manage their businesses more effectively and gain a competitive edge. Over the last three decades, increasingly large amounts of critical business data have been stored electronically and this volume is expected to continue to grow considerably in the near future. Yet despite this wealth of data, many companies have been unable to fully capitalize on its value. This is because information that is implicit in the data is not easy to discern. For example, a retail store may keep detailed information about the purchases made by each customer every day but still have difficulty pinpointing many subtle buying patterns. Similarly, an insurance firm may keep detailed historical information about claims that have been filed, and still have difficulty isolating some of the likely indicators of fraud.

Fortunately, advances in a field known as data mining are helping customers leverage their data more effectively and obtain insightful information that can give them a competitive edge. Simply put, data mining software enables customers to discover previously undetected facts present in their business-critical data -- data that may consume many gigabytes or terabytes of storage, may reside in files or various DBMS-managed databases, and may be stored on a variety of operating system platforms. Accuracy, efficiency, and an open architecture are important requirements of such data mining software.

This paper explores the business mandate for data mining as well as the relevant technologies that are enabling customers to enjoy success in this area. It also discusses IBM's approach to solving the data mining problem.


The Business Mandate

Firms in a number of industries -- including retail, finance, healthcare and insurance -- routinely maintain enormous amounts of data about the activities and preferences of their customers. Implicit within this data are patterns that reveal the typical behaviors of these consumers -- behaviors that can help firms fine-tune their marketing strategies, reduce their risks, and effectively improve their bottom line.

For example, retailers often wish to know what products typically sell together. Knowing that 72 percent of their customers who buy a certain brand of soda also buy a certain brand of potato chips can help the retailer determine appropriate promotional displays, optimal use of shelf space, and effective sales strategies. As a result of doing this type of affinity analysis, the retailer might decide not to discount the potato chips whenever the soda is on sale, as doing so would needlessly reduce profits.

A somewhat similar requirement is to understand historical buying patterns that occur over a period of time. For example, one such pattern might be that 64 percent of the people who order a certain type of sleeping bag and backpack will order a camping tent some time later. Knowing such information can help a mail-order firm increase sales by narrowing down the scope of a given mailing campaign or perhaps creating more custom mailings that each yield higher success rates.

Many other industries have similar requirements to detect hidden patterns in the data they routinely store to meet other business requirements. The stored data can be of a wide nature, such as oil-drilling data, stock market data, consumer data, etc. The challenge many such firms face is detecting these patterns in a reasonable time frame and at an acceptable cost. This is where recent technical advances -- such as those included in IBM's solution -- can play a critical role.


Potential Industry Application Areas

Although the finance and insurance industries have long recognized the benefits of data mining, this technology can be effectively applied in many areas. In this section we list some examples of application areas where IBM solutions can find unexpected but beneficial results.

Retail/Marketing

  • Identify buying behavior patterns from customers.
  • Find associations among customer demographic characteristics.
  • Predict which customers will respond to mailing.

Banking

  • Detect patterns of fraudulent credit card usage.
  • Identify "loyal" customers.
  • Predict customers that are likely to change their credit card affiliation.
  • Determine credit card spending by customer groups.
  • Find hidden correlations between different financial indicators.
  • Identify stocks trading rules from historical market data.

Insurance and Health Care

  • Claims analysis - determine which medical procedures are claimed together.
  • Predict which customers will buy new policies.
  • Identify behavior patterns of risky customers.
  • Identify fraudulent behavior.

Transportation

  • Determine the distribution schedules among outlets.
  • Analyze loading patterns.

Medicine

  • Characterize patient behavior to predict office visits.
  • Identify successful medical therapies for different illnesses.


Data mining: Verification vs. Discovery

Decision support systems (DSS), executive information systems, and query/report writing tools are used to produce reports about data, usually aggregating it through any number of dimensions. Another use of these tools is to detect trends and patterns in customer data that will help answer some questions about the business. When used in this mode, a query is created to access the records relevant to the question(s) being formulated. After the data is retrieved, it is examined to detect the existence of patterns or other useful information that can be used in answering the original question(s). We call this the verification mode. In this mode, the user of a DSS generates an hypothesis about the data, issues a query against the data and examines the results of the query looking for affirmation or negation of the hypothesis. In the first case, the process ends; in the latter case, a new query is reformulated and the process iterates until the resulting data either verifies the hypothesis or the user decides that the hypothesis is not valid for his data.

Consider the following example. A sales executive has a limited budget to do a mailing campaign for a new product. In order to optimize the use of this money, the marketing executive wants to identify the largest set of people that are the most likely candidates to buy the new product and which can be reached within the budget limitation. To identify these customers and to verify that the customer set has been adequately narrowed to match the available promotional budget, the executive makes a hypothesis about the potential customer set. Issuing a query against the databases that contain historical data about customer purchases and demographic information respectively, the set of customers that have made significant purchases of competitive products can be obtained. Furthermore, to limit the number of customers found to a reasonable number, the executive requests to only get information about those customers that are characterized by having ages between 30 and 45 years, being heads of household with combined incomes between $25,000 and $50,000 and living in some specific zip code regions. If the result of this query returns a number of customers that match the available budget for mailing promotions, the process ends. However, if either significantly more (or less) customers are found than the number that can be reached with the given budget, a new query limiting (or expanding) the set of customer addresses requested must be issued.

In the above example, the hypotheses used in formulating the queries where quite explicit (e.g., incomes between certain amounts.) Even when the hypotheses are implicit, the process of finding useful trends or patterns using queries can be described by the above behavior, as shown in the following example involving a query drill down process.

After a report about company sales shows that the last quarter sales where significantly lower than expected, the financial officer of the company wants to discover what caused this situation. A query is first issued to return the sales figures, by region, for the last quarter. The results of this query show that all the sales are up, except for one particular region. The financial officer begins to suspect that the problem may have occurred in some localized store. To better understand the nature of the problem, another query is issued that will return sales results for all the cities in the offending region. A result showing one city significantly lower that the rest reinforces the officers' suspicion; a result showing that sales were uniformly lower among all cities in this region requires that the initial guess about what caused the problem (i.e., the implicit hypothesis) be modified. New queries continuing to drill down looking for the results by store within an offending city follow the previous query; totally new queries need to be devised if the results of the last query contradict the implicit hypothesis.

Queries, such as those used in the previous two examples, always return records that satisfy the query predicates. Thus, little new information is created in this retrieval process: either the hypothesis is verified or it is negated. The process of information finding is done by the user by successive iterations upon examining the results of query after query and linking the verified and refined hypotheses. This is the essence of a verification model.

Many times, while performing a query, a request is made to compute functions related to the records being inspected during the query (e.g., count the number of records, find the average of a given field of the records, etc.) All these operations result in additional information being returned together with the query. For the purposes of this discussion, these derived facts are not considered.

Notice that, from the user perspective, he/she is discovering facts about the data. The use of queries to extract facts from databases is a common practice. There are other tools that, like query generators, are used in a mode that follows the verification model described above. Examples of these other tools are multidimensional analysis tools and visualization tools. Multidimensional tools make it easier for the user to formulate drill down queries such as those shown in the last example. Visualization tools are used, as their name implies, to present data in a visual manner and to allow the user to easily interact with the data in search of hidden patterns. The user of a visualization tool takes advantage of the human's visual perception capabilities to discern patterns. The three types of tools discussed above, queries, multidimensional analysis and visualization, all have in common that the user is essentially "guiding" the exploration of the data being inspected.

Data mining uses a different model for the creation of information about data. We call this the discovery model. In the next section we will describe methodologies that can sift through the data in search of frequently occurring patterns, can detect trends, produce generalizations about the data, etc. These tools can discover these types of information with very little (or no) guidance from the user. The discovery of these facts is not a consequence of an haphazard event. Quite to the contrary, a well designed data mining tool is one that is architected and built so that the exploration of the data is done in such a way as to yield as large a number of useful facts about the data as possible in the shortest amount of time.

Comparing the process of finding information in a collection of data to that of mining diamonds in a diamond mine, we can say that "verification" is like drilling individual holes in a lode with the expectation of finding diamonds. Finding all (or many) diamonds in this way can be very inefficient. "Discovery", on the other hand, is similar to scooping out all the material in the lode and dumping it on a plain field so that all the glittering stones are thrown up into the open. Diamonds are then separated from the quartz by further inspection. In Data Mining, large amounts of data are inspected, facts are discovered and brought to the attention of the person doing the mining. Unlike diamonds, which are easily distinguishable from quartz, business judgement must be used to separate the useful facts from those which are not. Because this last step does not involve sifting through the raw data, Data Mining is a more efficient mode of finding useful facts about data.


Enabling Technologies

There are a number of data mining methods. One way to classify them is by the function they perform. Another way is to classify them according to the class of applications that they can be used in. In the following section we discuss four classes of mining functions< Associations, Sequential patterns, Classifiers and Clustering> The above categorization is intended to be quite general. There is an extensive body of technology that exists and continues to evolve that can be used to construct data mining functions such as the four mentioned above. In the past, both Classification and Clustering functions have been widely utilized in various forms in Decision Support Systems. To a lesser extent, some Decision Support Systems have provided limited Association functionality. IBM has developed technologies that allow for the implementation of very powerful Association and Sequential Pattern functions (see section Solutions.) As we discuss these various classes of Data Mining functions, we will also give examples of classes of applications for which these functions are useful and mention some of the basic mathematical technologies that have been used to build them.

  • Associations. Given a collection of items and a set of records, each of which contain some number of items from the given collection, an association function is an operation against this set of records which return affinities that exist among the collection of items. These affinities can be expressed by rules such as "72% of all the records that contain items A, B and C also contain items D and E." The specific percentage of occurrences (in this case 72) is called the confidence factor of the rule. Also, in this rule, A,B and C are said to be on an opposite side of the rule to D and E. Associations can involve any number of items on either side of the rule. A typical application that can be built using an association function is Market Basket Analysis. In this application, a retailer will run an association operator over the point of sales transaction log. The transaction log contains, among other information, transaction identifiers and product identifiers. The collection of items mentioned above is, in this example, the set of all product descriptors or SKU's. Typically, this set is of the order of 100,000 or more items. The set of products identifiers listed under the same transaction identifier constitutes a record, as defined above. The output of the association function is, in this case, a list of product affinities. Thus, by invoking an association function, the market basket analysis application can determine affinities such as "20% of the time that a specific brand toaster is sold, customers also buy a set of kitchen gloves and matching cover sets." Another example of the use of associations is in an application that analyzes the claim forms submitted by patients to a medical insurance company. Every claim form contains a set of medical procedures that were performed to the given patient during one visit. By defining the set of items to be the collection of all medical procedures that can be performed on a patient and the records to correspond to each claim form, the application can find, using the association function, relationships among medical procedures that are often performed together.
  • Sequential patterns. In the transaction log discussed above, the identity of the customer that did the purchase is not generally known. If this information exists, an analysis can be made of the collection of related records of the same structure as above (i.e., consisting of a number of items drawn from a given collection of items). The records are related by the identity of the customer that did the repeated purchases. Such a situation is typical of a Direct Mail application. In this case, a catalog merchant has the information, for each customer, of the sets of products that the customer buys in every purchase order. A sequential pattern function will analyze such collections of related records and will detect frequently occurring patterns of products bought over time. A sequential pattern operator could also have been used in one of the examples in the previous section to discover the set of purchases that frequently precede the purchase of a microwave oven. Another example of the use of this function could be in the discovery of a rule that states that 68% of the time when Stock X increased its value by at most 10% over a 5-day trading period and Stock Y increased its value between 10% and 20% during the same period, then the value of Stock Z also increased in a subsequent week. Sequential pattern mining functions are quite powerful and can be used to detect the set of customers associated with some frequent buying patterns. Use of these functions on the set of insurance claims discussed above can lead to the identification of frequently occurring sequences of medical procedures applied to patients. This can help identify good medical practices as well as to potentially detect some medical insurance fraud.
  • Classifiers. Given a set of records, each comprised of a number of attributes, a set of tags (representing classes of records) and an assignment of a tag to each record, a classification function examines the set of tagged records and produces descriptions of the characteristics of records for each of the classes. The class description generated by a classification operator may be explicit (e.g., a set of rules describing each class) or implicit (e.g., a mathematical function which gives the class to which a record belongs to when this record is given as input to this function). These class descriptions can be used to tag new records by determining which class they fall into. The embodiment of the class descriptions is sometimes called a model. Many classification models have been developed. Typical ones are linear regression models, decision tree models, rules based models and neural network models. Decision tree classifiers are examples of explicit classifiers while neural network classifiers are examples of the implicit type.

    An example application for which a classification function is well suited is Credit Card Analysis. A credit card issuing company may have records about its customers, each record containing a number of descriptors, or attributes. For those customers for which their credit history is known, the customer record may be tagged with a GOOD, MEDIUM or POOR tag, meaning that the customer has been placed in the corresponding class of good (medium or poor) credit risk. A classifier can examine these tagged records and produce an explicit (or implicit, depending on the model) description of each of the three classes. An explicit model (such as a decision tree model) may be used if the application wishes to determine a description of each class of customers (for example, such a classifier would produce a description of the set of GOOD customers as those with "incomes over 25,000, age brackets between 45 and 55 and who live in XYZ neighborhood".) An implicit model, such as one built using a Neural Network, can for example be used effectively in an image identification application. In this case, a neural network classification model would be developed using as input a set of image features, or attributes, together with a tag (e.g., VALID, INVALID). The resulting model is a trained neural net which can be used to determine which class a given image belongs to.

    Classification functions have been used extensively in classification applications such as credit risk analysis, portfolio selection, health risk analysis, image and speech recognition, etc.

  • Clustering. Whereas the input to a classification operator is a set of tagged records, the input to a clustering operator is a collection of untagged records. No classes are known at the time the clustering operator is applied; in fact, the goal of a cluster function is to produce a reasonable segmentation of the set of input records according to some criteria. The criteria itself is defined by the clustering tool. Thus, different clustering functions may produce different segmentations of the set of input records. Clustering functions may produce explicit or implicit descriptions of the different segments produced. Examples of applications that can use clustering functions are market segmentation, discovering affinity groups, defect analysis, etc. Many of the mathematical technologies that can be used to build classification functions can also be used to build clustering functions.

Many times we see that data mining operators can be used cooperatively. For example, an association operator can be used to identify groups of products that have high propensity to be purchased together or a sequential pattern function can identify groups of customers that are likely to purchase some item after they have purchased others. These groupings can then be used to drive a classification function which produces a generalized description of products (or customers) in this class. If the marketing manager described in the previous section had been trying to sell a new product such as a microwave oven, he/she could have used a sequential pattern data mining operation to pinpoint significant classes of customers whose buying patterns lead them to purchase microwave ovens. Utilizing these buying patterns, the marketing executive could then use a classification method to characterize those sets of customers whose buying patterns follow the identified ones, except that no purchase of microwave ovens has yet been made. These sets of customers are likely to purchase a microwave in the future and so the marketing campaign is targeted to them.


Data Mining and the Information Warehouse framework

Data mining tools discover useful facts buried in the raw data (thus the term discovery model.) They complement the use of queries, multidimensional analysis and visualization tools to gain a better understanding about data. As such, good facilities to perform queries and data visualization as well as the availability of powerful data mining operators should be part of a well architected Decision Support environment. shows such an architecture.

Much like a regular mining process, which takes raw material as it may exist in a mine and through several steps extracts from the ore valuable metals, data mining comprises three distinct phases or steps< Data Preparation, Mining Operations and Presentation> The process of information discovery can be described as an iteration over the three phases of this process.

Datamining in an information warehouse environment

Data Mining in an Information Warehouse environment

The first phase, Data Preparation, can be further split into two: Data Integration and Data Selection and Pre-analysis. Data Integration refers to the process of merging data which typically resides in an operational environment having multiple files or databases. Resolving semantic ambiguities, handling missing values in data and cleaning dirty data sets are typical data integration issues. Because these issues are common with those found while building Data Warehouses, we will not discuss them here. A discussion of these topics is found in the companion white paper "IBM Information Warehouse Solution: A Data Warehouse - PLUS." Data Mining does not require that a Data Warehouse be built. Often, data can be downloaded from the operational files to flat files that contain the data ready for the Data Mining analysis. However, in many situations, like the one shown in , Data Mining can and will be performed directly from a Data Warehouse. Other issues that occur during integration that are specific to Data Mining deal with identifying the data required for mining and eliminating bias in the data. Identifying the data that is relevant to a given mining operation is a problem for which there is no good solution in the marketplace. The person doing the analysis has to determine which data is relevant to the mining operation being performed. For example, to discover product affinities in market basket analysis one may include information about advertising and shelf placement. Bias in the data can result in the "discovery" of erroneous information. For this reason bias in data should be detected and removed prior to performing the mining operations. As a result of the Data Integration step, data is placed in a Data Warehouse (or alternatively, in flat files.) Data Selection and Pre-analysis are then performed to subset the data. This subsetting is done to improve the quality of the mining results or to overcome limitations in current data mining products. Facilities for doing data selection and pre-analysis are usually provided with many mining tools. These facilities can also be provided through special front end products such as the IBM DataGuide* and the IBM Visualizer.

The second phase of the Data Mining process is the phase where the actual mining takes place. The Data Mining processor (see box labelled Data Mining in ) accesses a Data Warehouse that uses a relational database such as DB2* for AIX/6000*. This access is done through a standard SQL interface. Using a middleware product such as DataJoiner*, the same SQL interface allows mining of data from multiple sources. If the data to be mined has been downloaded to a flat file, the Data Mining processor is capable of accessing this file directly (this is not shown in .)

Following the completion of the second phase of the Data Mining process, the third phase of presentation of facts discovered takes place as well as any follow-up action that results from having discovered these facts. As in the first phase, this presentation can be done in the Data Mining processor or in a front end tool such as the IBM Visualizer or DIS.

If there is a need to iterate the process, either through queries or through the application of other mining operators, this should be accomplished working directly from the prior results.


The IBM Solution

Data mining is a key component of an Information Warehouse framework. It is IBM's strategy to provide a data mining solution that complements the capabilities our customers have in their Information Warehouse product family.

IBM has recently announced the availability of high-value, advanced data mining technology. Figure 2 illustrates the overall architecture of the IBM Data Mining Technology. The mining kernel contains modules that perform data mining functions such as those discussed in the section on technology. Initially, a unique and powerful combination of an association operator and a sequential pattern operator have been delivered. Other mining functions will be added later. There are a number of functional characteristics that are similar to both operators: as we discussed earlier, both mining operators follow the discovery model. Thus, when utilizing these tools, users will likely get information about their data that may be totally new to them.

Both operators also provide a notion of completeness, a major differentiation of the IBM Data Mining Technology. For example, when using the sequential pattern operator, ALL patterns that can be found in the data that are present in more than a given support level (see below) are returned to the user. Likewise, ALL the association rules having more than a given confidence and support (see below) are returned to the user.

Another aspect that significantly separates the IBM Data Mining Technology from other query based techniques is its performance. Because all the relationships that satisfy the input parameters are returned in one invocation of the association (or sequential pattern) function, this process can be significantly more effective in discovering valuable facts that exist in the data. For example, one customer reported that using the IBM Data Mining Technology, running on a single RS/6000 engine, a complete set of association rules existing in 1GB of data was returned in slightly less than one hour. In contrast, the same customer reported that utilizing their query based DSS, which ran on a parallel processor engine, would have taken them two weeks to validate all the association rules found by the IBM Data Mining Technology!

The IBM Data Mining Technology is simple to use. To start the association operation, the user needs only to identify the data being mined and two parameters: One is the confidence parameter defined earlier for the association operator. The other, called the support parameter S, directs both the association and the sequential pattern operator to find rules that can be supported by at least S% of the records being mined. The specification of the data to be mined is done in the Control UI component. This component allows users to specify various modes of access to flat files as well as to relational databases through an SQL interface.

Implemented as a client/server system, the mining kernel of the IBM Data Mining Technology performs data mining processing on AIX/6000, MVS and SP2* servers. Results are returned to AIX/6000 or OS/2 clients for review. The results are returned as standard ASCII files, so they may be accessed through a variety of front-end tools that accept ASCII files as input. Many spreadsheets and graphics programs, for example, feature input utilities that can be used for this purpose. This enables the results of mining operations to be smoothly integrated with a variety of tools and applications already in use at various firms. It also eliminates the need to extensively train staff members in a new environment. An API provides additional input from applications that may invoke the mining functions directly.

As was discussed in the previous section, users will often select a sub-collection of all the data to perform the mining operation on it. For example, more meaningful associations on products in typical market baskets may be obtained when different runs of the association operator are done on data that represents a given quarter, or a given week, as opposed to data that spans an entire year. The reason for this is that rules that exist for a given support level are less likely to appear when mining association data that represents long periods of time. Likewise, it may be more meaningful to discover associations among products in typical market baskets when the search is done using data from one store at a time. It is then useful to compare the results of the mining operations across several data sets. To support this mode of operation, a knowledge database is provided to store the rules from different mining runs.

A variety of data sources may be used to form the base of data to be mined. The current version of the IBM Data Mining Technology provides support for AIX/6000 input files and DB2 for AIX/6000 databases through an SQL interface. An open API is also supported to allow access to a variety of other data sources.

Overall architecture of the IBM Data Mining Technology

Overall architecture of the IBM Data Mining Technology.

The IBM solution is based, in part, on a number of years of research conducted on data mining technology at IBM's Research Division. IBM has the breadth to take this solution end-to-end: we understand our customers' business problems and can help build their business models; we can analyze and build their data warehouse, integrating the technology components and assist in helping analyze the results - leaving behind an operational system. In addition, our technology spans the variety of data mining functions discussed earlier, i.e., associations, sequential patterns, classification and clustering.


Customer Validation Examples

The research prototype for the IBM Data Mining Technology has already been tested in a number of customer environments to validate its usefulness to businesses. Some of these early customer tests involved market basket analysis applications (described previously), and it's worth taking a closer look at how IBM's efforts were used to satisfy these application requirements.

As noted earlier, a simple and common data mining application involves the analysis of sales data in retail environments; much of this is said to involve "market basket" analysis, as it requires understanding purchases made by a customer in a single transaction (or what items he/she placed in a shopping cart or market basket for purchase). Understanding customers' buying patterns is important, and retailers commonly seek to learn if and how sales promotions influence their customers' purchases. Such information can help retailers optimize their sales strategies and maximize their profits. It can also help them improve their responsiveness to changes in the market.

The basis for this kind of analysis is available in the form of huge amounts of sales transaction data. Such operational data is already being collected by most retailers as a part of their internal information systems. The data may be stored in a variety of formats, but relational DBMSs -- such as IBM's DB2 -- are a common choice. To discern relevant buying patterns, business analysts would often write queries -- sometimes highly complex queries -- to test their assumptions. As we have seen, this process can be quite time-consuming.

Using the IBM Data Mining Technology as a data mining tool eliminates the need for users to formulate queries to test such assumptions. Instead, the product automatically analyzes patterns from a given set of data and produces a more complete result. Furthermore, the user need not know the internal structure of the database or its interface (such as SQL). He/she can concentrate on the business view of the discovered rules.

One customer used IBM's solution for discovering associations to determine if product affinity patterns could be useful in optimizing the way orders are accumulated and picked from the distribution centers. Using the same data as source, it was determined that using associations, a set of dynamic rules could be generated which would allow orders to either be accumulated or picked based on the likelihood of another identical order occurring within the next few days. As a result, it was shown that optimizing these decisions appears to take nearly one day off the fulfilment cycle, which could result in substantial cost savings.

A third project was done to understand the effectiveness of "back page" advertising, which involves including a number of "loss leader" items for sale at low prices to draw customers into a store (and, hopefully, prompt them to purchase other items that offer the retailer a greater profit). Selecting appropriate items for such sales and advertising can have a significant influence on a retailer's profits. IBM's solution was used to conduct an "item affinity" analysis, or to discover the relationships between all items of all sales transactions. This helped the retailer determine which items would be best to advertise as "loss leaders," as they would be more likely to trigger purchases of other products. Furthermore, repeating the mining process over time helped the retailer analyze the effectiveness and influence of different types of advertisements.


Summary

Data mining offers firms in many industries the ability to discover hidden patterns in their data -- patterns that can help them understand customer behavior and market trends. The advent of parallel processing and new software technology enable customers to capitalize on the benefits of data mining more effectively than had been possible previously.

After working closely with some of its key customers and refining its research prototype, IBM has announced the IBM Data Mining Technology for performing data mining operations. Its open architecture enables it to be integrated into many existing environments and can complement existing data warehousing solutions. Replication, extraction, and decision support tools can also complement data mining activities. Various data sources and operating system platforms are supported, and IBM expects to enhance this support over time. Furthermore, available IBM consulting services can help customers apply data mining to problems affecting their firms and industries.

[an error occurred while processing this directive]