Storage Systems - Projects - Compliant Storage

Organizations today have amassed -- and continue to amass -- huge volumes of electronic records (e.g. electronic mail, instant messages, drug development logs, medical records). Properly managing the very large volume of records is both costly and challenging, especially because electronic records tend to be relatively easy to modify and to modify without leaving much of a trace. Furthermore, recent high-profiled corporate and personal misconduct has led to intense regulatory scrutiny of record-retention practices. In addition, the cost of electronic-record discovery to support litigation has been soaring. Properl and cost-effective management of electronic records is, therefore, a modern bisiness imperative.

In our Compliance and ILM-Driven Record Retention project, we are developing complete and cost-effective solutions for storing and managing large amounts of electronic data over long periods of time so our clients can maximize the value -- and minimize any liability -- of their data to the organization. This project is composed of two key parts - Fossilization^TM, which focuses on properly managing the records, and Holistic Information Lifecycle Management (ILM), which aims to cost-effectively manage the records.

Fossilization^TM

Early in our Fossilization project, we systematically analyzed the requirements for properly retaining and managing electronic records. In the process, we identified serious deficiencies in today's electronic record-keeping capabilities and overcame these deficiencies by developing technology the cuts across traditional disciplines -- such as database, operating system, computer architecture, security and packaging. For example, we contend that the records have to be protected not just from accidental loss and alteration, but also from intentional and even inside attacks. Furthermore, the current focus of storing records in Write-Once-Read-Many (WORM) storage is increasingly inadequate to ensure that the records are trustworthy. Specifically, we must use the WORM storage properly to preserve not only the records, but also any associated metadata and search index that refers to the records for it to be considered trustworthy from an end-to-end perspective -- from when the records are created to when they are actually used in an audit, a legal or regulatory discovery or an internal investigation. We call this the Fossilization of records. In addition, we contend that WORM storage devices are not created equal. Given the threat of intentional and inside attacks, proper record retention requires using WORM storage devices that are secure against such kinds of attack and that can efficiently support Fossilization.

Holistic ILM

A typical ILM solution contains a layered stack of system components including storage, middleware such as Content Management (CM) systems and applications such as Lotus Notes and SAP. Today significant gaps exist across system stacks. Such gaps lead to management complexity, poor performance and low resource utilization from an end-to-end solution point of view. To close such gaps, a holistic approach is needed to link bits and pieces of information to appropriate business functions at proper time. Specifically, we explore synergistic opportunities between storage and CM middleware to create streamlined and holistic ILM solutions. We envision that holistic ILM solutions will incorporate a new type of intelligence that is commonly missing in the traditional storage, yet instrumental to successful information management. Holistic ILM identifies crucial business semantics hidden in CM in form of workflow events and rich metadata. It then leverages CM capabilities and knowledge such as rich data model, full text indexing and content analysis and search, and links them to data management policies and drives appropriate data management actions, such as archiving, search/retrieval and retention, in accordance with business events for the overall ILM solution simplification, scalability and global optimization.

Archiv mit Dokumentenversteinerung, Windsor Hsu, Shauchi Ong and Birgit Pfitzmann, Zeitschrift für Datenrechtund Informationssicherheit, Vol. 5(2), 2005.

Fossilized Index: The Linchpin of Trustworthy Non-Alterable Electronic Records, Qingbo Zhu and Windsor Hsu, Proceedings of the ACM SIGMOD Conference,June 2005. (Acrobat PDF, 2.54 MB)

Information Valuation for Information Lifecycle Management Ying Chen, Proceedings of the IEEE International Conference on Autonomic Computing, June 2005. (Acrobat PDF, 2.54 MB)

Fossilization: A Process for Establishing Truly Trustworthy Records, Windsor Hsu and Shauchi Ong, IBM Research Report RJ 10331, October 2004.

Content Immutable Storage: Truly Trustworthy and Cost-Effective Storage for Electronic Records, Windsor Hsu, Lan Huang and Shauchi Ong, IBM Research Report RJ 10332, October 2004.

Duplicate Management for Reference Data, Timothy E. Denehy and Windsor W. Hsu, IBM Research Report RJ 10305, October 2003.

Fossilization of Electronic Records, Windsor Hsu, Stanford University Computer Systems Colloquium, October 2005.

Fossilization: Compliant Reference Storage Solutions, Innovation Matters, IBM Research, January 2005.