List representation applied to sparse datacubes for data warehousing and data mining

Frank Wang, Frahi Marir, John Gordon, Na Helian

Research output: Contribution to conferencePaperpeer-review

Abstract

Typically 80% of the data in the logical OLAP datacube, the core engine of data warehouses, are zero. When it comes to sparse, the performance quickly degrades due to the heavy I/O overheads in sorting and merging intermediate results. In this work, we first introduce a list representation in main memory for storing and computing datasets. The sparse transaction dataset is compressed as the empty cells are removed Accordingly we propose a new algorithm for association rule mining on the platform of list representation, which just needs to scan the transaction database once to generate all the possible rules. In contrast, the well-known apriori algorithm requires repeated scans of the databases, thereby resulting in heavy I/O accesses particularly when considering large candidate datasets. In our opinion, this new algorithm using list representation economizes storage space and accesses.
Original languageEnglish
Pages871-875
Publication statusPublished - 2003
EventIntelligent Data Engineering and Automated Learning: 4th International Conference - Hong Kong, China
Duration: 21 Mar 200323 Mar 2003

Conference

ConferenceIntelligent Data Engineering and Automated Learning: 4th International Conference
Abbreviated titleIDEAL 2003
Country/TerritoryChina
CityHong Kong
Period21/03/0323/03/03

Fingerprint

Dive into the research topics of 'List representation applied to sparse datacubes for data warehousing and data mining'. Together they form a unique fingerprint.

Cite this