PC-filter: a Robust filtering technique for duplicate record detection in large databases

Zhang, Ji and Ling, Tok Wang and Bruckner, Robert and Liu, Han (2004) PC-filter: a Robust filtering technique for duplicate record detection in large databases. In: 15th International Conference on Database and Expert Systems Applications (DEXA'04), 30 August - 3 Sept 2004, Zaragoza, Spain.

PDF (Accepted Version)

Download (543Kb)


[Abstract]: In this paper, we will propose PC-Filter (PC stands for Partition Comparison), a robust data filter for approximately duplicate record detection in
large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database
and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition
pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of
O(N4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the “Key Selection” problem, the “Scope Specification” problem and the
“Low Recall” problem that the existing methods suffer from.

Statistics for USQ ePrint 5653
Statistics for this ePrint Item
Item Type: Conference or Workshop Item (Commonwealth Reporting Category E) (Paper)
Refereed: Yes
Item Status: Live Archive
Additional Information: Author's version deposited in accordance with the copyright policy of the publisher. The original publication is available at www.springerlink.com)
Faculty / Department / School: Historic - Faculty of Sciences - Department of Maths and Computing
Date Deposited: 09 Sep 2009 23:21
Last Modified: 02 Jul 2013 23:23
Uncontrolled Keywords: PC-filter; partition comparision filter; duplicate record detection
Fields of Research : 08 Information and Computing Sciences > 0801 Artificial Intelligence and Image Processing > 080109 Pattern Recognition and Data Mining
URI: http://eprints.usq.edu.au/id/eprint/5653

Actions (login required)

View Item Archive Repository Staff Only