Plagiarism detection by similarity join

Plagiarism detection by similarity join

Schellenberger, R.

De Vries, A.P. (mentor)
De Bruijn, S.T.J. (mentor)

Electrical Engineering, Mathematics and Computer Science

Media and Knowledge Engineering

2009-08-18

Since the internet is so big and most of its content is public, it is very hard to find out where the information came from originally. There are many websites that publish news articles, so people and organizations can easily lose track of where their articles are reused with or without their permission. This paper presents a plagiarism detection algorithm that allows us to quickly compare online news articles with a collection of personal news articles and detect plagiarized passages with the same quality as a human. The algorithm uses a basic shingle index and a Signature Tree as a more advanced pre-filtering step to narrow down the viable documents to a query. The algorithm achieves a score of 0.96 precision and 0.94 recall but is too resource intensive to be considered scalable. When only the pre-filtering step is used, it achieves 0.85 precision and recall creating a speedup of nearly one order of magnitude.

plagarism detection
similarity join
shingles
signature tree
news articles

http://resolver.tudelft.nl/uuid:a18b62c5-e73e-44fc-9336-83a78275f266

2011-08-18

Student theses

master thesis

(c) 2009 Schellenberger, R.