Sage Journals: Discover world-class research

Abstract

In this paper we examine the use of crowdsourcing as a means to detect Linked Data quality problems that are difficult to uncover automatically. We base our approach on the analysis of the most common errors encountered in the DBpedia dataset, and a classification of these errors according to the extent to which they are likely to be amenable to crowdsourcing. We then propose and study different crowdsourcing approaches to identify these Linked Data quality issues, employing DBpedia as our use case: (i) a contest targeting the Linked Data expert community, and (ii) paid microtasks published on Amazon Mechanical Turk. We secondly focus on adapting the Find-Fix-Verify crowdsourcing pattern to exploit the strengths of experts and lay workers. By testing two distinct Find-Verify workflows (lay users only and experts verified by lay users) we reveal how to best combine different crowds’ complementary aptitudes in Linked Data quality issue detection. Empirical results show that a combination of the two styles of crowdsourcing is likely to achieve more effective results than each of them used in isolation, and that human computation is a promising and affordable way to enhance the quality of DBpedia.

Keywords

Quality assessment quality issues Linked Data crowdsourcing microtasks experts

Get full access to this article

View all access options for this article.

References

Acosta,

Simperl,

Flöck and

Vidal, HARE: A hybrid SPARQL engine to enhance query answers via crowdsourcing, in: Proc. of the 8th International Conference on Knowledge Capture, K-CAP 2015, Palisades, NY, USA, October 7–10, 2015,

Barker and

J.M.

Gómez-Pérez, eds, ACM, 2015, pp. 11:1–11:8. doi:10.1145/2815833.2815848.

Acosta,

Zaveri,

Simperl,

Kontokostas,

Auer and

Lehmann, Crowdsourcing Linked Data quality assessment, in: The Semantic Web – ISWC 2013 – Proc. of the 12th International Semantic Web Conference, Part II, Sydney, NSW, Australia, October 21–25, 2013,

Alani,

Kagal,

Fokoue,

Groth,

Biemann,

Parreira,

Aroyo,

Noy,

Welty and

Janowicz, eds, Lecture Notes in Computer Science, Vol. 8219, Springer, Berlin, Heidelberg, 2013, pp. 260–276. doi:10.1007/978-3-642-41338-4_17.

Bernstein,

J.M.

Leimeister,

N.F.

Noy,

Sarasua and

Simperl, Crowdsourcing and the Semantic Web, Dagstuhl Reports 4(7) (2014), 25–51. doi:10.4230/DagRep.4.7.25.

M.S.

Bernstein,

Little,

R.C.

Miller,

Hartmann,

M.S.

Ackerman,

D.R.

Karger,

Crowell and

Panovich, Soylent: A word processor with a crowd inside, in: Proc. of the 23rd Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA, October 3–6, 2010,

Perlin,

Czerwinski and

Miller, eds, ACM, 2010, pp. 313–322. doi:10.1145/1866029.1866078.

Bizer and

Cyganiak, Quality-driven information filtering using the WIQA policy framework, Journal of Web Semantics 7(1) (2009), 1–10. doi:10.1016/j.websem.2008.02.005.

Bizer,

Lehmann,

Kobilarov,

Auer,

Becker,

Cyganiak and

Hellmann, DBpedia – A crystallization point for the Web of Data, Journal of Web Semantics 7(3) (2009), 154–165. doi:10.1016/j.websem.2009.07.002.

M.J.

Cafarella,

A.Y.

Halevy,

D.Z.

Wang,

Wu and

Zhang, WebTables: exploring the power of tables on the web, The Proceedings of the VLDB Endowment 1(1) (2008), 538–549. doi:10.14778/1453856.1453916.

Cherix,

Usbeck,

Both and

Lehmann, Lessons learned – The case of CROCUS: cluster-based ontology data cleansing, in: The Semantic Web: ESWC 2014 Satellite Events – ESWC 2014 Satellite Events, Revised Selected Papers, Anissaras, Crete, Greece, May 25–29, 2014,

Presutti,

Blomqvist,

Troncy,

Sack,

Papadakis and

Tordai, eds, Lecture Notes in Computer Science, Vol. 8798, Springer, 2014, pp. 14–24. doi:10.1007/978-3-319-11955-7_2.

Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20(1) (1960), 37–46. doi:10.1177/001316446002000104.

10.

Demartini,

D.E.

Difallah and

Cudré-Mauroux, ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking, in: Proc. of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16–20, 2012,

Mille,

F.L.

Gandon,

Misselis,

Rabinovich and

Staab, eds, ACM, 2012, pp. 469–478. doi:10.1145/2187836.2187900.

11.

D.E.

Difallah,

Catasta,

Demartini,

P.G.

Ipeirotis and

Cudré-Mauroux, The dynamics of micro-task crowdsourcing: The case of Amazon MTurk, in: Proc. of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18–22, 2015,

Gangemi,

Leonardi and

Panconesi, eds, ACM, 2015, pp. 238–247. doi:10.1145/2736277.2741685.

12.

Feyisetan,

Simperl,

M.V.

Kleek and

Shadbolt, Improving paid microtasks through gamification and adaptive furtherance incentives, in: Proc. of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18–22, 2015,

Gangemi,

Leonardi and

Panconesi, eds, ACM, 2015, pp. 333–343. doi:10.1145/2736277.2741639.

13.

Fleischhacker,

Paulheim,

Bryl,

Völker and

Bizer, Detecting errors in numerical linked data using cross-checked outlier detection, in: The Semantic Web – ISWC 2014 – Proc. of the 13th International Semantic Web Conference, Part I, Riva del Garda, Italy, October 19–23, 2014,

Mika,

Tudorache,

Bernstein,

Welty,

C.A.

Knoblock,

Vrandecic,

P.T.

Groth,

N.F.

Noy,

Janowicz and

C.A.

Goble, eds, Lecture Notes in Computer Science, Vol. 8796, Springer, 2014, pp. 357–372. doi:10.1007/978-3-319-11964-9_23.

14.

Fleiss

et al., Measuring nominal scale agreement among many raters, Psychological Bulletin 76(5) (1971), 378–382. doi:10.1037/h0031619.

15.

Flemming, Quality characteristics of Linked Data publishing datasources, Master’s thesis, Humboldt-Universität of Berlin, 2010.

16.

Fürber and

Hepp, Using Semantic Web resources for data quality management, in: Knowledge Engineering and Management by the Masses – Proc. of the 17th International Conference, EKAW 2010, Lisbon, Portugal, October 11–15, 2010,

Cimiano and

H.S.

Pinto, eds, Lecture Notes in Computer Science, Vol. 6317, Springer, 2010, pp. 211–225.

17.

Fürber and

Hepp, Using SPARQL and SPIN for data quality management on the Semantic Web, in: Business Information Systems, Proc. of the 13th International Conference, BIS 2010, Berlin, Germany, May 3–5, 2010,

Abramowicz and

Tolksdorf, eds, Lecture Notes in Business Information Processing, Vol. 47, Springer, 2010, pp. 35–46. doi:10.1007/978-3-642-12814-1_4.

18.

Fürber and

Hepp, SWIQA – A Semantic Web information quality assessment framework, in: 19th European Conference on Information Systems, ECIS 2011, Helsinki, Finland, June 9–11, 2011,

V.K.

Tuunainen,

Rossi and

Nandhakumar, eds, ECIS 2011, Vol. 15, IEEE Computer Society, 2011, pp. 19–30, http://is2.lse.ac.uk/asp/aspecis/20110077.pdf .

19.

Guéret,

P.T.

Groth,

Stadler and

Lehmann, Assessing Linked Data mappings using network measures, in: The Semantic Web: Research and Applications – Proc. of the 9th Extended Semantic Web Conference, ESWC 2012, Heraklion, Crete, Greece, May 27–31, 2012,

Simperl,

Cimiano,

Polleres,

Ó.

Corcho and

Presutti, eds, Lecture Notes in Computer Science, Vol. 7295, Springer, 2012, pp. 87–102. doi:10.1007/978-3-642-30284-8_13.

20.

Heath and

Bizer, Linked Data: Evolving the Web into a global data space, Synthesis Lectures on the Semantic Web: Theory and Technology (2011), Morgan & Claypool Publishers. doi:10.2200/S00334ED1V01Y201102WBE001.

21.

Hogan,

Harth,

Passant,

Decker and

Polleres, Weaving the pedantic Web, in: Proc. of the WWW2010 Workshop on Linked Data on the Web, LDOW 2010, Raleigh, USA, April 27, 2010,

Bizer,

Heath,

Berners-Lee and

Hausenblas, eds, CEUR Workshop Proceedings, Vol. 628, CEUR-WS.org, 2010, http://ceur-ws.org/Vol-628/ldow2010_paper04.pdf .

22.

Hogan,

Umbrich,

Harth,

Cyganiak,

Polleres and

Decker, An empirical survey of Linked Data conformance, Journal of Web Semantics 14 (2012), 14–44. doi:10.1016/j.websem.2012.02.001.

23.

Howe, The rise of crowdsourcing, Wired Magazine, June 2006, http://www.wired.com/2006/06/crowds/.

24.

J.M.

Juran,

F.M.

Gryna and

R.S.

Bingham, The Quality Control Handbook, McGraw-Hill, 1974.

25.

M.V.

Kleek,

D.A.

Smith,

H.S.

Packer,

Skinner and

N.R.

Shadbolt, Carpé data: Supporting serendipitous data integration in personal information management, in: ACM SIGCHI Conference on Human Factors in Computing Systems, CHI’13, Paris, France, April 27–May 2, 2013,

W.E.

Mackay,

S.A.

Brewster and

Bødker, eds, ACM, 2013, pp. 2339–2348. doi:10.1145/2470654.2481324.

26.

Kontokostas,

Westphal,

Auer,

Hellmann,

Lehmann and

Cornelissen, Databugger: A test-driven framework for debugging the Web of Data, in: 23rd International World Wide Web Conference, WWW’14, Companion Volume, Seoul, Republic of Korea, April 7–11, 2014,

Chung,

A.Z.

Broder,

Shim and

Suel, eds, ACM, 2014, pp. 115–118. doi:10.1145/2567948.2577017.

27.

Kontokostas,

Westphal,

Auer,

Hellmann,

Lehmann,

Cornelissen and

Zaveri, Test-driven evaluation of Linked Data quality, in: 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7–11, 2014,

Chung,

A.Z.

Broder,

Shim and

Suel, eds, ACM, 2014, pp. 747–758. doi:10.1145/2566486.2568002.

28.

Kontokostas,

Zaveri,

Auer and

Lehmann, TripleCheckMate: A tool for crowdsourcing the quality assessment of Linked Data, in: Knowledge Engineering and the Semantic Web – Proc. of the 4th International Conference, KESW 2013, St. Petersburg, Russia, October 7–9, 2013,

Klinov and

Mouromtsev, eds, Communications in Computer and Information Science, Vol. 394, Springer, 2013, pp. 265–272. doi:10.1007/978-3-642-41360-5_22.

29.

Lehmann and

Bühmann, ORE – A tool for repairing and enriching knowledge bases, in: The Semantic Web – ISWC 2010 – 9th International Semantic Web Conference, ISWC 2010, Part II, Revised Selected Papers, Shanghai, China, November 7–11, 2010,

P.F.

Patel-Schneider,

Pan,

Hitzler,

Mika,

Zhang,

J.Z.

Pan,

Horrocks and

Glimm, eds, Lecture Notes in Computer Science, Vol. 6497, Springer, 2010, pp. 177–193. doi:10.1007/978-3-642-17749-1_12.

30.

Lehmann,

Isele,

Jakob,

Jentzsch,

Kontokostas,

P.N.

Mendes,

Hellmann,

Morsey,

van Kleef,

Auer and

Bizer, DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia, Semantic Web 6(2) (2015), 167–195. doi:10.3233/SW-140134.

31.

J.M.

Leimeister,

M.J.

Huber,

Bretschneider and

Krcmar, Leveraging crowdsourcing: Activation-supporting components for it-based ideas competition, Journal of Management Information Systems 26(1) (2009), 197–224. doi:10.2753/MIS0742-1222260108.

32.

Li,

Xu and

Zhong, Probabilistic error detecting in numerical linked data, in: Database and Expert Systems Applications – Proc. of the 26th International Conference, DEXA 2015, Part I, Valencia, Spain, September 1–4, 2015,

Chen,

Hameurlain,

Toumani,

Wagner and

Decker, eds, Lecture Notes in Computer Science, Vol. 9261, Springer, 2015, pp. 61–75. doi:10.1007/978-3-319-22849-5_5.

33.

Luczak-Rösch,

Simperl,

Stadtmüller and

Käfer, The role of ontology engineering in Linked Data publishing and management: An empirical study, International Journal on Semantic Web and Information Systems 10(3) (2014), 74–91. doi:10.4018/IJSWIS.2014070103.

34.

Markotschi and

Völker, GuessWhat?! – Human intelligence for mining Linked Data, in: Proc. of the 1st Workshop on Knowledge Injection Into and Extraction from Linked Data Lisbon, Lisbon, Portugal, October 15, 2010,

Presutti,

Scharffe and

Svatek, eds, CEUR Workshop Proceedings, Vol. 631, CEUR-WS.org, 2010, pp. 28–39, http://ceur-ws.org/Vol-631/paper4.pdf .

35.

McCann,

Shen and

Doan, Matching schemas in online communities: A Web 2.0 approach, in: Proc. of the 24th International Conference on Data Engineering, ICDE 2008, Cancún, México, April 7–12, 2008,

Alonso,

J.A.

Blakeley and

A.L.P.

Chen, eds, IEEE Computer Society, 2008, pp. 110–119. doi:10.1109/ICDE.2008.4497419.

36.

P.N.

Mendes,

Mühleisen and

C.B.

Sieve, Linked Data quality assessment and fusion, in: Proc. of the 2012 Joint EDBT/ICDT Workshops, Berlin, Germany, March 30, 2012,

Srivastava and

Ari, eds, Vol. 30, ACM, 2012, pp. 116–123. doi:10.1145/2320765.2320803.

37.

Paulheim and

Bizer, Type inference on noisy RDF data, in: The Semantic Web – ISWC 2013 – Proc. of the 12th International Semantic Web Conference, Part I, Sydney, NSW, Australia, October 21–25, 2013,

Alani,

Kagal,

Fokoue,

P.T.

Groth,

Biemann,

J.X.

Parreira,

Aroyo,

N.F.

Noy,

Welty and

Janowicz, eds, Lecture Notes in Computer Science, Vol. 8218, Springer, 2013, pp. 510–525. doi:10.1007/978-3-642-41335-3_32.

38.

Paulheim and

Bizer, Improving the quality of Linked Data using statistical distributions, International Journal on Semantic Web and Information Systems 10(2) (2014), 63–86. doi:10.4018/ijswis.2014040104.

39.

Phillips and

Davis, Tags for identifying languages, BCP 47, RFC Editor, September 2009, http://tools.ietf.org/html/bcp47.

40.

Popov, mashpoint: Supporting data-centric navigation on the Web, in: CHI Conference on Human Factors in Computing Systems, CHI’12, Extended Abstracts Volume, Austin, TX, USA, May 5–10, 2012,

J.A.

Konstan,

E.H.

Chi and

Höök, eds, ACM, 2012, pp. 2249–2254. doi:10.1145/2212776.2223784.

41.

Prud’hommeaux,

J.E.L.

Gayo and

H.R.

Solbrig, Shape expressions: An RDF validation and transformation language, in: Proc. of the 10th International Conference on Semantic Systems, SEMANTICS 2014, Leipzig, Germany, September 4–5, 2014,

Sack,

Filipowska,

Lehmann and

Hellmann, eds, ACM, 2014, pp. 32–40. doi:10.1145/2660517.2660523.

42.

Sabou,

Bontcheva,

Scharl and

Föls, Games with a purpose or mechanised labour?: A comparative study, in: 13th International Conference on Knowledge Management and Knowledge Technologies, I-KNOW’13, Graz, Austria, September 4–6, 2013,

S.N.

Lindstaedt and

Granitzer, eds, ACM, 2013, pp. 19:1–19:8. doi:10.1145/2494188.2494210.

43.

Sarasua,

Simperl and

N.F.

Noy, CrowdMap: Crowdsourcing ontology alignment with microtasks, in: The Semantic Web – ISWC 2012 – Proc. of the 11th International Semantic Web Conference, Part I, Boston, MA, USA, November 11–15, 2012,

Cudré-Mauroux,

Heflin,

Sirin,

Tudorache,

Euzenat,

Hauswirth,

J.X.

Parreira,

Hendler,

Schreiber,

Bernstein and

Blomqvist, eds, Lecture Notes in Computer Science, Vol. 7649, Springer, 2012, pp. 525–541.

44.

Sarasua,

Simperl,

N.F.

Noy,

Bernstein and

J.M.

Leimeister, Crowdsourcing and the Semantic Web: A research manifesto, Human Computation 2(1) (2015), 3–17. doi:10.15346/hc.v2i1.2.

45.

Schraefel,

Golbeck,

Degler,

Bernstein and

Rutledge, Semantic Web user interactions: Exploring HCI challenges, in: Extended Abstracts Proc. of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, Florence, Italy, April 5–10, 2008,

Czerwinski,

A.M.

Lund and

D.S.

Tan, eds, ACM, 2008, pp. 3929–3932. doi:10.1145/1358628.1358959.

46.

Schraefel and

Rutledge, User interaction in Semantic Web research, Journal of Web Semantics 8(4) (2010), 375–376. doi:10.1016/j.websem.2010.10.003.

47.

Simperl,

Cuel and

Stein, Incentive-centric Semantic Web application engineering, Synthesis Lectures on the Semantic Web: Theory and Technology 4 (2013), Morgan & Claypool Publishers. doi:10.2200/S00460ED1V01Y201212WBE004.

48.

Siorpaes and

Hepp, OntoGame: Weaving the Semantic Web by online games, in: The Semantic Web: Research and Applications, Proc. of the 5th European Semantic Web Conference, ESWC 2008, Tenerife, Canary Islands, Spain, June 1–5, 2008,

Bechhofer,

Hauswirth,

Hoffmann and

Koubarakis, eds, Lecture Notes in Computer Science, Vol. 5021, Springer, 2008, pp. 751–766. doi:10.1007/978-3-540-68234-9_54.

49.

Siorpaes and

E.P.B.

Simperl, Human intelligence in the process of semantic content creation, World Wide Web 13(1–2) (2010), 33–59. doi:10.1007/s11280-009-0078-0.

50.

Terwiesch and

Xu, Innovation contests, open innovation, and multiagent problem solving, Manage. Sci. 54(9) (Sept. 2008), 1529–1543. doi:10.1287/mnsc.1080.0884.

51.

Thaler,

E.P.B.

Simperl and

Siorpaes, SpotTheLink: A game for ontology alignment, in: 6th Conference on Professional Knowledge Management: From Knowledge to Action, Innsbruck, Austria, February 21–23, 2011,

Maier, ed., LNI, Vol. 182, GI, 2011, pp. 246–253, http://subs.emis.de/LNI/Proceedings/Proceedings182/article23.html .

52.

Thaler,

Siorpaes,

Mear,

E.P.B.

Simperl and

Goodman, SeaFish: A game for collaborative and visual image annotation and interlinking, in: The Semanic Web: Research and Applications – Proc. of the 8th Extended Semantic Web Conference, ESWC 2011, Part II, Heraklion, Crete, Greece, May 29–June 2, 2011,

Antoniou,

Grobelnik,

E.P.B.

Simperl,

Parsia,

Plexousakis,

P.D.

Leenheer and

J.Z.

Pan, eds, Lecture Notes in Computer Science, Vol. 6644, Springer, 2011, pp. 466–470. doi:10.1007/978-3-642-21064-8_36.

53.

Töpper,

Knuth and

Sack, DBpedia ontology enrichment for inconsistency detection, in: I-SEMANTICS 2012 – 8th International Conference on Semantic Systems, I-SEMANTICS’12, Graz, Austria, September 5–7, 2012,

Presutti and

H.S.

Pinto, eds, ACM, 2012, pp. 33–40. doi:10.1145/2362499.2362505.

54.

V.S.

Uren,

Lei,

Lopez,

Liu,

Motta and

Giordanino, The usability of semantic search tools: A review, Knowledge Engineering Review 22(4) (2007), 361–377. doi:10.1017/S0269888907001233.

55.

Villazón-Terrazas and

Corcho, Methodological guidelines for publishing Linked Data, Una Profesión, un futuro: Actas de las XII Jornadas Españolas de Documentación: Málaga 25(26) (2011), 20.

56.

Waitelonis,

Ludwig,

Knuth and

Sack, WhoKnows? Evaluating Linked Data heuristics with a quiz that cleans up DBpedia, International Journal of Interactive Technology and Smart Education (ITSE) 8(4) (2011), 236–248, Emerald. doi:10.1108/17415651111189478.

57.

Wang,

Kraska,

M.J.

Franklin and

Feng, CrowdER: Crowdsourcing entity resolution, Proceedings of the VLDB Endowment 5(11) (July 2012), 1483–1494, http://vldb.org/pvldb/vol5/p1483_jiannanwang_vldb2012.pdf . doi:10.14778/2350229.2350263.

58.

Zaveri,

Kontokostas,

M.A.

Sherif,

Bühmann,

Morsey,

Auer and

Lehmann, User-driven quality evaluation of DBpedia, in: i-SEMANTICS 2013 – 9th International Conference on Semantic Systems, ISEM’13, Graz, Austria, September 4–6, 2013,

Sabou,

Blomqvist,

T.D.

Noia,

Sack and

Pellegrini, eds, ACM, 2013, pp. 97–104. doi:10.1145/2506182.2506195.

59.

Zaveri,

Rula,

Maurino,

Pietrobon,

Lehmann and

Auer, Quality assessment methodologies for Linked Data: A survey, Semantic Web 7(1) (2016), 63–93. doi:10.3233/SW-150175.