4REAL Workshop

Workshop on Research Results Reproducibility
and Resources Citation
in Science and Technology of Language


28 May 2016
Portorož, Slovenia
Collocated with LREC2016 – 10th Language Resources and Evaluation Conference


Important dates:

15 February 2016: Deadline for submissions
15 March 2016: Notification of authors
15 April 2016: Deadline for camera-ready
28 May 2016: Workshop



09:00 – 10:30 Reproducibility
Chair: Joseph Mariani

Seeking to Reproduce “Easy Domain Adaptation”
Luís Gomes, Gertjan van Noord, António Branco and Steven Neale

Reproducibility in natural language processing: A case study of two R libraries for mining PubMed/MEDLINE
Kevin Cohen, Jingbo Xia, Christophe Roeder and Lawrence Hunter

Gonito.net – open platform for research competition, cooperation and reproducibility
Filip Graliski, Rafal Jaworski, Lukasz Borchmann and Piotr Wierzchon

10:30 – 11:00 Coffee break

11:00 – 12:00 Round table

Reproducibility in Language Science and Technology: Ready for the Integrity Debate?
Panelists: Piek Vossen, Nicoletta Calzolari, Edouard Geoffrois, Khalid Choukri
Chair: António Branco

12:00 – 13:00 Citation
Chair: Justus Roux

SHORTREF.ORG Making URLs Easy-to-Cite
Jozef Misutka, Ondrej Kosarko and Amor Kamran

Linking Language Resources and NLP papers
Gil Francopoulo, Joseph Mariani and Patrick Paroubek




Branco, António, Nicoletta Calzolari and Khalid Choukri (eds.), 2016, Proceedings of the Workshop on Research Results Reproducibility and Resources Citation in Science and Technology of Language.


Call for papers:

The discussion on research integrity has grown in importance as the resources allocated to and societal impact of scientific activities have been expanding (e.g. Stodden, 2013, Aarts et al., 2015), to the point that it has recently crossed the borders of the research world and made its appearance in important mass media and was brought to the attention of the general public (e.g. Nail and Gautam, 2011, Zimmer, 2012, Begley and Sharon 2012, The Economist, 2013).

The immediate motivation for this increased interest is to be found in a number of factors, including the realization that for some published results, their replication is not being obtained (e.g. Prinz et al., 2011; Belgley and Ellis, 2012); that there may be problems with the commonly accepted reviewing procedures, where deliberately falsified submissions, with fabricated errors and fake authors, get accepted even in respectable journals (e.g. Bohannon, 2013); that the expectation of researchers vis a vis misconduct, as revealed in inquiries to scientists on questionable practices, scores higher than one might expect or would be ready to accept (e.g. Fanelli, 2009); among several others.

Doing justice to and building on the inherent ethos of scientific inquiry, this issue has been under thorough inquiry leading to a scrutiny on its possible immediate causes and underlying factors, and to initiatives to respond to its challenge, namely by the setting up of dedicated conferences (e.g. WCRI – World Conference on Research Integrity), dedicated journals (e.g. RIPR – Research Integrity and Peer review), support platforms (e.g. COS – Center for Open Science), revised and more stringent procedures (e.g. Nature, 2013), batch replication studies (e.g. Aarts et al, 2015), investigations on misconduct (e.g. Hvistendahl, 2013), etc.

This workshop seeks to foster the discussion and the advancement on a topic that has been so far given insufficient attention in the research area of language processing tools and resources (Branco, 2013, Fokkens et al., 2013) and that has been an important topic emerging in other scientific areas. That is the topic of the reproducibility of research results and the citation of resources, and its impact on research integrity.

We are thus inviting submissions of articles that present pioneering cases, either with positive or negative results, of actual replication exercises of previous published results in our area. We are interested also in articles discussing the challenges, the risk factors, the procedures, etc. specific to our area or that should be adopted, or adapted from other neighboring areas, possibly including of course the new risks raised by the replication articles themselves and their own integrity, in view of the preservation of the reputation of colleagues and works whose results are reported has having been replicated, etc.

By the same token, this workshop is interested also on articles addressing methodologies for monitoring, maintaining or improving citation of language resources and tools and to assess the importance of data citation for research integrity and for the advancement of natural language science and technology.



Please submit your papers, duly anonymized, at https://www.softconf.com/lrec2016/4REAL. Accepted papers will be presented as oral presentations or as posters. All accepted papers will be published in the workshop proceedings.

Papers should be formatted according to the stylesheet to be provided on the LREC 2016 website and should not exceed 8 pages, including references and appendices. Papers should be submitted in PDF format through the URL mentioned above.

When submitting a paper, authors will be asked to provide essential information, if this is appropriate for their cases, about resources (in a broad sense, i.e., also technologies, standards, evaluation kits, etc.) that have been used for the work described in the paper or are a new result of your research. Moreover, ELRA encourages all LREC authors to share the described LRs (data, tools, services, etc.) to enable their reuse and replicability of experiments (including evaluation ones).


Organization committee:

António Branco (University of Lisbon)
Nicoletta Calzolari (ILC)
Khalid Choukri (ELRA)


Program committee:

António Branco (University of Lisbon)
Iryna Gurevych (Universität Darmstadt)
Isabel Trancoso (INESC-ID / IST, University of Lisbon)
Joseph Mariani (CNRS/LIMSI)
Justus Roux (NWVU)
Khalid Choukri (ELRA)
Maria Gavrilidou (ILSP)
Marko Grobelnik (Jozef Stefan Institute)
Marko Tadic (University of Zagreb)
Mike Rosner (University of Malta)
Nicoletta Calzolari (ILC)
Nick Campbell (Trinity College Dublin)
Senja Pollak (Jozef Stefan Institute)
Stelios Piperidis (ILSP)
Steven Krauwer (University of Utrecht)
Thierry Declerck (DFKI)
Torsten Zesch (University of Duisburg-Essen)
Yohei Murakami (Language Grid Japan)



Aarts, et al., 2015, “Estimating the reproducibility of psychological science”, Science.
The Economist, 2013, Unreliable Research: Trouble at the Lab, The Economist, October 19, 2013, online.
Begley, 2012, In cancer science, many “discoveries” don’t hold up, Reuters, March 28th, online.
Begley and Ellis, 2012, Drug development: Raise standards for preclinical cancer research, Nature.
Bohannon, John, 2013, Who’s Afraid of Peer Review?, Science.
Branco, António, 2013, Reliability and Meta-reliability of Language Resources: Ready to initiate the Integrity Debate?, TLT12.
COS, Centre for Open Science.
Fanelli,2009 How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data, PL OS ONE.
Fokkens et al., 2013, Offspring from Reproduction Problems: What Replication Failure Teaches US, ACL.
Hvistendahl, 2013, “China’s Publication Bazaar”, Science.
Nail, 2011, Scientists’ Elusive Goal: Reproducing Study Results, The Wall Street Journal.
Nature, 2013, “Announcement: Reducing our irreproducibility”, Nature, Editorial.
Prinz, et al., 2011, Believe it or not: how much can we rely on published data on potential drug targets?, Nature Reviews Drug Discovery 10, 712.
RIPR, Research Integrity and Peer Review.
Stodden, 2013, Resolving Irreproducibility in Empirical and Computational Research, IMS Bulletin Online.
WCRI, World Conference on Research Integrity.
Zimmer, Carl, 2012, A Sharp Rise in Retractions Prompts Calls for Reform, The New York Times.