#lingdata

This new commentary article is a response to a target article from Laura Becker and Matías Guzmán Naranjo titled "Replication and methodological robustness in quantitative typology".

In this paper, Becker and Guzmán Naranjo explore what happens when typological work is reproduced by another team, and what the differences in results mean for the field of linguistic typology.

I wrote this article alongside my former co-chairs of Linguistic Data Interest Group (LDIG) of the Research Data Alliance, reflecting on the work needed to ensure that we are all doing the kind of research that is useful to a more transparent way of doing work in linguistic typology.

From the commentary:

We thank B&GN for their work, but we also thank Dryer (2018), Seržant (2021), Shcherbakova et al. (2023) and Berg (2020), for doing work that could be reproduced (a benchmark much scholarship falls short on) and subsequently having their work scrutinised and evaluated in this way. As B&GN note, “[t]here is no specific reason for choosing these papers other than the fact that the authors made their datasets available”. To work in an open and transparent manner is to open yourself to critical evaluation. Linguistic typology advances because of researchers who have created accessible data as part of their work. This includes individual researchers working on specific languages, whose data is the basis of typological work, as well as those typologists who share the databases of their work. Open ways of working require open mindedness from the whole research community.

Reference

Gawne, L., H.N. Andreassen, L. Ferrara, A.L. Berez-Kroeker (2025). Open research requires open mindedness: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo. Linguistic Typology. doi: 10.1515/lingty-2025-0018

Related links

Linguistic Data Interest Group: Five years of improving data citation practices in linguistics

New Paper: Reproducible research in linguistics: A position statement on data citation and attribution in our field

New Article Published: Reflections on reproducible research, in Reflections on Language Documentation 20 Years after Himmelmann 1998

After five years with the Linguistics Data Interest Group of the Research Data Alliance, I’ve stepped down as a co-chair of the group. I wanted to use this as a chance to collate some of the work I was involved with over the last five years. I’ll still be a member of the LDIG for its next chapter, and thrilled that Andrea Berez-Kroeker, Helene N. Andreassen, and Lindsay Ferrara will be heading things up. And, of course, the group has many excellent members (if you’re a linguist and/or interested in data management, you can join the LDIG too!).

The Austin Principles of Data Citation in Linguistics

The Austin Principles of Data Citation were the first major output for the LDIG, it focuses on the why of data citation. This short document is a position statement on the importance of data in linguistic work. From the preamble:

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The official Austin Principles website

The Superlinguo post

The Trømso Recommendations in academic publishing

If the Austin Principles are the why the Tromsø Recommendations are the how.

The Tromsø Recommendations provide clear guidance for data citation for referencing language data, both in the bibliography and in the text of linguistics publications. The recommendations have been written to account for the rich variety of linguistic data, and include clear guidance and examples.

The official Tromsø Recommendations documents

The Superlinguo post

Building uptake for the Trømso Recommendations

Now that the Tromsø Recommendations have been published, there’s an ongoing campaign to normalise their use in academic publishing and grant writing. Get involved by encouraging your favourite publishers to include to Tromsø Recommendations in their author guidelines!

The Linguistics Data Interest Group (a working group in the Research Data Alliance) have developed the Tromsø Recommendations in collaboration with linguists working in a range of disciplines. The next step is to help encourage citation of data by encouraging journals to include the Tromsø Recommendations in their instructions for authors.

bit.ly/trecs-campaign

Publications

Alongside LDIG colleagues I was involved in a number of publications looking at the role of data in linguistics. Below are links to the Superlinguo posts with more information, the abstract and links to open access versions.

Reproducible research in linguistics: A position statement on data citation and attribution in our field

Situating Linguistics in the Social Science Data Movement. Chapter in the Open Handbook of Linguistic Data Management

Data transparency and citation in the journal Gesture

Reflections on reproducible research, in Reflections on Language Documentation 20 Years after Himmelmann 1998

Putting practice into words: The state of data and methods transparency in grammatical descriptions

Suzy Styles is one of my favourite people to talk to about research data, the importance of transparency in research methods, and how we can always do things better. So when the editors of the Open Handbook of Linguistic Data Management invited us to submit a chapter about the things linguists can learn about data management from discussions in other fields of social science (particularly experimental psychology), I was so excited to sit down with Suzy and bring together everything I’ve learnt in the last few years of our discussions.

The Open Handbook of Linguistic Data Management has 56 chapters, all available as open access PDFs for you to download. Print copies are also available on a print-on-demand model. Chapters cover a range of specific case studies, and approaches, and languages. Putting this volume together has been a major effort, and we’ve been so grateful for all the work done by Andrea Berez-Kroeker, Brad McDonnell, Eve Koller and Lauren Collister. 

Alongside the handbook is a free and open online companion course that covers the first 13 chapters (including ours). The course component for our chapter includes a summary, keyword definitions, links to other chapters, activities you can do for your own work, and a revision quiz! The course page also has a colour version of the Coin Flipping Cowboys illustration that is printed in black and white in the chapter itself. (you’ll have to read the chapter to learn how these cowboys can help you think about your data!)

Opening paragraph

Linguists spend a lot of time working with data, but we do not always give much thought to the role that data plays in building the larger research culture in our field. We can learn a lot about good data management in our own discipline by learning from what is happening in related fields, both in terms of innovations and new benchmarks, as well as when things have not gone right. We look at how data have been conceptualized and managed in other areas of the social sciences, particularly social psychology, and how current attitudes are shaping the future of research. The fundamental theme of this discourse is the centrality of openness, both in terms of transparency of methodology and making primary data more accessible to people beyond the original researchers. This move toward open research aims to reduce biases, both for individual researchers and for the discipline, and encourages more considered data collection and presentation.

Reference

Gawne, L. & S. Styles. 2022. Situating linguistics in the social science data movement. In A.L. Berez-Kroeker, B. McDonnell, E. Koller & L.B. Collister (Eds), The Open Handbook of Linguistic Data Management, 9-25. MIT Press. doi: 10.7551/mitpress/12200.003.0006 [Open Access PDF]

Other resources

The handbook is completely Open Access, the other 55 chapters are available here

Free Open Access companion course covering the first 13 chapters (including ours)

See also:

Reproducible research in linguistics: A position statement on data citation and attribution in our field (article)

The Tromsø Recommendations

The Austin Principles of Data Citation in Linguistics

Data transparency and citation in the journal Gesture (article)

Putting practice into words:  The state of data and methods transparency in grammatical descriptions (article)

The Tromsø recommendations for citation of research data in linguistics

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The Tromsø Recommendations provide clear guidance for data citation for referencing language data, both in the bibliography and in the text of linguistics publications. The recommendations have been written to account for the rich variety of linguistic data, and include clear guidance and examples.

The Linguistics Data Interest Group (LDIG, a working group in the Research Data Alliance) have developed the Tromsø Recommendations in collaboration with linguists working in a range of disciplines.

Making the Tromsø Recommendations part of academic publishing

The next step is to help encourage citation of data by encouraging journals to include the Tromsø Recommendations in their instructions for authors.

The LDIG has launched a campaign to encourage publishers, archives and other groups that present linguistic data to adopt the Tromsø Recommendations.

What would it look like to include the Tromsø Recommendations in academic publishing?

The Tromsø Recommendations cover all types of linguistic data, and have citation formats for all levels of detail - from a whole corpus to a single line of a text. They are designed to focus on content, not style, so they work with the formatting for a particular journal.

To include the recommendations can be as minimal as adding a line of text like this to a publication’s Information for Authors page:

<Journal Name> [encourages|requires] the citation of linguistic data in all published articles. Citation structure should follow the The Tromsø Recommendations for Citation of Research Data in Linguistics. For more on the importance of data citation see The Austin Principles of Data Citation in Linguistics.

How can I help support the Tromsø Recommendations in academic publishing?

If you are a journal or series editor, member of a journal editorial board or publisher, update your author instructions to encourage authors to include data citation.

If you are a contributor to a journal in your field or popular publisher, write to the editors and ask them to consider adopting the Tromsø Recommendations. We have some suggested text for your email or letter in the next tab.

If you are a member of a PhD program, a funding body or a body that gives a linguistics award, consider adopting data citation as a category for assessment.

If you are a linguist or language scientist, encourage the journals, societies, funding and awards bodies you are a member of to take up the Tromsø Recommendations.

For this last group, we’ve created a spreadsheet that has email templates, background information and a spreadsheet where we are tracking which journals have been contacted, and whether they’ve adopted the Tromsø Recommendations: bit.ly/trecs-campaign

You do not have to be an LDIG or RDA member to be involved! Also, if the journals you publish in do not immediately adopt the Tromsø Recommendations for the whole publication, you can always adopt them yourself for the next thing you publish!

See also:

The Austin Principles of Data Citation in Linguistics

Reproducible research in linguistics: A position statement on data citation and attribution in our field (article)

Data transparency and citation in the journal Gesture (article)

Putting practice into words: The state of data and methods transparency in grammatical descriptions (article)

This new paper extends the data transparency survey method that I’ve been using with the Linguistic Data Interest Group to the journal Gesture. It not only brings together two research areas I care deeply about: gesture and data management, but also two groups of collaborators I love working with: Andrea Berez-Kroeker and Helene Andreassen in lingdata and Chelsea Krajcik and Barb Kelly in gesture.

This is the first time we have published a paper on this methodology focusing on a specific journal, and I’m grateful to Sotaro Kita and the Gesture editorial board for being open to this reflexive experience. Some key observations:

Gesture research encompasses a wide range of methodologies

There are a small number of strongly represented languages (hello English), but a lot of linguistic diversity too

Researchers mostly work with their own data, but don’t always give a lot of detail on where that data is located

There is scope to improve the way data is cited back to the original source - something that makes it easier for the reader to engage with the paper but is also helpful for the author!

Gesture is a feature of language that is highly context depended, and is hard to replicate for a new author. Allowing for more transparent access to underlying examples for the reader allows for the possibility of reproducing the analysis (or not!) based on the original data. This approach means the foundation of the field can be built stronger.

Why am I blogging about this in 2020 when it was published in 2019?

The official publication date for this paper is 2019, but Gesture is running a bit behind on its publication schedule. You can see on the Gesture website page for this article that it was first published in 2020. I’m just glad to be able to finally share it with you!

Abstract

Data is central to scholarly research, but the nature and location of data used is often under-reported in research publications. Greater transparency and citation of data have positive effects for the culture of research. This article presents the results of a survey of data citation in six years of articles published in the journal Gesture (12.1–17.2). Gesture researchers draw on a broad range of data types, but the source and location of data are often not disclosed in publications. There is also still a strong research focus on only a small range of the world’s languages and their linguistic diversity. Published papers rarely cite back to the primary data, unless it is already published. We discuss both the implications of these findings and the ways that scholars in the field of gesture studies can build a positive culture around open data.

Keywords:

data citation, open data, data management, gesture studies

Reference

Gawne, Lauren, Chelsea Krajcik, Helene N. Andreassen, Andrea L. Berez-Kroeker & Barbara F. Kelly. (2019). Data Transparency and Citation in the Journal Gesture. Gesture 18(1):  83–109. doi: https://doi.org/10.1075/gest.00034.gaw

Green Open Access version available on Figshare: https://doi.org/10.26181/5f57fddc85ebb

Related research publications on data citation:

Reproducible research in linguistics: A position statement on data citation and attribution in our field

Putting practice into words: New paper out about methods and data in descriptive grammar writing

Survey of Current Reproducibility Practices in Linguistics Journals, 2003-2012

General resources on data citation in linguistics

The Austin principles of data citation in linguistics

The Tromsø recommendations for citation of research data in linguistics

In 1998 Nikolas Himmelmann wrote "Documentary and descriptive linguistics”, an article for the journal Linguistics (I’m not going to link because it’s paywalled, but you should be able to find a copy floating around online). Himmelmann 1998 is an often cited work that makes a case for the work of documentation (collecting data) being as important as the work of description (describing data) in understanding how languages work.

To look at the 20 years since this publication, Bradley McDonnell, Andrea L. Berez-Kroeker and Gary Holton brought together over 30 linguists to reflect on how the field has changed and grown in that time. This is a Special Publication of Language Documentation & Conservation, available open access.  

From the introduction to the special issue:

[W]e invited 38 experts from around the world to reflect on either particular issues within the realm of language documentation or particular regions where language documentation projects are being carried out. The issues addressed in this volume represent a broad and diverse set of topics from multiple perspectives and for multiple purposes that continue to be relevant to documentary linguists and language communities. Some topics have been hotly debated over the past two decades, while others have emerged more recently.

All of the chapters are short and to the point (they’re around 10 pages each). I has a great time writing with Andrea Berez-Kroeker about how language documentation has changed over the last two decades in regards to data management and sharing. Himmelmann talks about the importance of sharing primary data, but the nature of that data, and how we can share it has changed so much since 1998.

Abstract

Reproducibility in language documentation and description means that the analysis given in descriptive publication is presented in a way that allows the reader to access the data on which the claims are based, to verify the analysis for themself. Linguists, including Himmelmann, have long pointed to the centrality of documentation data to linguistic description. Over the twenty years since Himmelmann’s 1998 paper we have seen a growth in digital archiving, and the rise of the Open Access movement. Although there is good infrastructure in place to make reproducible research possible, few descriptive publications clearly link to underlying data, and very little documentation data is publicly accessible. We discuss some of the institutional roadblocks to reproducibility, including a lack of support for the development of published primary data. We also look at what work on language documentation and description can learn from the recent replication crisis in psychology.

Reference

I’m so excited to share this paper on data citation in linguistics, published in Linguistics. Not just because it was co-authored with 13 other linguists, including some of my linguistics heroes, but because it articulates so much of what we need to do to improve the way linguists think about data. (it’s Open Access, so you can read/download and share as you wish!)

I’m often reminded of something Nicholas Evans likes to say:

“Linguistics has a unique but unfulfilled destiny, as the most scientific of the humanities and the most humanistic of the sciences.”

It captures what I love about linguistics, but I think it also explains why we have a complicated relationship with the idea of data, and its management. This position paper outlines why it is important that linguists think about the data on which linguistic claims are made.

My Favourite line

Linguistic data are the very building blocks of our field. Given that linguistic theories need to be borne out through data, we believe that linguistic data are important resources in their own right and represent valuable assets for the field. Therefore, our field needs to accept responsibility for the proper documentation, preservation, attribution, and citation of these assets. The responsibility to do so is an integral part of linguistic research, and it should be collectively shared by individual scientists and researchers, data stewards, research institutions, and funding organizations.

Article writing Trivia

I’ve been working on some parts of this paper with Andrea Berez-Kroeker since 2013. Andrea has been working on some of the central ideas in this paper for even longer. Academia is a slow game sometimes, but this paper says a lot of things that a lot of the authors have been thinking about for a long time.

A lot of ideas in this paper formed the initial motivation for the Austin Principles of Data Citation in Linguistics

There is a reference to work on bacteria in human navels as an example of replicable research because I had read about Carl Zimmer’s participation in this research on his blog a couple of years early and it amused me greatly.

Abstract

This paper is a position statement on reproducible research in linguistics, including data citation and attribution, that represents the collective views of some 41 colleagues. Reproducibility can play a key role in increasing verification and accountability in linguistic research, and is a hallmark of social science research that is currently under-represented in our field. We believe that we need to take time as a discipline to clearly articulate our expectations for how linguistic data are managed, cited, and maintained for long-term access.

See also

Putting practice into words: New paper out about methods and data in descriptive grammar writing

Linguistics Data Interest Group - New RDA group to improve data citation and transparency

The Austin Principles of Data Citation in Linguistics

Reference

The Austin Principles of Data Citation in Linguistics (Berez-Kroeker et al. 2017)

The Austin Principles of Data Citation in Linguistics

The Linguistics Data Interest Group in the Research Data Alliance have spent the last year working on a document that explains why it’s so important for linguists to make their data ‘citable’. Linguistics builds itself as a discipline that uses empirical evidence to make claims, but so often the reader doesn’t have access to this underlying data to verify the claims (or test their own ideas!). The Austin Principles of Data Citation in Linguistics provide the reasons that citation should be central to all linguistic work. Even if you work with introspection, you may not have data to cite, but making clear that the data come from in introspection is still an important, and often omitted, piece of information.  

If you’re a linguist, you can endorse the principles

Even if you’re only just starting your linguistic research career, you can add your name to the list of linguists who endorse the principles. You can also use the principles to start conversations about data citation and management in your lab, class, department, linguistics society, or journal.

If you would like to help shape the future of linguistic data, you can also join us in the RDA Linguistics Data Interest Group (LDIG). This post from April explains what the LDIG is, and why it’s important.

From the Preamble of the Austin Principles:

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The FORCE11 Joint Declaration of Data Citation Principles* state that “[s]ound, reproducible scholarship rests upon a foundation of robust, accessible data. For this to be so in practice as well as theory, data must be accorded due importance in the practice of scholarship and in the enduring scholarly record. In other words, data should be considered legitimate, citable products of research. Data citation, like the citation of other evidence and sources, is good research practice and is part of the scholarly ecosystem supporting data reuse.”

The FORCE11 Joint Declaration of Data Citation Principles is intentionally broad so to be as inclusive of data from as many scientific disciplines as possible. This document, the Austin Principles of Data Citation in Linguistics, interprets the FORCE11 document to address linguistic data specifically. These guiding principles have been created to enable linguists to make decisions about their data that ensure it is as accessible and transparent as possible. Some subfields of linguistics may already have specific guidelines for data citation; in these cases the Austin Principles can supplement extant guidelines to ensure that data citation conforms with current best practices.

What’s next for the Linguistics Data Interest Group

The Austin Principles of Data Citation are the first major output for the LDIG, it focuses on the why of data citation. We’re now focusing on the how of data citation. This includes sharing the Austin Principles, developing more training on research data management, and also working with researchers, archives and publishers to develop data citation standards.

Reference