#ldig

This new commentary article is a response to a target article from Laura Becker and Matías Guzmán Naranjo titled "Replication and methodological robustness in quantitative typology".

In this paper, Becker and Guzmán Naranjo explore what happens when typological work is reproduced by another team, and what the differences in results mean for the field of linguistic typology.

I wrote this article alongside my former co-chairs of Linguistic Data Interest Group (LDIG) of the Research Data Alliance, reflecting on the work needed to ensure that we are all doing the kind of research that is useful to a more transparent way of doing work in linguistic typology.

From the commentary:

We thank B&GN for their work, but we also thank Dryer (2018), Seržant (2021), Shcherbakova et al. (2023) and Berg (2020), for doing work that could be reproduced (a benchmark much scholarship falls short on) and subsequently having their work scrutinised and evaluated in this way. As B&GN note, “[t]here is no specific reason for choosing these papers other than the fact that the authors made their datasets available”. To work in an open and transparent manner is to open yourself to critical evaluation. Linguistic typology advances because of researchers who have created accessible data as part of their work. This includes individual researchers working on specific languages, whose data is the basis of typological work, as well as those typologists who share the databases of their work. Open ways of working require open mindedness from the whole research community.

Reference

Gawne, L., H.N. Andreassen, L. Ferrara, A.L. Berez-Kroeker (2025). Open research requires open mindedness: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo. Linguistic Typology. doi: 10.1515/lingty-2025-0018

Related links

Linguistic Data Interest Group: Five years of improving data citation practices in linguistics

New Paper: Reproducible research in linguistics: A position statement on data citation and attribution in our field

New Article Published: Reflections on reproducible research, in Reflections on Language Documentation 20 Years after Himmelmann 1998

After five years with the Linguistics Data Interest Group of the Research Data Alliance, I’ve stepped down as a co-chair of the group. I wanted to use this as a chance to collate some of the work I was involved with over the last five years. I’ll still be a member of the LDIG for its next chapter, and thrilled that Andrea Berez-Kroeker, Helene N. Andreassen, and Lindsay Ferrara will be heading things up. And, of course, the group has many excellent members (if you’re a linguist and/or interested in data management, you can join the LDIG too!).

The Austin Principles of Data Citation in Linguistics

The Austin Principles of Data Citation were the first major output for the LDIG, it focuses on the why of data citation. This short document is a position statement on the importance of data in linguistic work. From the preamble:

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The official Austin Principles website

The Superlinguo post

The Trømso Recommendations in academic publishing

If the Austin Principles are the why the Tromsø Recommendations are the how.

The Tromsø Recommendations provide clear guidance for data citation for referencing language data, both in the bibliography and in the text of linguistics publications. The recommendations have been written to account for the rich variety of linguistic data, and include clear guidance and examples.

The official Tromsø Recommendations documents

The Superlinguo post

Building uptake for the Trømso Recommendations

Now that the Tromsø Recommendations have been published, there’s an ongoing campaign to normalise their use in academic publishing and grant writing. Get involved by encouraging your favourite publishers to include to Tromsø Recommendations in their author guidelines!

The Linguistics Data Interest Group (a working group in the Research Data Alliance) have developed the Tromsø Recommendations in collaboration with linguists working in a range of disciplines. The next step is to help encourage citation of data by encouraging journals to include the Tromsø Recommendations in their instructions for authors.

bit.ly/trecs-campaign

Publications

Alongside LDIG colleagues I was involved in a number of publications looking at the role of data in linguistics. Below are links to the Superlinguo posts with more information, the abstract and links to open access versions.

Reproducible research in linguistics: A position statement on data citation and attribution in our field

Situating Linguistics in the Social Science Data Movement. Chapter in the Open Handbook of Linguistic Data Management

Data transparency and citation in the journal Gesture

Reflections on reproducible research, in Reflections on Language Documentation 20 Years after Himmelmann 1998

Putting practice into words: The state of data and methods transparency in grammatical descriptions

The Tromsø recommendations for citation of research data in linguistics

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The Tromsø Recommendations provide clear guidance for data citation for referencing language data, both in the bibliography and in the text of linguistics publications. The recommendations have been written to account for the rich variety of linguistic data, and include clear guidance and examples.

The Linguistics Data Interest Group (LDIG, a working group in the Research Data Alliance) have developed the Tromsø Recommendations in collaboration with linguists working in a range of disciplines.

Making the Tromsø Recommendations part of academic publishing

The next step is to help encourage citation of data by encouraging journals to include the Tromsø Recommendations in their instructions for authors.

The LDIG has launched a campaign to encourage publishers, archives and other groups that present linguistic data to adopt the Tromsø Recommendations.

What would it look like to include the Tromsø Recommendations in academic publishing?

The Tromsø Recommendations cover all types of linguistic data, and have citation formats for all levels of detail - from a whole corpus to a single line of a text. They are designed to focus on content, not style, so they work with the formatting for a particular journal.

To include the recommendations can be as minimal as adding a line of text like this to a publication’s Information for Authors page:

<Journal Name> [encourages|requires] the citation of linguistic data in all published articles. Citation structure should follow the The Tromsø Recommendations for Citation of Research Data in Linguistics. For more on the importance of data citation see The Austin Principles of Data Citation in Linguistics.

How can I help support the Tromsø Recommendations in academic publishing?

If you are a journal or series editor, member of a journal editorial board or publisher, update your author instructions to encourage authors to include data citation.

If you are a contributor to a journal in your field or popular publisher, write to the editors and ask them to consider adopting the Tromsø Recommendations. We have some suggested text for your email or letter in the next tab.

If you are a member of a PhD program, a funding body or a body that gives a linguistics award, consider adopting data citation as a category for assessment.

If you are a linguist or language scientist, encourage the journals, societies, funding and awards bodies you are a member of to take up the Tromsø Recommendations.

For this last group, we’ve created a spreadsheet that has email templates, background information and a spreadsheet where we are tracking which journals have been contacted, and whether they’ve adopted the Tromsø Recommendations: bit.ly/trecs-campaign

You do not have to be an LDIG or RDA member to be involved! Also, if the journals you publish in do not immediately adopt the Tromsø Recommendations for the whole publication, you can always adopt them yourself for the next thing you publish!

See also:

The Austin Principles of Data Citation in Linguistics

Reproducible research in linguistics: A position statement on data citation and attribution in our field (article)

Data transparency and citation in the journal Gesture (article)

Putting practice into words: The state of data and methods transparency in grammatical descriptions (article)

In 1998 Nikolas Himmelmann wrote "Documentary and descriptive linguistics”, an article for the journal Linguistics (I’m not going to link because it’s paywalled, but you should be able to find a copy floating around online). Himmelmann 1998 is an often cited work that makes a case for the work of documentation (collecting data) being as important as the work of description (describing data) in understanding how languages work.

To look at the 20 years since this publication, Bradley McDonnell, Andrea L. Berez-Kroeker and Gary Holton brought together over 30 linguists to reflect on how the field has changed and grown in that time. This is a Special Publication of Language Documentation & Conservation, available open access.  

From the introduction to the special issue:

[W]e invited 38 experts from around the world to reflect on either particular issues within the realm of language documentation or particular regions where language documentation projects are being carried out. The issues addressed in this volume represent a broad and diverse set of topics from multiple perspectives and for multiple purposes that continue to be relevant to documentary linguists and language communities. Some topics have been hotly debated over the past two decades, while others have emerged more recently.

All of the chapters are short and to the point (they’re around 10 pages each). I has a great time writing with Andrea Berez-Kroeker about how language documentation has changed over the last two decades in regards to data management and sharing. Himmelmann talks about the importance of sharing primary data, but the nature of that data, and how we can share it has changed so much since 1998.

Abstract

Reproducibility in language documentation and description means that the analysis given in descriptive publication is presented in a way that allows the reader to access the data on which the claims are based, to verify the analysis for themself. Linguists, including Himmelmann, have long pointed to the centrality of documentation data to linguistic description. Over the twenty years since Himmelmann’s 1998 paper we have seen a growth in digital archiving, and the rise of the Open Access movement. Although there is good infrastructure in place to make reproducible research possible, few descriptive publications clearly link to underlying data, and very little documentation data is publicly accessible. We discuss some of the institutional roadblocks to reproducibility, including a lack of support for the development of published primary data. We also look at what work on language documentation and description can learn from the recent replication crisis in psychology.

Reference

I’m so excited to share this paper on data citation in linguistics, published in Linguistics. Not just because it was co-authored with 13 other linguists, including some of my linguistics heroes, but because it articulates so much of what we need to do to improve the way linguists think about data. (it’s Open Access, so you can read/download and share as you wish!)

I’m often reminded of something Nicholas Evans likes to say:

“Linguistics has a unique but unfulfilled destiny, as the most scientific of the humanities and the most humanistic of the sciences.”

It captures what I love about linguistics, but I think it also explains why we have a complicated relationship with the idea of data, and its management. This position paper outlines why it is important that linguists think about the data on which linguistic claims are made.

My Favourite line

Linguistic data are the very building blocks of our field. Given that linguistic theories need to be borne out through data, we believe that linguistic data are important resources in their own right and represent valuable assets for the field. Therefore, our field needs to accept responsibility for the proper documentation, preservation, attribution, and citation of these assets. The responsibility to do so is an integral part of linguistic research, and it should be collectively shared by individual scientists and researchers, data stewards, research institutions, and funding organizations.

Article writing Trivia

I’ve been working on some parts of this paper with Andrea Berez-Kroeker since 2013. Andrea has been working on some of the central ideas in this paper for even longer. Academia is a slow game sometimes, but this paper says a lot of things that a lot of the authors have been thinking about for a long time.

A lot of ideas in this paper formed the initial motivation for the Austin Principles of Data Citation in Linguistics

There is a reference to work on bacteria in human navels as an example of replicable research because I had read about Carl Zimmer’s participation in this research on his blog a couple of years early and it amused me greatly.

Abstract

This paper is a position statement on reproducible research in linguistics, including data citation and attribution, that represents the collective views of some 41 colleagues. Reproducibility can play a key role in increasing verification and accountability in linguistic research, and is a hallmark of social science research that is currently under-represented in our field. We believe that we need to take time as a discipline to clearly articulate our expectations for how linguistic data are managed, cited, and maintained for long-term access.

See also

Putting practice into words: New paper out about methods and data in descriptive grammar writing

Linguistics Data Interest Group - New RDA group to improve data citation and transparency

The Austin Principles of Data Citation in Linguistics

Reference

The Austin Principles of Data Citation in Linguistics (Berez-Kroeker et al. 2017)

The Austin Principles of Data Citation in Linguistics

The Linguistics Data Interest Group in the Research Data Alliance have spent the last year working on a document that explains why it’s so important for linguists to make their data ‘citable’. Linguistics builds itself as a discipline that uses empirical evidence to make claims, but so often the reader doesn’t have access to this underlying data to verify the claims (or test their own ideas!). The Austin Principles of Data Citation in Linguistics provide the reasons that citation should be central to all linguistic work. Even if you work with introspection, you may not have data to cite, but making clear that the data come from in introspection is still an important, and often omitted, piece of information.  

If you’re a linguist, you can endorse the principles

Even if you’re only just starting your linguistic research career, you can add your name to the list of linguists who endorse the principles. You can also use the principles to start conversations about data citation and management in your lab, class, department, linguistics society, or journal.

If you would like to help shape the future of linguistic data, you can also join us in the RDA Linguistics Data Interest Group (LDIG). This post from April explains what the LDIG is, and why it’s important.

From the Preamble of the Austin Principles:

Data is central to empirical linguistic research. Linguistic data comes in many different forms, and is collected and processed with a wide range of methods. Data citation recognizes the centrality of data to research. Furthermore, it facilitates verification of claims and repurposing of data for other studies.

The FORCE11 Joint Declaration of Data Citation Principles* state that “[s]ound, reproducible scholarship rests upon a foundation of robust, accessible data. For this to be so in practice as well as theory, data must be accorded due importance in the practice of scholarship and in the enduring scholarly record. In other words, data should be considered legitimate, citable products of research. Data citation, like the citation of other evidence and sources, is good research practice and is part of the scholarly ecosystem supporting data reuse.”

The FORCE11 Joint Declaration of Data Citation Principles is intentionally broad so to be as inclusive of data from as many scientific disciplines as possible. This document, the Austin Principles of Data Citation in Linguistics, interprets the FORCE11 document to address linguistic data specifically. These guiding principles have been created to enable linguists to make decisions about their data that ensure it is as accessible and transparent as possible. Some subfields of linguistics may already have specific guidelines for data citation; in these cases the Austin Principles can supplement extant guidelines to ensure that data citation conforms with current best practices.

What’s next for the Linguistics Data Interest Group

The Austin Principles of Data Citation are the first major output for the LDIG, it focuses on the why of data citation. We’re now focusing on the how of data citation. This includes sharing the Austin Principles, developing more training on research data management, and also working with researchers, archives and publishers to develop data citation standards.

Reference

Descriptive grammars take a lot of time to write, often based on years of fieldwork by the author. For many languages they are the only major source of information we have about how the language works. Although a grammar takes a lot of effort to produce, it’s not often made clear where the data come from, who the author worked with, or whether the data can be accessed by other people. Barbara Kelly, Andrea Berez-Kroeker, Tyler Heston and I decided to explore just how people are talking about their method of work and their research data in descriptive grammars. We have all done work in language documentation and description, and felt like there was a gap in how linguists talked about their work with each other and what they wrote in published grammars. This work is now published in the latest volume of Language Documentation & Conservation.

A survey of 50 published grammars and 50 dissertations

We looked at 100 grammars published between 2003 and 2012. We assessed how transparent they were in their data collection methods, looking at whether they explicitly mentioned things like how many people they worked with, and the recording equipment used. We also looked at whether researchers link from the grammar to the original example. This is important because it means that it’s possible to revisit the original recording or example.

Some of the findings:

Fewer than a third of grammars mentioned the equipment used to collect data or software used to analyse it

For over 2/3rds of the grammars we have no idea where the data used in the analysis is now

Over half of the grammars do not include any metadata for the examples, meaning it’s unclear where they came from, if they’re elicited or whether they were written or recorded.

We can be optimistic about the future; dissertations performed better than published grammars, suggesting that a newer generation of scholars are aiming for greater transparency in their research

We don’t think linguists are doing bad research, we just think that we need to make the work we are doing clearer.

Abstract

Language documentation and description are closely related practices, often performed as part of the same fieldwork project on an un(der)-studied language. Research trends in recent decades have seen a great volume of publishing in regards to the methods of language documentation, however, it is not clear that linguists' awareness of the importance of robust data-collection methods is translating into transparency about those methods or data citation in resultant publications. We analyze 50 dissertations and 50 grammars from a ten-year span (2003-2012) to assess the current state of the field. Publications are critiqued on the basis of transparency of data collection methods, analysis and storage, as well as citation of primary data. While we found examples of transparent reporting in these areas, much of the surveyed research does not include key information about methodology or data. We acknowledge that descriptive linguists often practice good methodology in data collection, but as a field we need to build a better culture with regard to making this clear in research writing. Thus we conclude with suggested benchmarks for the kind of information we believe is vital for creating a rich and useful research methodology in both long and short format descriptive research writing.

Reference

A really big part of linguistic research is having data. Whether it’s a language documentation corpus, a set of experiments, or your own intuitions about how language works, this is all data on which analyses and theories are built.

While linguists have always relied on data, we’re not the best at making clear where the data we are talking about come from. There has been little incentive or support for how to do that. But it is important. Making it clear where your data come from, and giving other people access to it helps make linguistic research more reproducible. 

I think that it’s important that we encourage linguists to cite where their data come from, and make that data more easily accessible to other researchers. That is why I’ve joined an awesome group of researchers to create the Linguistics Data Interest Group (LDIG) as part of the Research Data Alliance. I’m a co-chair of the group (!!!) along with Andrea L. Berez-Kroeker (U Hawai‘i), Susan S. Kung (U Texas) and Helene N. Andreassen (UiT The Arctic University). If you’re a linguist who works with data (i.e. any linguist) and you think that it’s important that we strive to do the best kind of science we can, then you’re welcome to join the LDIG through joining the RDA (it’s free to join, and puts you in a really great network).

I’m especially hoping that some people who are doing their PhD or PostDoc will join, because we’re often at the front line of data collection and management.

From the LDIG announcement:

The Linguistics Data Interest Group (RDA)  has been established through the Research Data Alliance (RDA) and aims to develop the discipline-wide adoption of common standards for data citation and attribution. In our parlance citation refers to the practice of identifying the source of linguistic data, and attribution refers to mechanisms for assessing the intellectual and academic value of data citations. The LDIG aims to encourage an international discussion of these topics, bolstering discussions that are already happening in specific sub-disciplines of linguistics in different countries.  

The LDIG is for people who work with linguistic and language data. This work includes, but is not limited to, the collection, management and analysis of linguistic data. We encourage participation from academic and speaker communities.