Triple-masking v Double-masking: a Trial of Scientific Publication in Public Health

Aim:

To determine whether masking (i.e., triple-masked: editors, authors, reviewers) or not (i.e., double-masked: authors, reviewers) the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers to the editors leads to a greater proportion of accepted manuscripts compared with double-masked review. Additionally, we will conduct exploratory analyses to determine whether the differences are related to the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers.

Trial design: two group, parallel, superiority trial with 1:1 allocation

Study setting: The trial will solely take place at the American Journal of Public Health in a virtual capacity. All submissions will be electronic via the Editorial Manager (https://www.editorialmanager.com/ajph/default2.aspx). Eligible manuscripts in the blinded editorial process arm will be masked electronically before being entered in the trial by the editorial assistant, who will also place all manuscripts in Dropbox folders to be accessed by the editors. The questionnaire that will be sent to all authors, reviewers, and editors is housed in Survey Monkey and will be administered via email and/or text messaging by someone external to the trial.

Inclusions: All eligible, new submissions (research, notes from the field, non-commissioned editorials, analytic essays, systematic reviews) for the regular issue and supplements from September 5, 2023 to March 1, 2024.

Exclusions: Commissioned editorials, book reviews, Letters to the Editor, Public Health of Consequence editorials, Editor's Choice, and re-submissions

Randomization procedure: 1:1 allocation of eligible articles to either triple-masked or usual process in the order of submission, after exclusion of non-eligible submissions by the editorial assistant.

Planned generation: Randomization, stratified by three article types (ie, Research Articles, Analytic Essay, Editorial & Notes from the Field), by blocks of random sizes to balance articles by type over time, and to conceal allocation order.

Administration/implementation: Editorial assistant allocates the submissions following computer generated allocation orders for each of the three article types.

Masking securities: Editors will be restricted from accessing the Editorial Manager for the duration of the project. Additionally, since articles won't be assigned to editors, they won't be able to see anything from their immediate accounts. This should block them from the masked information and preserve the integrity of the project. Editors will be able to search for reviewers with specific expertise from an external, cleaned nominative version of the Editorial Manager database. They won't however know who, from the reviewers selected, accepted the invitation and wrote the review. In this sense, reviews' authorship will be masked to the editor.

Procedures:

1. Questionnaire: All authors of new, eligible submissions will receive an email from a research assistant external to the trial that will explain the on-going trial. The email will invite them to fill out an optional, short, Survey Monkey questionnaire asking for age, gender, race, ethnicity, career position, and institution location. The questionnaire data will be collected in a stand-alone database, handled by the research assistant. Missing value are expected to be equally distributed in both arms of the study. However, for secondary analysis they may be correlated with the outcome. The same questionnaire will be sent but only once to the editors.

Primary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are accepted for publication (i.e., not rejected at any stage); risk difference, unadjusted

Secondary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are sent for peer review (i.e., not desk rejected); risk difference, unadjusted

Proportion accepted in the double-masked group: ~10% Clinically meaningful difference: 5% Framework: Superiority N= 1500 (750 manuscripts in each group), 80% power to detect a difference of 4.8% (RR=1.48) in acceptance rate with 5% type I error for a two-sided test. This translates to an acceptance rate of 14.8% for the triple=masked. Additionally, the trial is projected to have 84% power to detect a difference of 5% (RR=1.5), and 90% power to detect a difference of 5.5% (RR=1.55) while remaining a 5% type I error for a two-sided test.

The statistical power to detect differences if they exist in subgroups defined by article type or authors identities will be very small because of the small numbers of articles. It is expected that the proportion of articles with a diverse authorship will not exceed one third of the submissions in each groups.

Ethics committee The protocol has been exempted from institutional review board by the IRB of Queens College CUNY, where the editor in chief and PI of the supporting grant has his primary appointment (Appendix B).