Practicing Open Science

What is Open Science?

The goal of Open Science is to make scientific research and its dissemination accessible to all levels of society. By doing so, Open Science has the potential to make the scientific process more transparent, inclusive, and democratic. It is about making data and other resources, such as code, surveys, or study stimuli, findable, accessible, interoperable, and reusable (FAIR principles). This is the only way to ensure that experiments can be reproduced and artifacts can be reused by other researchers.

Learn more about open science and the FAIR principles here (What is open science?) and here (FAIR Principles)

Why Open Science at AutoUI?

While the benefits of open science and open data practices are well understood, the AutoUI community has been slow to adapt. The proportion of papers that are open access and share their data, code, and other research artifacts is only slowly increasing. As a community, we want to change this and accelerate the practice of open science by providing information and guidance to authors on how to publish research artifacts alongside their papers, and how to ensure that these artifacts are useful to everyone.

How to Achieve the ACM Artifact Available Badge?

Any author who makes artifacts available on a FAIR platform (see below) will receive an Artifact Available badge for their AutoUI 2024 paper.  The Artifact Available badge asserts that author-created artifacts relevant to the publication are available in an archival repository. These badges are reviewed by the Open Data Chairs, but do not need to be comprehensive in how they support the associated publication. Papers with the Artifact Available badge include this icon in the ACM Digital Library, are searchable and filterable by this badge, and have this metadata embedded within the article landing page to improve discoverability.

When you submit your material for AutoUI 2024, you will see a new field on the submission form asking if you want your paper to be considered for the Artifact Available badge. Similar to CHI 2024, we will support Video, Audio, Software, Datasets, and Other artifact types.

Audio

You can upload audio files as artifacts that support your publication.  This could be recordings of sounds critical to your paper, audio recordings of oral data, or any other audio that underpins your publication.

Datasets

You can upload datasets that support your publication.  This could be logs from empirical work, data used in your analysis, transcripts, or any other data that underpins your publication.

Video

You can upload additional video artifacts, which are different to the video presentation and video preview you can already provide.  This could be a video demonstration of a hardware prototype, a video walkthrough of a virtual world, or any other videos that underpin your publication.

Software

You can upload the software that supports your publication.  This could be experimental software, software that runs simulations, code that reproduces analysis, or any other software artifact that underpins your publication.

Other

There are many artifacts not covered by the current categories supported by ACM, but that are still valuable to include with your publication.  This could be survey and interview protocols, details of experiment pre-registration, technical diagrams like CAD files, 3D models, or any other artifact that underpins your publication.

How to Open Science?

Requirements for the Platform to Publish Artifacts

Irrespective of the nature of the published data, all articles must be accessible freely and publicly through a digital repository at the time of submission. The chosen platform should meet the following criteria (inspired by the HRI 2024 Short Contributions https://humanrobotinteraction.org/2024/short-contribution):

  1. Endorsement and recognition within the AutoUI community.
  2. Provision of permanent identifiers for submitted artifacts (e.g., accession numbers, DOIs).
  3. Supply of metadata and other documentation to facilitate the use and reusability of data within a designated archive.
  4. Assurance of long-term preservation of artifacts and associated documentation.
  5. Permitting peer reviewers to access the data anonymously for evaluation purposes.
  6. Implementation of active pursuit of implementing the FAIR principles (refer to recommended platforms below).

Facilitating access to data without unnecessary restrictions, and in cases where restrictions are necessary, providing clear criteria and mechanisms for access.

On Which Platforms to Publish My Artifacts?

There are many places where we can publish open access artifacts. It is good practice to keep your data on a FAIR platform. Here are some examples (updated February 2024):

  1. ACM Digital Library (FAIR, https://dl.acm.org). AutoUI is an ACM conference, and the platform allows the sharing of (small) supplementary material with the publication.
  2. OSF (FAIR, https://osf.io). Allows to make repos anonymous for review, which can be handy for the submission process. It is a ‘swiss knife’ for publishing artifacts in open access, and the platform allows a great level of flexibility. File versioning is straightforward, and OSF also allows you to pre-register your study. It is also easy to make your material anonymous for the review process.
  3. Zenodo (FAIR, https://zenodo.org). Data is stored at CERN, which is reliable. It is becoming a common place to share data, which may make it user-friendly. They accept up to 50GB per dataset with an option to have multiple datasets.
  4. GitHub (not FAIR, https://github.com). Everyone knows how to navigate around a repo on GitHub, which can be an advantage for outreach. But it is not FAIR. What can be practiced is to provide a link to the active project in the manuscript, and then still upload materials to a FAIR platform for reproducibility.
  5. 4TU.ResearchData (FAIR, https://data.4tu.nl). Local storage for the technical universities of the Netherlands. They do quite a good job of making the management of datasets easy. Likely, there is a similar portal in your institution/country.

Anonymization for peer review

It is important to keep in mind that the submission at AutoUI is anonymized. Hence not only the manuscript but also all data shared should be anonymized as well. The platforms for publishing open artifacts usually give you support in achieving that (e.g., OSF allows making repos anonymous for the review stage).