Write a Blog >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

ICSE 2019 Artifacts

According to ACM’s "Artifact Review and Badging" policy, an "artifact" is a "digital object that was either created by the authors to be used as part of the study or generated by the experiment itself." Artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results. The review of artifacts of accepted research papers increases the likelihood that results can be independently replicated and reproduced by other researchers.

In this spirit — and for the first time at ICSE — the artifacts track aims to review, promote, share and catalog the research artifacts of papers accepted to the research track.

Call For Artifact Submissions

Authors of papers accepted to the Technical Track are invited to submit an artifact to the ICSE Artifact Track. If the artifact is accepted it will receive one of the following badges on the front page of their paper and in the proceedings:

Functional Reusable Available Replicated Reproduced
No Badge
Artifacts documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation Functional + very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to. Functional + Placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided. Available + main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author. Available + The main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.

Papers with such badges contain reusable products that other researchers can use to bootstrap their own research. Experience shows that such papers earn increased citations and greater prestige in the research community. Artifacts of interest include (but are not limited to) the following.

  • Software, which are implementations of systems or algorithms potentially useful in other studies.

  • Data repositories, which are data (e.g., logging data, system traces, survey raw data) that can be used for multiple software engineering approaches.

  • Frameworks, which are tools and services illustrating new approaches to software engineering that could be used by other researchers in different contexts.

This list is not exhaustive, but if your proposed artifact is not on this list, please email the chairs before submitting.

Selection Criteria

The ICSE artifacts track will be evaluated using the criteria in the last row of the above table.

The goal of this track is to encourage reusable research products. Hence, no “functional” badges will be awarded. Note that for the badges “replicated” and “reproduced”, authors will need to offer appropriate documentation that their artifacts have reached that stage. So it can be anticipated that most of our artifacts will be “reusable” and “available”.

Submission Process

The configuration and installation for your artifact should take less than 30 minutes or it is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it.

If you envision difficulties, please provide your artifact in the form of a VirtualBox machine image or a Docker container image. Whichever the case, your artifact should be made available as a link to a public repository or to a single archive file using a widely available compressed archive format such as ZIP (.zip), tar and gzip (.tgz), or tar and bzip2 (.tbz2).

Prepare all the supporting materials needed to execute an artifact:

  • a README main file describing what the artifact does;

  • A STATUS file stating what kind of badge you are applying for (one of reusable, available, replicated, reproduced) as well as the reasons why you think your artifact deserves that badge.

  • a LICENSE file describing the distribution rights (and note that to score “available” or higher, then that license needs to be some form of open source license).

  • an INSTALL file with installation instructions. These instructions should include notes on what output to expect that confirms

    • the code is installed and working; and
    • the code is doing something interesting and useful.
  • Enough associated code and data such that some CS person with a reasonable knowledge of scripting, build tools, etc. could install, build, run your code.

Go to the submission site and fill in a submission form, which will ask you to include a link to your zip file and a link to your accepted paper.