Artifact EvaluationICSE 2019

According to ACM’s “Artifact Review and Badging” policy, an “artifact” is a “digital object that was either created by the authors to be used as part of the study or generated by the experiment itself.” Artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results. The review of artifacts of accepted research papers increases the likelihood that results can be independently replicated and reproduced by other researchers.

In this spirit — and for the first time at ICSE — the artifacts track aims to review, promote, share and catalog the research artifacts of papers accepted to the research track. The top two artifacts selected by the program committee will be awarded the best artifact awards.

Call For Artifact Submissions

PDF Version

Authors of papers accepted to the Technical Track are invited to submit an artifact to the ICSE Artifact Track. If the artifact is accepted it will receive one of the following badges on the front page of their paper and in the proceedings:

Functional	Reusable	Available	Replicated	Reproduced
No Badge
Artifacts documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation	Functional + very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.	Functional + Placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.	Available + main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.	Available + The main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.

Papers with such badges contain reusable products that other researchers can use to bootstrap their own research. Experience shows that such papers earn increased citations and greater prestige in the research community. Artifacts of interest include (but are not limited to) the following.

• Software, which are implementations of systems or algorithms potentially useful in other studies.

• Data repositories, which are data (e.g., logging data, system traces, survey raw data) that can be used for multiple software engineering approaches.

• Frameworks, which are tools and services illustrating new approaches to software engineering that could be used by other researchers in different contexts.

This list is not exhaustive, so the authors are asked to email the chairs before submitting if their proposed artifact is not on this list.

Evaluation Criteria

The ICSE artifact evaluation track uses a single-blind review process. Artifacts will be evaluated using the criteria in the last row of the above table.

The goal of this track is to encourage reusable research products. Hence, no “functional” badges will be awarded. Note that for the badges “replicated” and “reproduced” authors will need to offer appropriate documentation that their artifacts have reached that stage. So it can be anticipated that most of the artifacts will be “reusable” and “available”.

Best Artifact Awards

There will be two ICSE 2019 Best Artifact Awards to recognize the effort of authors creating and sharing outstanding research artifacts.

Artifact Submission Process

Authors of the papers accepted to the Technical Track must perform the following steps to submit an artifact:

1. Preparing the artifact
2. Making the artifact available
3. Documenting the artifact
4. Submitting the artifact

1. Preparing the Artifact

There are two options depending on the nature of the artifacts: Installation Package or Simple Package. In both cases, the configuration and installation for the artifact should take less than 30 minutes. Otherwise the artifact is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it.

Installation Package. If the artifact consists of a tool or software system, then the authors need to prepare an installation package so that the tool can be installed and run in the evaluator’s environment. Provide enough associated instruction, code, and data such that some CS person with a reasonable knowledge of scripting, build tools, etc. could install, build, and run the code. If the artifact contains or requires the use of a special tool or any other non-trivial piece of software the authors must provide a VirtualBox VM image or a Docker container image with a working environment containing the artifact and all the necessary tools.

We expect that the artifacts have been vetted on a clean machine before submission.

Simple Package. If the artifact only contains documents which can be used with a simple text editor, a PDF viewer, or some other common tool (e.g., a spreadsheet program in its basic configuration) the authors can just save all documents in a single package file (zip or tar.gz).

2. Making the Artifact Available

The authors need to make the packaged artifact (installation package or simple package) available so that the Evaluation Committee can access it. We suggest a link to a public repository or to a single archive file in a widely available archive format. If the authors are aiming for the badges “available” and beyond the artifact needs to publicly accessible. In other cases, the artifacts do not necessarily have to be publicly accessible for the review process. In this case, the authors are asked to provide a private link or a password-protected link.

3. Documenting the Artifact

The authors need to write and submit a documentation explaining how to obtain the artifact package, how to unpack the artifact, how to get started, and how to use the artifacts in more detail. The artifact submission must only describe the technicalities of the artifacts and uses of the artifact that are not already described in the paper. The submission should contain the following documents (in plain text or pdf format) in a zip archive:

• A README main file describing what the artifact does and where it can be obtained (with hidden links and access password if necessary). Also, there should be a clear description how to reproduce the results presented in the paper.
• A STATUS file stating what kind of badge(s) the authors are applying for as well as the reasons why the authors believe that the artifact deserves that badge(s).
• A LICENSE file describing the distribution rights. Note that to score “available” or higher, then that license needs to be some form of open source license.
• An INSTALL file with installation instructions. These instructions should include notes illustrating a very basic usage example or a method to test the installation. This could be, for instance, on what output to expect that confirms that the code is installed and working; and the code is doing something interesting and useful.
• A copy of the accepted paper in pdf format.

4. Submitting the Artifact

By January 13, 2019 register your research artifact at the ICSE EasyChair site by submitting an abstract describing your artifact. The abstract should include the paper title, the purpose of the research artifact, the badge(s) you are claiming, and the technology skills assumed by the reviewer evaluating the artifact. Please also mention if running your artifact requires specific Operation Systems or other environments.

By January 27, 2019 complete your artifact submission by uploading the zip archive containing the documentation.

Before the actual evaluation reviewers will check the integrity of the artifact and look for any possible setup problems that may prevent it from being properly evaluated (e.g., corrupted or missing files, VM won’t start, immediate crashes on the simplest example, etc.). The Evaluation Committee may contact the authors within the rebuttal period to request clarifications on the basic installation and start-up procedures or to resolve simple installation problems. Authors are informed of the outcome and, in case of technical problems, they can help solve them during a brief author response period. Given the short review time available, the authors are expected to respond within a 48-hour period. Authors may update their research artifact after submission only for changes requested by reviewers in the rebuttal phase. Author submitting an open source repository link, are expected to give a tag to time-stamp your submission.

Important Dates

December 12, 2018: ICSE technical paper notification

January 13, 2019: Artifact registration deadline (mandatory)

January 27, 2019: Artifact submission deadline

February 13, 2019: ICSE camera ready deadline

February 17, 2019: End of rebuttal period

February 28, 2019: Artifact notification