Homework 1 for Computer Graphics¶

What Thesis I Choose¶

Code Replicability in Computer Graphics

Nicolas Bonneel, David Coeurjolly, Julie Digne, Nicolas Mellado

ACM Transactions on Graphics (Proceeding of SIGGRAPH 2020, 39:4)

Assess replicability in Computer Graphics, by evaluating whether the code is available and whether it works properly.
Develop a website which aims at helping practitioners run existing codes on current hardware and software generations, with build instructions for 151 codes found online.

Classify each paper into a single category (i.e., Rendering, Animation and Simulation, Geometry, Images, Virtual Reality, Fabrication) to simplify analyses.
Share the task of evaluating 374 submissions across 4 full-time tenured researchers (authors of the paper), largely experienced in programming and running complex computer graphics systems. Reasonable efforts were made to find and compile the provided code, including retrieving outdated links from the WayBack Machine, recreating missing Makefiles, debugging, trying on multiple OS (compiling was tested on Windows 10, Debian Buster, Ubuntu 18.04 and 19.10 and MacOS 10.151 ), or adapting the code to match libraries having evolved.
For papers without published code, this includes information as to whether the paper contains explicit algorithms and how much effort is deemed required to implement them (on a scale of 1 to 5). For algorithms requiring little reimplementation effort (with a score of 5) – typically for short shaders or short self-contained algorithms – this can give an indication as to why releasing the code was judged unnecessary. For papers containing code, we evaluate how difficult it was to replicate results through a number of questions on a scale of 1 to 5.

Code sharing is correlated with paper citation count, and has improved over time.
Code sharing practices largely vary with sub-fields of computer graphics.
It is often not enough to share code for a paper to be replicable. Build instructions with precise dependencies version numbers as well as example command lines and data are important.