Recently there's been an increasing amount of excitement around industrial tracks. They are becoming well attended, competitive, and discussion-worthy. Alas, this recent enthusiasm could be quickly extinguished if we, the program committees of these industrial tracks, provide inappropriate or low quality reviews. Imagine the frustration of a practitioner if their real world case study gets rejected while an experiment with ten freshman undergraduate students is accepted. To avoid this, we need to provide clear, distinct guidelines for evaluating industry track papers.
An industry track focuses on the same topics and expects the same rigor as the corresponding research track, yet papers featured in the industry track are distinct. What sets the industry track apart is that it values impact and realism over novelty. How should this affect paper reviews? Industry track reviews should be the same in terms of quality, length, objectiveness, etc. *except* that they should emphasize industrial values instead of academic values. For instance, the ICSME Research track in 2015 (credit: Robillard & Krinke) used the following list of criteria to rate their papers (note: ordering is my interpretation of importance):
Potential impact refers to the potential impact of the findings, tool, or technique on industry. The potential for impact manifests itself in two ways. First, it can be measured by the magnitude of the result. An evaluation of different regression test selection techniques that shows that on one industrial system, one technique is 5% more effective than another would garner a low score in terms of potential impact. That same study showing that one technique is 50% more effective across several industrial systems would rate higher. Second, it can be measured by the applicability of the technique; an improved solution for the overly specific task of model-based, aspect-oriented, real-time fuzz testing is less widely applicable than an improved solution for the more general task of unit testing.
Real world focus differs depending on the type or section of the paper. For a tool-focused paper it refers to the maturity of the tool (an open source tool with 100s of users is better than a closed source tool with 100s of users is better than an open source tool with a few users..). For an evaluation section, it refers to the evaluation environment (an industrial environment with real developers working on their own code is better than industrial developers working on toy tasks is better than students..). For a case study it refers to the type of industrial environment (a large software company is better than a small startup is better than an undergraduate class) or the type of software systems studied (both proprietary and open source systems is better than just open source systems is better than student/research projects..).
Related work is extremely important in research papers because it is used to establish novelty. In industry track papers novelty is much less important, and thus related work serves a diminished role. In industry track papers related work should give context to the contribution, cite original sources of ideas, and cite original studies. Yet reviewers should be lenient with authors who miss related work, as practitioners may not always keep up with late-breaking results. Reviewers should provide feedback to the related work section in an industrial track with a collaborative mindset.
While the underlying values of a review are important, appropriate organization can dramatically affect a review's interpretation. To avoid unnecessary complexity, we recommend that you include the following in your reviews (from B. Adams and D. Poshyvanyk). Note that this advice is also applicable for research track reviews.
1. Paper summary
2. Points for the paper
3. Points against the paper
4. Supporting argumentation for your points
5. Suggested paper improvements
Although the Program Committee Co-Chairs determine the available ratings for each track, we recommend using the following ratings and guidelines whenever possible. This rating system is preferred over the five point rating system because it makes it easier to identify the champion (+3) and the opponent (-3) for each paper.
3: strong accept - I will object if rejected
2: accept - I support acceptance
1: weak accept - accept, but could reject
0: borderline paper - I have no opinion
-1: weak reject - reject, but could accept
-2: reject - I support rejection
-3: strong reject - I will object if accepted
While it's important to have clear values and formatting, I always find it easiest to learn through examples. Here's an anonymized example of a well done review:
Summary: The authors introduce an interesting problem, the performance of…, and provides a case study on their approach to dramatically reducing the runtime of... They also include some discussion on both the challenges and future opportunities.
Summary of Opinion: This paper introduces an interesting problem and offers a solution that improves performance by 50X (in some cases). While I wish the authors had included an example walkthrough of the porting process and that the takeaways were more interesting, I feel that the introduction of the problem and the improvement of performance is valuable enough to warrant publication.
In addition to the above (nearly) full review I'd like to include a few snippets from industry track reviews which you may not have ever seen in research track reviews:
It's also important to know what NOT to include in this type of review. Here are a few examples of phrases I hope to NOT see in an industry track review:
While it's great that the software engineering community is rallying around the importance of bridging the gap between academia and industry we face a major threat to this progress: our own biases. If we do not consciously shift our values when reviewing industry papers we will end up alienating practitioners further. Now that we have positive momentum let's put in the work to create professional, fair, and useful reviews.
David Shepherd leverages software engineering research to create useful additions to the IDE.