How to Review an Industry Track Paper

Recently there's been an increasing amount of excitement around industrial tracks. They are becoming well attended, competitive, and discussion-worthy. Alas, this recent enthusiasm could be quickly extinguished if we, the program committees of these industrial tracks, provide inappropriate or low quality reviews. Imagine the frustration of a practitioner if their real world case study gets rejected while an experiment with ten freshman undergraduate students is accepted. To avoid this, we need to provide clear, distinct guidelines for evaluating industry track papers.

---

Publish at an industry track: SANER Industrial Track, ICPC Industry Track, FSE Industry Track

---

Values

An industry track focuses on the same topics and expects the same rigor as the corresponding research track, yet papers featured in the industry track are distinct. What sets the industry track apart is that it values impact and realism over novelty. How should this affect paper reviews? Industry track reviews should be the same in terms of quality, length, objectiveness, etc. *except* that they should emphasize industrial values instead of academic values. For instance, the ICSME Research track in 2015 (credit: Robillard & Krinke) used the following list of criteria to rate their papers (note: ordering is my interpretation of importance):

1. soundness 
2. originality 
3. importance
4. presentation
5. related work

In contrast, an industrial track should use the following list of criteria:

1. soundness
2. potential impact
3. real world focus
4. presentation
5. related work

Of this list, the potential impact and real world focus differ from items on the research track list, and a few words about related work must be said.

Potential impact refers to the potential impact of the findings, tool, or technique on industry. The potential for impact manifests itself in two ways.  First, it can be measured by the magnitude of the result. An evaluation of different regression test selection techniques that shows that on one industrial system, one technique is 5% more effective than another would garner a low score in terms of potential impact. That same study showing that one technique is 50% more effective across several industrial systems would rate higher. Second, it can be measured by the applicability of the technique; an improved solution for the overly specific task of model-based, aspect-oriented, real-time fuzz testing is less widely applicable than an improved solution for the more general task of unit testing.

Real world focus differs depending on the type or section of the paper. For a tool-focused paper it refers to the maturity of the tool (an open source tool with 100s of users is better than a closed source tool with 100s of users is better than an open source tool with a few users..). For an evaluation section, it refers to the evaluation environment (an industrial environment with real developers working on their own code is better than industrial developers working on toy tasks is better than students..). For a case study it refers to the type of industrial environment (a large software company is better than a small startup is better than an undergraduate class) or the type of software systems studied (both proprietary and open source systems is better than just open source systems is better than student/research projects..).
​
Related work is extremely important in research papers because it is used to establish novelty. In industry track papers novelty is much less important, and thus related work serves a diminished role. In industry track papers related work should give context to the contribution, cite original sources of ideas, and cite original studies. Yet reviewers should be lenient with authors who miss related work, as practitioners may not always keep up with late-breaking results. Reviewers should provide feedback to the related work section in an industrial track with a collaborative mindset.

Format

While the underlying values of a review are important, appropriate organization can dramatically affect a review's interpretation. To avoid unnecessary complexity, we recommend that you include the following in your reviews (from B. Adams and D. Poshyvanyk). Note that this advice is also applicable for research track reviews.

1. Paper summary
2. Points for the paper
3. Points against the paper
4. Supporting argumentation for your points
5. Suggested paper improvements

Although the Program Committee Co-Chairs determine the available ratings for each track, we recommend using the following ratings and guidelines whenever possible. This rating system is preferred over the five point rating system because it makes it easier to identify the champion (+3) and the opponent (-3) for each paper.

 3: strong accept - I will object if rejected
 2: accept - I support acceptance
 1: weak accept - accept, but could reject
 0: borderline paper - I have no opinion
-1: weak reject - reject, but could accept
-2: reject - I support rejection
-3: strong reject - I will object if accepted

Examples

While it's important to have clear values and formatting, I always find it easiest to learn through examples. Here's an anonymized example of a well done review:

---

Summary: The authors introduce an interesting problem, the performance of…, and provides a case study on their approach to dramatically reducing the runtime of... They also include some discussion on both the challenges and future opportunities.

Summary of Opinion: This paper introduces an interesting problem and offers a solution that improves performance by 50X (in some cases). While I wish the authors had included an example walkthrough of the porting process and that the takeaways were more interesting, I feel that the introduction of the problem and the improvement of performance is valuable enough to warrant publication.

Positives:

• Presents a real world problem and a real world solution while at the same time introducing a new problem for academics to focus on, which makes it ideally suited for an industrial track at a research conference.
• The approach shows that they are able to achieve dramatic speedup.
• The paper is well-written.

Issues:

• The Case Study sections Design - Implementation are either too generic (e.g., we parallelized things) or too specific (e.g., we had to serialize and deserialize user-defined types). I found myself wanting you to walk me through a small example of how a small part of … had been parallelized, visiting all phases. Perhaps adding an "example" section before IV.A?
• The Opportunities section was of limited value. I would remove it. The Challenges section was not especially valuable either, I would consider reducing/removing it.
• I would have liked to hear more about the total cost, in terms of man-hours, of the porting process. This is an important part of determining whether the port was worth it. I would have also liked to hear more about the maintainability of the ported model. Are NN workers able to modify it directly? Or do they have to port it every time it is changed?

Supporting Arguments/Details:

• There's a typo in..
• …

---

In addition to the above (nearly) full review I'd like to include a few snippets from industry track reviews which you may not have ever seen in research track reviews:

• "I appreciated the paper in that this topic is underrepresented in the software engineering community. Although one might expect this type of topic to presented in the industrial systems communities, the paper presents a good argument that much of industrial control is in fact software, and that software engineers are the ones who need to be tackling these challenges."
• "Tackles a hairy problem, one that has a larger scope than many 'typical' research papers."
• "The biggest problem with this paper is that the motivation is extremely weak. Why is it problematic that… Why would anyone really care enough to run a tool of this kind?"

It's also important to know what NOT to include in this type of review. Here are a few examples of phrases I hope to NOT see in an industry track review:

• "My main concern with this paper is that the <findings> are not new as many of them are well discussed …"
• "The paper lacks concrete scientific contributions. As a reader, I did not learn anything new and interesting."
• This work is not novel.

While novelty is a major concern for research track papers it is NOT a large factor for the industry track. The bar that academia has for findings being "well discussed" or "widely accepted" is dramatically different than in industry, and so study replications or real world trials of proposed approaches are welcomed.

Conclusion

While it's great that the software engineering community is rallying around the importance of bridging the gap between academia and industry we face a major threat to this progress: our own biases. If we do not consciously shift our values when reviewing industry papers we will end up alienating practitioners further.  Now that we have positive momentum let's put in the work to create professional, fair, and useful reviews.