- related work
- potential impact
- real world focus
- related work
Potential impact refers to the potential impact of the findings, tool, or technique on industry. The potential for impact manifests itself in two ways. First, it can be measured by the magnitude of the result. An evaluation of different regression test selection techniques that shows that on one industrial system, one technique is 5% more effective than another would garner a low score in terms of potential impact. That same study showing that one technique is 50% more effective across several industrial systems would rate higher. Second, it can be measured by the applicability of the technique; an improved solution for the overly specific task of model-based, aspect-oriented, real-time fuzz testing is less widely applicable than an improved solution for the more general task of unit testing.
Real world focus differs depending on the type or section of the paper. For a tool-focused paper it refers to the maturity of the tool (an open source tool with 100s of users is better than a closed source tool with 100s of users is better than an open source tool with a few users..). For an evaluation section, it refers to the evaluation environment (an industrial environment with real developers working on their own code is better than industrial developers working on toy tasks is better than students..). For a case study it refers to the type of industrial environment (a large software company is better than a small startup is better than an undergraduate class) or the type of software systems studied (both proprietary and open source systems is better than just open source systems is better than student/research projects..).
Related work is extremely important in research papers because it is used to establish novelty. In industry track papers novelty is much less important, and thus related work serves a diminished role. In industry track papers related work should give context to the contribution, cite original sources of ideas, and cite original studies. Yet reviewers should be lenient with authors who miss related work, as practitioners may not always keep up with late-breaking results. Reviewers should provide feedback to the related work section in an industrial track with a collaborative mindset.
1. Paper summary
2. Points for the paper
3. Points against the paper
4. Supporting argumentation for your points
5. Suggested paper improvements
Although the Program Committee Co-Chairs determine the available ratings for each track, we recommend using the following ratings and guidelines whenever possible. This rating system is preferred over the five point rating system because it makes it easier to identify the champion (+3) and the opponent (-3) for each paper.
3: strong accept - I will object if rejected
2: accept - I support acceptance
1: weak accept - accept, but could reject
0: borderline paper - I have no opinion
-1: weak reject - reject, but could accept
-2: reject - I support rejection
-3: strong reject - I will object if accepted
Summary of Opinion: This paper introduces an interesting problem and offers a solution that improves performance by 50X (in some cases). While I wish the authors had included an example walkthrough of the porting process and that the takeaways were more interesting, I feel that the introduction of the problem and the improvement of performance is valuable enough to warrant publication.
- Presents a real world problem and a real world solution while at the same time introducing a new problem for academics to focus on, which makes it ideally suited for an industrial track at a research conference.
- The approach shows that they are able to achieve dramatic speedup.
- The paper is well-written.
- The Case Study sections Design - Implementation are either too generic (e.g., we parallelized things) or too specific (e.g., we had to serialize and deserialize user-defined types). I found myself wanting you to walk me through a small example of how a small part of … had been parallelized, visiting all phases. Perhaps adding an "example" section before IV.A?
- The Opportunities section was of limited value. I would remove it. The Challenges section was not especially valuable either, I would consider reducing/removing it.
- I would have liked to hear more about the total cost, in terms of man-hours, of the porting process. This is an important part of determining whether the port was worth it. I would have also liked to hear more about the maintainability of the ported model. Are NN workers able to modify it directly? Or do they have to port it every time it is changed?
- There's a typo in..
- "I appreciated the paper in that this topic is underrepresented in the software engineering community. Although one might expect this type of topic to presented in the industrial systems communities, the paper presents a good argument that much of industrial control is in fact software, and that software engineers are the ones who need to be tackling these challenges."
- "Tackles a hairy problem, one that has a larger scope than many 'typical' research papers."
- "The biggest problem with this paper is that the motivation is extremely weak. Why is it problematic that… Why would anyone really care enough to run a tool of this kind?"
It's also important to know what NOT to include in this type of review. Here are a few examples of phrases I hope to NOT see in an industry track review:
- "My main concern with this paper is that the <findings> are not new as many of them are well discussed …"
- "The paper lacks concrete scientific contributions. As a reader, I did not learn anything new and interesting."
- This work is not novel.