1 Reviewer 1
   2 ----------
   3 
   4 I like this work, because it addresses an important concern arising from any research on FPGA tools and architectures: how well can academic accomplishments be transferred to commercial/ industrial products? It is not necessary, in my opinion, that every researcher has impact on commercial products in mind when starting his/ her research. But for those who do, this paper is worth reading.
   5 
   6 Before this paper, not much research is there to analyze and explore this concern. So I consider this paper novel, although it doesn’t propose new FPGA tool or architecture ideas.
   7 
   8 Evaluation framework used in this paper is reasonable and sound, and I believe the results are credible. Commonly used metrics like speed-performance (ie, critical path delay), area, and runtime are reported. An interesting pointer is that an analytical placer is badly needed, to replace the current simulated annealing-based VPR placer. I consider this an important take-away message for FPGA tool researchers. 
   9 
  10 Another interesting finding is reported in Section V (and illustrated in Figure 7). That is, divide between academic and commercial benchmarks. Commercial benchmarks not only contains more diversified modules (memories and DSPs), but also smaller logic depths. I think academic benchmarks should converge towards commercial benchmarks in this regard. Smaller logic depth should be the trend, given that designs are heavily pipelined to achieve high performance and FPGAs are equipped with more registers to aid pipelining.
  11 
  12 This paper is well-written, except for a few minor issues: (1) Figure 1 (and left half of Figure 7) is not well readable when printed on a monochrome printer; (2) A chunk of space is left blank at end of page 2, which looks a bit strange.
  13 
  14 Reviewer 2
  15 ----------
  16 The authors compare the CAD tools with the latest Xilinx commercial tools in the major figures of merit, propose a hybrid design flow using Vivado and ABC, and discuss the difference in terms of benchmark design. Some issues related to this paper:
  17 
  18 1. Although showing the gap with groups of results, the authors can try to explain the reason that the gap exists.
  19 
  20 2. The hybrid evaluation flow combining academic and commercial tools is only a compromising solution for academia.
  21 
  22 3. The paper focuses on a hybrid design flow to improve the synthesis process, which takes only 4% of the total runtime. More attention should be addressed to place and route.
  23 
  24 4. It is necessary to compare the benchmark suite between academia and industry, but it is more important to address the criteria of design benchmark. Academia can possibly lead industry only by proposing a more reasonable criteria and matching it better.
  25 
  26 5. Missing reference (not limited to): "Mind The (Synthesis) Gap: Examining Where Academic FPGA Tools Lag Behind Industry"
  27 
  28 Reviewer 3
  29 ----------
  30 The paper provides an interesting assessment of academic and commercial CAD tools. It is worthwhile to try to understand the magnitude of this gap so that academic work focuses on areas where they can have most impact. 
  31 
  32 There are a few things that could strengthen the paper:
  33 - One of the key comparisons is Table 1 where the Fmax gap is the key result. However, this result is less meaningful without a direct comparison of the underlying architecture. It would be prudent to compare the delays of the basic elements to understand the performance differences of the academic and commercial architectures. With the current data, it is unknown whether the routing or LAB delays might be a major or minor reason behind the Fmax gap.
  34 - For the benchmark suite comparison, it seems odd to compare to a proprietary benchmark set and not offer access to it. A paper on a CAD improvement that focused on gains exclusively for a proprietary would be strongly questioned. This seems similar in just saying one set is better than another without details. It will be better if [16] can include links to the new set or at a minimum there should be more details on the characteristics of this new benchmark set.
  35 
  36 Reviewer 4
  37 ----------
  38 This paper engages in a very pressing endeavor: analyzing the gap between the results that can be currently attained by commercial FPGA design tools on the one hand, and those published by the scientific community on the other, which according to the authors, lag in terms of speed/performance by a factor of 2.2x when compared with the former. The kind of design tools and methods tackled by the authors concern more the logic synthesis and implementation phases of the FPGA design flow. According to the authors, the results obtained by academic tools can be improved at least partially by making use of various logical optimizations, which are presented here as an extended flow integrating ABC and the VTR framework.
  39 
  40 The introduction and second section of the paper (Background and Related Work) effectively present the motivation for the studies and analyses presented by the authors in the article. The authors mainly focus on VPR, ABC Logic and ODIN II, which have been combined in the VTR framework to enable a unified flow for FPGA compilation and implementation. The authors build upon VTR, since they argue that current tools tackle decade old devices and that they are not comparable with recent developments by the industry. Moreover, the majority of academic tools make use of intermediate format no longer supported by FPGA vendors, which only augments the divide between the two worlds. 
  41 
  42 From section III onwards, the authors embark in a thorough analysis of the most recent developments in design tools, addressing gaps in area-efficiency and runtime scaling. They first compare commercial versus an academic implementations of various algorithmic benchmarks; through this analysis, they conclude that a speed-performance of about 2.2x is obtained by commercial tools, compared to the solution based on VTR and furthermore, that this divide is largely due to synthesis phase of the design flow. The area-efficiency obtained by academic tools, although significant (9%) is considered acceptable by the authors when compared to the other figures of merit.
  43 
  44 Regarding the runtime for compilation, the authors identify opportunities for improvement, especially with regards to the placement, which could impact the performance of academic tools if analytical placement techniques are utilized. Furthermore the authors go a step forward by proposing a modified VTR-to-Bitstream chain to demonstrate that Vivado outperforms significantly VTB with regard to the actual area utilization and maximum operating frequency. More experiments with an hybrid flow in mind further demonstrate the increased gap between academic and commercial design tools. Thus, the main difference comes from the backend tools, which is not a surprised since this is where commercial tools have a great advantage.
  45 
  46 I believe that this analysis and discussion alone could prove of interest for the FPT community and will without any doubt elicit a vivid (and necessary) debate, which could be beneficial for all the interested parties. My only complaint is with the introduction of the Benchmark Design Suite section at the end of the paper, which seems a bit disjointed from the flow of the discourse. However, this is a minor quaint and given the soundness of the experimental results and the discussion, I recommend this paper for presentation at the FPT conference.
  47 
  48 Moreover, the presented results and comparison might only be valid only for VPR, and thus the conclusion is for one RELEVANT academic tool only, and not generalizable for all academic possibilities. Nonetheless, it would be impossible to make a comprenhensive comparison, that being of the main obstacle to academic/commercial cooperations. Another issue is the choice of synthesis options, which neeeds to be better stated to make their point across. To conclude, the authors might need to maker their conclusions less general, since the obtained results might not be applicable to other scenarios.