Lin Tan

Our research on comment analysis has made to Waterloo Engineering Annual Report.

/* Leveraging Code Comments to Improve Software Reliability */

Motivation: Software reliability are critically important. However, software bugs greatly hurt software reliability. A NIST study shows that software bugs cost the US economy $60 billion dollars annually.

A fundamental challenge in improving software reliability is to obtain important information about the software and the programmers. For example, what is the correct behavior of a program (e.g., where a lock is needed, where an interrupt should be disabled, etc.) so that a bug detection tool can check if the code matches with the correctness information? Additionally, what problems are important so that new language features or tools should be developed or improved with enhanced usability in order to address these problems?

Previous work uses a wide variety of information sources, including source code, execution traces, and human aids. While these are great sources of information, they are limited due to the amount of information that can be extracted, the large amount of manual effort required, or the poor scalability of these approaches. While we definitely should promote and encourage utilizing source code and human aids for improving software reliability, we need to seek alternatives.

Fortunately, a rich source of information --- code comments --- is widely available in most software. These comments can provide a great data source for obtaining programs' correctness information, discovering important problems, and understanding programmers' needs. As comments are more flexible and expressive than source code, developers use them to complement their source code for various purposes, such as explanations, assumptions, and specifications. However, these useful comments are ignored by almost all current compilers and bug detection tools.

What we have done in this direction: We proposed and conducted the first studies to leverage code comments to automatically detect software bugs and bad comments. We achieve these goals by combining techniques from different areas, including natural language processing (NLP), machine learning, information retrieval, program analysis and statistics.

(1) cComment: Understanding comments and the potential of utilizing comments. We conduct a comprehensive comment characteristics study on 6 pieces of large software, i.e., Linux, FreeBSD, OpenSolaris, MySQL, Firefox, and Eclipse, which are different types of software (OS, server, and desktop application) and are written in different programming languages (C, C++, and Java). By studying comments written by programmers, we have learned the real needs' of programmers, which can (1) motivate the design of new techniques or improving the usability of the existing tools for improving software reliability, and (2) help developers identify pervasive and important problems and adopt some existing tools or languages for help. We learned many findings including that at least 52.6 ± 2.9% of the comments could be leveraged by existing or to-be-proposed tools for improving reliability.

(2) iComment: Using comments to detect software bugs and bad comments. When comments and code mismatch, it indicates either (1) bugs -- source code does not follow the correct comment, or (2) bad comments -- the comment is wrong or outdated, which can later lead to bugs. iComment takes the first step to detect such comment-code inconsistencies by automatically extracting specifications from comments, and then using flow-sensitive and context-sensitive static program analysis tools to check these specifications against source code. iComment has found 60 previously unknown bugs and bad comments in large software, i.e., Linux, Mozilla, Apache and Wine, and many of them have already been confirmed and fixed by the corresponding developers.

(3) aComment: Mining Annotations from Comments and Code to Detect Interrupt-Related Concurrency Bugs. To detect OS concurrency bugs, we proposed a new type of annotations interrupt related annotations and automatically generated 96,821 such annotations for the Linux kernel with little manual effort. These annotations have been used to automatically detect 9 real OS concurrency bugs (7 are previously unknown). A key technique is using a hybrid approach to extract annotations from both code and comments written in natural language to achieve better coverage and accuracy in annotation extraction and bug detection.

(4) @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies

Related Publications:

@tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. Shin Hwei Tan, Darko Marinov, Lin Tan and Gary T. Leavens. In the proceedings of the 5th International Conference on Software Testing, Verification and Validation. April, 2012. Montreal, Quebec. (10 pages) Acceptance Rate: 26.9% (39/145). [Slides in PDF] [BIBTEX]


aComment: Mining Annotations from Comments and Code to Detect Interrupt-Related Concurrency Bugs. Lin Tan, Yuanyuan Zhou and Yoann Padioleau. In the proceedings of the International Conference on Software Engineering. May, 2011. Waikiki, Honolulu, Hawaii. (10 pages) Acceptance Rate: 14.1% (62/441). [Slides (no animation)] [Slides (with animation)] [BIBTEX] [Cited 20+ Times] (Press Coverage)


Listening to Programmers - Taxonomies and Characteristics of Comments in Operating System Code. (Alphabetic order) Yoann Padioleau, Lin Tan and Yuanyuan Zhou. In the proceedings of the International Conference on Software Engineering. May, 2009. Vancouver, BC. (11 pages) Acceptance Rate: 12.3% (50/405). [PS] [Slides in PDF] [BIBTEX] [Data & Software] [Cited 20+ Times] (Press Coverage)


/* iComment: Bugs or Bad Comments? */ Lin Tan, Ding Yuan, Gopal Krishna and Yuanyuan Zhou. In the Proceedings of the 21st ACM Symposium on Operating Systems Principles, October 2007. Stevenson, Washington. (14 pages) Acceptance Rate: 19.1% (25/131). [PS] [Slides in PDF] [Slides in PDF with NO animation] [BIBTEX] [Cited 50+ Times] [In other people's words]. (Press Coverage)


HotComments: How to Make Program Comments More Useful? Lin Tan, Ding Yuan and Yuanyuan Zhou. In the Proceedings of the 11th Workshop on Hot Topics in Operating Systems, May 2007. San Diego, California. (6 pages) Acceptance Rate: 20.0% (21/105). [BIBTEX]

NLP Tools We Used: Semantic Role Labeler, Weka