ActionNet: Vision-based Workflow Action Recognition From Programming ScreencastsTechnical Track
Programming screencasts have two important applications in software engineering context: study developer behaviors and information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we present the first technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers’ real work, we demonstrate the usefulness of our technique in a practical application for action-aware extraction of key-code frames in developers’ work.
Wed 29 MayDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 18:00 | SE Datasets, Research Infrastructure, and MethodologyJournal-First Papers / New Ideas and Emerging Results / Demonstrations / Papers / Technical Track at Viger Chair(s): Rashina Hoda The University of Auckland | ||
16:00 20mTalk | BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and FixesTechnical Track Technical Track Naji Dmeiri University of California, Davis, David A Tomassi University of California, Davis, Yichen Wang University of California, Davis, Antara Bhowmick University of California, Davis, Yen-Chuan Liu University of California, Davis, Prem Devanbu University of California, Bogdan Vasilescu Carnegie Mellon University, Cindy Rubio-González University of California, Davis Pre-print | ||
16:20 20mTalk | DefeXts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM LanguagesDemos Demonstrations Samuel Benton The University of Texas at Dallas, Ali Ghanbari The University of Texas at Dallas, Lingming Zhang | ||
16:40 10mTalk | Open Collaborative Data – using OSS principles to share data in SW engineeringNIER New Ideas and Emerging Results Per Runeson Lund University | ||
16:50 10mTalk | Leveraging Small Software Engineering Data Sets with Pre-trained Neural NetworksNIER New Ideas and Emerging Results | ||
17:00 20mTalk | ActionNet: Vision-based Workflow Action Recognition From Programming ScreencastsTechnical Track Technical Track Dehai Zhao , Zhenchang Xing Australia National University, Chunyang Chen Monash University, Xin Xia Monash University, Guoqiang Li Shanghai Jiao Tong University | ||
17:20 10mTalk | The ABC of Software Engineering ResearchJournal-First Journal-First Papers Klaas-Jan Stol University College Cork and Lero, Ireland, Brian Fitzgerald Lero - The Irish Software Research Centre and University of Limerick Link to publication DOI | ||
17:30 10mTalk | Mining Plausible Hypotheses from the Literature via Meta-AnalysisNIER New Ideas and Emerging Results Vladimir Ivanov , Giancarlo Succi Innopolis University, Jooyong Yi UNIST (Ulsan National Institute of Science and Technology) | ||
17:40 10mTalk | Analyzing Families of Experiments in SE: a Systematic Mapping StudyJournal-First Journal-First Papers Adrian Santos Parrilla , Omar Gomez Escuela Superior Politecnica de Chimborazo Riobamba, Natalia Juristo Universidad Politecnica de Madrid | ||
17:50 10mTalk | Discussion Period Papers | ||