Apply computer vision in GUI automation for industrial applications

研究成果: 雜誌貢獻期刊論文同行評審

10 引文 斯高帕斯(Scopus)


Technology has reshaped the workplace and the rapid improvements have transformed how we work nowadays. In the pursuit of industry 4.0, we build smart machines and robots to replace manual labor. While the manual labor is replaced by machines, in many cases, humans are transformed into desktop software users. Jobs such as testing, quality inspection, data monitoring, data entry, and routine editing remain to be done by humans in front of desktop computers. The operations to software applications in principle can be reduced to screen output understanding and mouse and keyboard operations. When the characteristics of these jobs are repetitive, tedious, and monotonous, they can be replaced by GUI automation techniques. GUI automation can be achieved by different underlying technologies, each has its pros and cons. In this paper, we describe a tool-Korat, which uses computer-vision to achieve maximum cross-platform capability for industrial applications, including test automation and robotic process automation. Although Korat has been successfully adopted by several industrial customers, difficult problems remain to be addressed. The problems and difficulties in applying computer vision for GUI automation are discussed and studied in this paper, particularly the experiences of applying open source OCR to GUI automation over color screenshots. By introducing critical pre-processing stages and algorithms, the recognition rate is significantly increased and becomes feasible for practical usage.

頁(從 - 到)7526-7545
期刊Mathematical Biosciences and Engineering
出版狀態已出版 - 2019


深入研究「Apply computer vision in GUI automation for industrial applications」主題。共同形成了獨特的指紋。