Qualitative evaluation of automatic assignment of keywords to images

Chih Fong Tsai, Ken McGarry, John Tait

研究成果: 雜誌貢獻期刊論文同行評審

19 引文 斯高帕斯(Scopus)


In image retrieval, most systems lack user-centred evaluation since they are assessed by some chosen ground truth dataset. The results reported through precision and recall assessed against the ground truth are thought of as being an acceptable surrogate for the judgment of real users. Much current research focuses on automatically assigning keywords to images for enhancing retrieval effectiveness. However, evaluation methods are usually based on system-level assessment, e.g. classification accuracy based on some chosen ground truth dataset. In this paper, we present a qualitative evaluation methodology for automatic image indexing systems. The automatic indexing task is formulated as one of image annotation, or automatic metadata generation for images. The evaluation is composed of two individual methods. First, the automatic indexing annotation results are assessed by human subjects. Second, the subjects are asked to annotate some chosen images as the test set whose annotations are used as ground truth. Then, the system is tested by the test set whose annotation results are judged against the ground truth. Only one of these methods is reported for most systems on which user-centred evaluation are conducted. We believe that both methods need to be considered for full evaluation. We also provide an example evaluation of our system based on this methodology. According to this study, our proposed evaluation methodology is able to provide deeper understanding of the system's performance.

頁(從 - 到)136-154
期刊Information Processing and Management
發行號1 SPEC. ISS
出版狀態已出版 - 1月 2006


深入研究「Qualitative evaluation of automatic assignment of keywords to images」主題。共同形成了獨特的指紋。