Performance measure of image and video quality assessment algorithms: subjective root-mean-square error

Mikko Nuutinen; Toni Virtanen; Jukka P. Häkkinen

doi:10.1117/1.JEI.25.2.023012

28 March 2016 Performance measure of image and video quality assessment algorithms: subjective root-mean-square error

Mikko Nuutinen, Toni Virtanen, Jukka P. Häkkinen

Author Affiliations +

Journal of Electronic Imaging, Vol. 25, Issue 2, 023012 (March 2016). https://doi.org/10.1117/1.JEI.25.2.023012

Abstract

Evaluating algorithms used to assess image and video quality requires performance measures. Traditional performance measures (e.g., Pearson’s linear correlation coefficient, Spearman’s rank-order correlation coefficient, and root mean square error) compare quality predictions of algorithms to subjective mean opinion scores (mean opinion score/differential mean opinion score). We propose a subjective root-mean-square error (SRMSE) performance measure for evaluating the accuracy of algorithms used to assess image and video quality. The SRMSE performance measure takes into account dispersion between observers. The other important property of the SRMSE performance measure is its measurement scale, which is calibrated to units of the number of average observers. The results of the SRMSE performance measure indicate the extent to which the algorithm can replace the subjective experiment (as the number of observers). Furthermore, we have presented the concept of target values, which define the performance level of the ideal algorithm. We have calculated the target values for all sample sets of the CID2013, CVD2014, and LIVE multiply distorted image quality databases.The target values and MATLAB implementation of the SRMSE performance measure are available on the project page of this study.

Citation Download Citation

Mikko Nuutinen, Toni Virtanen, and Jukka P. Häkkinen "Performance measure of image and video quality assessment algorithms: subjective root-mean-square error," Journal of Electronic Imaging 25(2), 023012 (28 March 2016). https://doi.org/10.1117/1.JEI.25.2.023012

Published: 28 March 2016

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available