An established way of validating and testing new image quality assessment (IQA) algorithms have been to compare how
well they correlate with subjective data on various image databases. One of the most common measures is to calculate
linear correlation coefficient (LCC) and Spearman’s rank order correlation coefficient (SROCC) against the subjective
mean opinion score (MOS). Recently, databases with multiply distorted images have emerged 1,2. However with
multidimensional stimuli, there is more disagreement between observers as the task is more preferential than that of
distortion detection. This reduces the statistical differences between image pairs. If the subjects cannot distinguish a
difference between some of the image pairs, should we demand any better performance with IQA algorithms? This paper
proposes alternative performance measures for the evaluation of IQA’s for the CID2013 database. One proposed
alternative performance measure is root-mean-square-error (RMSE) value for the subjective data as a function of the
number of observers. The other alternative performance measure is the number of statistical differences between image
pairs. This study shows that after 12 subjects the RMSE value saturates around the level of three, meaning that a target
RMSE value for an IQA algorithm for CID2013 database should be three. In addition, this study shows that the state-of-the-art IQA algorithms found the better image from the image pairs with a probability of 0.85 when the image pairs with
statistically significant differences were taken into account.
To understand the viewing strategies employed in a quality estimation task, we compared two visual tasks—quality estimation and difference estimation. The estimation was done for a pair of natural images having small global changes in quality. Two groups of observers estimated the same set of images, but with different instructions. One group estimated the difference in quality and the other the difference between image pairs. The results demonstrated the use of different visual strategies in the tasks. The quality estimation was found to include more visual planning during the first fixation than the difference estimation, but afterward needed only a few long fixations on the semantically important areas of the image. The difference estimation used many short fixations. Salient image areas were mainly attended to when these areas were also semantically important. The results support the hypothesis that these tasks’ general characteristics (evaluation time, number of fixations, area fixated on) show differences in processing, but also suggest that examining only single fixations when comparing tasks is too narrow a view. When planning a subjective experiment, one must remember that a small change in the instructions might lead to a noticeable change in viewing strategy.
The most common tasks in subjective image estimation are change detection (a detection task) and image quality
estimation (a preference task). We examined how the task influences the gaze behavior when comparing detection and
preference tasks. The eye movements of 16 naïve observers were recorded with 8 observers in both tasks. The setting
was a flicker paradigm, where the observers see a non-manipulated image, a manipulated version of the image and again
the non-manipulated image and estimate the difference they perceived in them. The material was photographic material
with different image distortions and contents. To examine the spatial distribution of fixations, we defined the regions of
interest using a memory task and calculated information entropy to estimate how concentrated the fixations were on the
image plane. The quality task was faster and needed fewer fixations and the first eight fixations were more concentrated
on certain image areas than the change detection task. The bottom-up influences of the image also caused more variation
to the gaze behavior in the quality estimation task than in the change detection task The results show that the quality
estimation is faster and the regions of interest are emphasized more on certain images compared with the change
detection task that is a scan task where the whole image is always thoroughly examined. In conclusion, in subjective
image estimation studies it is important to think about the task.
The goal of the study was to develop a method for quality computation of digitally printed images. We wanted to use
only the attributes which have a meaning for subjective visual quality experience of digitally printed images. Based on
the subjective data and our assessments the attributes for quality calculation were sharpness, graininess and color
contrast. The proposed graininess metric divides the fine detail image into blocks and used the low energy blocks for
graininess calculation. The proposed color contrast metric computes the contrast of dominant colors using the coarse
scale image. The proposed sharpness metric divides the coarse scale image into blocks and uses the high energy blocks
for sharpness calculation. The reduced reference features of sharpness and graininess metrics are the numbers of high or
low energy blocks. The reduced reference features of the color contrast metric are the directions of the dominant colors
in reference image. The overall image quality was calculated by combining the values. The performance of proposed
application specific image quality metric was high compared to the state of the art reduced reference applicationindependent
image quality metric. Linear correlation coefficients between subjective and predicted MOS were 0.88 for
electrophotography and 0.98 for ink-jet printed samples, for a sample set of 21 prints for electrophotography and for inkjet,
subjectively evaluated by 28 observers.
Subjective image quality data for 9 image processing pipes and 8 image contents (taken with mobile phone
camera, 72 natural scene test images altogether) from 14 test subjects were collected. A triplet comparison setup
and a hybrid qualitative/quantitative methodology were applied. MOS data and spontaneous, subjective
image quality attributes to each test image were recorded. The use of positive and negative image quality
attributes by the experimental subjects suggested a significant difference between the subjective spaces of low
and high image quality. The robustness of the attribute data was shown by correlating DMOS data of the test
images against their corresponding, average subjective attribute vector length data. The findings demonstrate
the information value of spontaneous, subjective image quality attributes in evaluating image quality at variable
quality levels. We discuss the implications of these findings for the development of sensitive performance
measures and methods in profiling image processing systems and their components, especially at high image
quality levels.
A test image for color still image processes was developed. The image is based on general requirements on the content
and specific requirements arising from the quality attributes of interest. The quality attributes addressed in the study
include sharpness, noise, contrast, colorfulness and gloss. These were chosen based on visual relevance in studies of the
influence of paper in digital printing. Further requirements such as arising from the use cases of the image are discussed
based on eye tracking data and self-report of the usefulness of different objects for quality evaluation. From the
standpoint of being sufficiently sensitive to quality variations of the imaging systems to be measured the reference test
image needs to represent quality maxima in terms of the relevant quality parameters. As for different viewing times, no
object should be exceedingly salient. The paper presents the procedure of developing the test image and discusses its
merits and shortcomings from the standpoint of future development.
Subjective quality rating does not reflect the properties of the image directly, but it is the outcome of a quality decision
making process, which includes quantification of subjective quality experience. Such a rich subjective content is often
ignored. We conducted two experiments (with 28 and 20 observers), in order to study the effect of paper grade on image
quality experience of the ink-jet prints. Image quality experience was studied using a grouping task and a quality rating
task. Both tasks included an interview, but in the latter task we examined the relations of different subjective attributes in
this experience. We found out that the observers use an attribute hierarchy, where the high-level attributes are more
experiential, general and abstract, while low-level attributes are more detailed and concrete. This may reflect the
hierarchy of the human visual system. We also noticed that while the observers show variable subjective criteria for IQ,
the reliability of average subjective estimates is high: when two different observer groups estimated the same images in
the two experiments, correlations between the mean ratings were between .986 and .994, depending on the image
content.
The subjective quality of an image is a non-linear product of several, simultaneously contributing subjective factors such
as the experienced naturalness, colorfulness, lightness, and clarity. We have studied subjective image quality by using a
hybrid qualitative/quantitative method in order to disclose relevant attributes to experienced image quality. We describe
our approach in mapping the image quality attribute space in three cases: still studio image, video clips of a talking head
and moving objects, and in the use of image processing pipes for 15 still image contents. Naive observers participated in
three image quality research contexts in which they were asked to freely and spontaneously describe the quality of the
presented test images. Standard viewing conditions were used. The data shows which attributes are most relevant for
each test context, and how they differentiate between the selected image contents and processing systems. The role of
non-HVS based image quality analysis is discussed.
Stereoscopic technologies have developed significantly in recent years. These advances require also more understanding
of the experiental dimensions of stereoscopic contents. In this article we describe experiments in which we explore the
experiences that viewers have when they view stereoscopic contents. We used eight different contents that were shown
to the participants in a paired comparison experiment where the task of the participants was to compare the same content
in stereoscopic and non-stereoscopic form. The participants indicated their preference but were also interviewed about
the arguments they used when making the decision. By conducting a qualitative analysis of the interview texts we
categorized the significant experiental factors related to viewing stereoscopic material. Our results indicate that reality-likeness
as well as artificiality were often used as arguments in comparing the stereoscopic materials. Also, there were
more emotional terms in the descriptions of the stereoscopic films, which might indicate that the stereoscopic projection
technique enhances the emotions conveyed by the film material. Finally, the participants indicated that the three-dimensional
material required longer presentation time, as there were more interesting details to see.
Due to the rise in performance of digital printing, image-based applications are gaining popularity. This creates needs for
specifying the quality potential of printers and materials in more detail than before. Both production and end-use
standpoints are relevant. This paper gives an overview of an
on-going study which has the goal of determining a
framework model for the visual quality potential of paper in color image printing. The approach is top-down and it is
founded on the concept of a layered network model. The model and its subjective, objective and instrumental
measurement layers are discussed. Some preliminary findings are presented. These are based on data from samples
obtained by printing natural image contents and simple test fields on a wide range of paper grades by ink-jet in a color
managed process. Color profiles were paper specific. Visual mean opinion score data by human observers could be
accounted for by two or three dimensions. In the first place these are related to brightness and color brightness. Image
content has a marked effect on the dimensions. This underlines the challenges in designing the test images.
The psychological complexity of multivariate image quality evaluation makes it difficult to develop general image quality metrics. Quality evaluation includes several mental processes and ignoring these processes and the use of a few test images can lead to biased results. By using a qualitative/quantitative (Interpretation Based Quality, IBQ) methodology, we examined the process of pair-wise comparison in a setting, where the quality of the images printed by laser printer on different paper grades was evaluated. Test image consisted of a picture of a table covered with several objects. Three other images were also used, photographs of a woman, cityscape and countryside. In addition to the pair-wise comparisons, observers (N=10) were interviewed about the subjective quality attributes they used in making their quality decisions. An examination of the individual pair-wise comparisons revealed serious inconsistencies in observers' evaluations on the test image content, but not on other contexts. The qualitative analysis showed that this inconsistency was due to the observers' focus of attention. The lack of easily recognizable context in the test image may have contributed to this inconsistency. To obtain reliable knowledge of the effect of image context or attention on subjective image quality, a qualitative methodology is needed.
The visual quality of images is outward in image presentation, compression and analysis. Depending on the use, the quality of images may give more information or more experiences to the viewer. However, the relations between mathematical and human methods for grouping the images are not obvious. For example, different humans think differently and so, they make the grouping differently. However, there may be some connections between image mathematical features and human selections. Here we try to find such relations that could give more possibilities for developing the actual quality of images for different purposes. In this study, we present some methods and preliminary results that are based on psychological tests to humans, MPEG-7 based features of the images and face detection methods. We also show some notes and questions belonging to this problem and plans for the future research.
Image evaluation schemes must fulfill both objective and subjective requirements. Objective image quality evaluation models are often preferred over subjective quality evaluation, because of their fastness and cost-effectiveness. However, the correlation between subjective and objective estimations is often poor. One of the key reasons for this is that it is not known what image features subjects use when they evaluate image quality. We have studied subjective image quality evaluation in the case of image sharpness. We used an Interpretation-based Quality (IBQ) approach, which combines both qualitative and quantitative approaches to probe the observer's quality experience. Here we examine how naive subjects experienced and classified natural images, whose sharpness was changing. Together the psychometric and qualitative information obtained allows the correlation of quantitative evaluation data with its underlying subjective attribute sets. This offers guidelines to product designers and developers who are responsible for image quality. Combining these methods makes the end-user experience approachable and offers new ways to improve objective image quality evaluation schemes.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.