Breast cancer risk prediction algorithms are used to identify subpopulations that are at increased risk for developing breast cancer. They can be based on many different sources of data such as demographics, relatives with cancer, gene expression, and various phenotypic features such as breast density. Women who are identified as high risk may undergo a more extensive (and expensive) screening process that includes MRI or ultrasound imaging in addition to the standard full-field digital mammography (FFDM) exam. Given that there are many ways that risk prediction may be accomplished, it is of interest to evaluate them in terms of expected cost, which includes the costs of diagnostic outcomes. In this work we perform an expected-cost analysis of risk prediction algorithms that is based on a published model that includes the costs associated with diagnostic outcomes (true-positive, false-positive, etc.). We assume the existence of a standard screening method and an enhanced screening method with higher scan cost, higher sensitivity, and lower specificity. We then assess expected cost of using a risk prediction algorithm to determine who gets the enhanced screening method under the strong assumption that risk and diagnostic performance are independent. We find that if risk prediction leads to a high enough positive predictive value, it will be cost-effective regardless of the size of the subpopulation. Furthermore, in terms of the hit-rate and false-alarm rate of the of the risk prediction algorithm, iso-cost contours are lines with slope determined by properties of the available diagnostic systems for screening.
KEYWORDS: Medical imaging, Medical statistics, Clinical trials, Image quality, Imaging devices, Monte Carlo methods, Statistical analysis, Error analysis, Data modeling, Imaging systems, Silver
The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesn't matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.
KEYWORDS: Lab on a chip, Mathematical modeling, Convolution, Performance modeling, Data modeling, Error analysis, Medical imaging, Monte Carlo methods, Image analysis, Systems modeling
In localization tasks, an observer is asked to give the location of some target or feature of interest in an image. Scanning linear observer models incorporate the search implicit in this task through convolution of an observer template with the image being evaluated. Such models are becoming increasingly popular as predictors of human performance for validating medical imaging methodology. In addition to convolution, scanning models may utilize internal noise components to model inconsistencies in human observer responses. In this work, we build a probabilistic mathematical model of this process and show how it can, in principle, be used to obtain estimates of the observer template using maximum likelihood methods. The main difficulty of this approach is that a closed form probability distribution for a maximal location response is not generally available in the presence of internal noise. However, for a given image we can generate an empirical distribution of maximal locations using Monte-Carlo sampling. We show that this probability is well approximated by applying an exponential function to the scanning template output. We also evaluate log-likelihood functions on the basis of this approximate distribution. Using 1,000 trials of simulated data as a validation test set, we find that a plot of the approximate log-likelihood function along a single parameter related to the template profile achieves its maximum value near the true value used in the simulation. This finding holds regardless of whether the trials are correctly localized or not. In a second validation study evaluating a parameter related to the relative magnitude of internal noise, only the incorrect localization images produces a maximum in the approximate log-likelihood function that is near the true value of the parameter.
KEYWORDS: Binary data, Data modeling, Statistical analysis, Monte Carlo methods, Error analysis, Diagnostics, Mathematical modeling, Performance modeling, Medical imaging, Data analysis
We treat multireader multicase (MRMC) reader studies for which a reader’s diagnostic assessment is converted to binary agreement (1: agree with the truth state, 0: disagree with the truth state). We present a mathematical model for simulating binary MRMC data with a desired correlation structure across readers, cases, and two modalities, assuming the expected probability of agreement is equal for the two modalities (P1=P2). This model can be used to validate the coverage probabilities of 95% confidence intervals (of P1, P2, or P1−P2 when P1−P2=0), validate the type I error of a superiority hypothesis test, and size a noninferiority hypothesis test (which assumes P1=P2). To illustrate the utility of our simulation model, we adapt the Obuchowski–Rockette–Hillis (ORH) method for the analysis of MRMC binary agreement data. Moreover, we use our simulation model to validate the ORH method for binary data and to illustrate sizing in a noninferiority setting. Our software package is publicly available on the Google code project hosting site for use in simulation, analysis, validation, and sizing of MRMC reader studies with binary agreement data.
KEYWORDS: Statistical analysis, Monte Carlo methods, Receivers, Magnetic resonance imaging, Signal detection, Statistical modeling, Electronic filtering, Data modeling, Image quality, Image processing
In an effort to generalize task-based assessment beyond traditional signal detection, there is a growing interest in performance evaluation for combined detection and estimation tasks, in which signal parameters, such as size, orientation, and contrast are unknown and must be estimated. One motivation for studying such tasks is their rich complexity, which offers potential advantages for imaging system optimization. To evaluate observer performance on combined detection and estimation tasks, Clarkson introduced the estimation receiver operating characteristic (EROC) curve and the area under the EROC curve as a summary figure of merit. This work provides practical tools for EROC analysis of experimental data. In particular, we propose nonparametric estimators for the EROC curve, the area under the EROC curve, and for the variance/covariance matrix of a vector of correlated EROC area estimates. In addition, we show that reliable confidence intervals can be obtained for EROC area, and we validate these intervals with Monte Carlo simulation. Application of our methodology is illustrated with an example comparing magnetic resonance imaging k-space sampling trajectories. MATLAB® software implementing the EROC analysis estimators described in this work is publicly available at http://code.google.com/p/iqmodelo/.
The availability of large medical image datasets is critical in many applications such as training and testing of computer aided diagnosis (CAD) systems, evaluation of segmentation algorithms, and conducting perceptual studies. However, collection of large repositories of clinical images is hindered by the high cost and difficulties associated with both the accumulation of data and establishment of the ground truth. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a real lesion extracted from a source image into a different location on a target image. In this study we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the blended image by limiting user involvement to two simple steps: the user first draws a casual boundary around the nodule of interest in the source, and then selects the center of desired insertion area in the target. We demonstrate examples of the performance of the proposed system on samples taken from the Lung Image Database Consortium (LIDC) dataset, and compare the noise power spectrum (NPS) of blended nodules versus that of native nodules in simulated phantoms.
Purpose: The purpose of this work is to present a platform for designing and executing studies that compare pathologists interpreting histopathology of whole slide images (WSI) on a computer display to pathologists interpreting glass slides on an optical microscope. Methods: Here we present eeDAP, an evaluation environment for digital and analog pathology. The key element in eeDAP is the registration of theWSI to the glass slide. Registration is accomplished through computer control of the microscope stage and a camera mounted on the microscope that acquires images of the real time microscope view. Registration allows for the evaluation of the same regions of interest (ROIs) in both domains. This can reduce or eliminate disagreements that arise from pathologists interpreting different areas and focuses the comparison on image quality. Results: We reduced the pathologist interpretation area from an entire glass slide (≈10-30 mm)2 to small ROIs <(50 um)2. We also made possible the evaluation of individual cells. Conclusions: We summarize eeDAP’s software and hardware and provide calculations and corresponding images of the microscope field of view and the ROIs extracted from the WSIs. These calculations help provide a sense of eeDAP’s functionality and operating principles, while the images provide a sense of the look and feel of studies that can be conducted in the digital and analog domains. The eeDAP software can be downloaded from code.google.com (project: eeDAP) as Matlab source or as a precompiled stand-alone license-free application.
The majority of the literature on task-based image quality assessment has focused on lesion detection tasks, using the receiver operating characteristic (ROC) curve, or related variants, to measure performance. However, since many clinical image evaluation tasks involve both detection and estimation (e.g., estimation of kidney stone composition, estimation of tumor size), there is a growing interest in performance evaluation for joint detection and estimation tasks. To evaluate observer performance on such tasks, Clarkson introduced the estimation ROC (EROC) curve, and the area under the EROC curve as a summary figure of merit. In the present work, we propose nonparametric estimators for practical EROC analysis from experimental data, including estimators for the area under the EROC curve and its variance. The estimators are illustrated with a practical example comparing MRI images reconstructed from different k-space sampling trajectories.
Task-based image quality assessment is a valuable methodology for development, optimization and evaluation of new image formation processes in x-ray computed tomography (CT), as well as in other imaging modalities. A simple way to perform such an assessment is through the use of two (or more) alternative forced choice (AFC) experiments. In this paper, we are interested in drawing statistical inference from outcomes of multiple AFC experiments that are obtained using multiple readers as well as multiple cases. We present a non-parametric covariance estimator for this problem. Then, we illustrate its usefulness with a practical example involving x-ray CT simulations. The task for this example is classification between presence or absence of one lesion with unknown location within a given object. This task is used for comparison of three standard image reconstruction algorithms in x-ray CT using four human observers.
KEYWORDS: Reconstruction algorithms, Signal to noise ratio, Mathematical modeling, Kidney, Monte Carlo methods, Imaging systems, Medical imaging, Statistical analysis, Data modeling, Image quality
Task-based assessments of image quality constitute a rigorous, principled approach to the evaluation of imaging system performance. To conduct such assessments, it has been recognized that mathematical model observers are very useful, particularly for purposes of imaging system development and optimization. One type of model observer that has been widely applied in the medical imaging community is the channelized Hotelling observer (CHO). In the present work, we address the need for reliable confidence interval estimators of CHO performance. Specifically, we observe that a procedure proposed by Reiser for interval estimation of the Mahalanobis distance can be applied to obtain confidence intervals for CHO performance. In addition, we find that these intervals are well-defined with theoretically-exact coverage probabilities, which is a new result not proved by Reiser. The confidence intervals are tested with Monte Carlo simulation and demonstrated with an example comparing x-ray CT reconstruction strategies.
X-ray CT data collected in fan-beam geometry is often rebinned to parallel-beam geometry before image reconstruction.
In this work, we propose an exact, efficient method to analytically calculate the covariance between
image pixels for an FBP reconstruction when such a rebinning step is performed. As a special case, the method
also enables calculation of image variance. Our approach builds on a previous investigation of noise covariance
for direct fan-beam FBP reconstruction, and encompasses commonly used variants of parallel-beam rebinning
algorithms that include data interleaving steps to process fan-beam data acquired with either a quarter-detector
offset or a flying focal spot. The accuracy of the variance and covariance calculation results is validated through
comparison with Monte Carlo estimates.
In contrast to the ROC assessment paradigm, localization ROC (LROC) analysis provides a means to jointly
assess the accuracy of visual search and detection in an observer study. In a typical multireader, multicase
(MRMC) evaluation, the data sets are paired so that correlations arise in observer performance both between
observers and between image reconstruction methods (or modalities). Therefore,MRMC evaluations motivate the
need for a statistical methodology to compare correlated LROC curves. In this work, we suggest a nonparametric
strategy for this purpose. Specifically, we find that seminal work of Sen on U-statistics can be applied to estimate
the covariance matrix for a vector of LROC area estimates. The resulting covariance estimator is the LROC
analog of the covariance estimator given by DeLong et al. for ROC analysis. Once the covariance matrix is
estimated, it can be used to construct confidence intervals and/or confidence regions for purposes of comparing
observer performance across reconstruction methods. The utility of our covariance estimator is illustrated with
a human-observer LROC evaluation of three reconstruction strategies for fan-beam CT.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.