-
Journal of Biomedical Informatics May 2020This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs)... (Review)
Review
OBJECTIVES
This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies.
METHODOLOGY
A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy.
RESULTS
A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database.
CONCLUSION
This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.
Topics: Adult; Early Warning Score; Humans; Intensive Care Units; Models, Statistical; Prognosis; Vital Signs
PubMed: 32278089
DOI: 10.1016/j.jbi.2020.103410 -
The International Journal of Behavioral... Oct 2011The International Physical Activity Questionnaire-Short Form (IPAQ-SF) has been recommended as a cost-effective method to assess physical activity. Several studies... (Review)
Review
BACKGROUND
The International Physical Activity Questionnaire-Short Form (IPAQ-SF) has been recommended as a cost-effective method to assess physical activity. Several studies validating the IPAQ-SF have been conducted with differing results, but no systematic review of these studies has been reported.
METHODS
The keywords "IPAQ", "validation", and "validity" were searched in PubMed and Scopus. Studies published in English that validated the IPAQ-SF against an objective physical activity measuring device, doubly labeled water, or an objective fitness measure were included.
RESULTS
Twenty-three validation studies were included in this review. There was a great deal of variability in the methods used across studies, but the results were largely similar. Correlations between the total physical activity level measured by the IPAQ-SF and objective standards ranged from 0.09 to 0.39; none reached the minimal acceptable standard in the literature (0.50 for objective activity measuring devices, 0.40 for fitness measures). Correlations between sections of the IPAQ-SF for vigorous activity or moderate activity level/walking and an objective standard showed even greater variability (-0.18 to 0.76), yet several reached the minimal acceptable standard. Only six studies provided comparisons between physical activity levels derived from the IPAQ-SF and those obtained from objective criterion. In most studies the IPAQ-SF overestimated physical activity level by 36 to 173 percent; one study underestimated by 28 percent.
CONCLUSIONS
The correlation between the IPAQ-SF and objective measures of activity or fitness in the large majority of studies was lower than the acceptable standard. Furthermore, the IPAQ-SF typically overestimated physical activity as measured by objective criterion by an average of 84 percent. Hence, the evidence to support the use of the IPAQ-SF as an indicator of relative or absolute physical activity is weak.
Topics: Exercise; Humans; Physical Exertion; Physical Fitness; Reproducibility of Results; Surveys and Questionnaires; Validation Studies as Topic; Walking
PubMed: 22018588
DOI: 10.1186/1479-5868-8-115 -
The reliability and validity of ultrasound to quantify muscles in older adults: a systematic review.Journal of Cachexia, Sarcopenia and... Oct 2017This review evaluates the reliability and validity of ultrasound to quantify muscles in older adults. The databases PubMed, Cochrane, and Cumulative Index to Nursing and... (Meta-Analysis)
Meta-Analysis Review
This review evaluates the reliability and validity of ultrasound to quantify muscles in older adults. The databases PubMed, Cochrane, and Cumulative Index to Nursing and Allied Health Literature were systematically searched for studies. In 17 studies, the reliability (n = 13) and validity (n = 8) of ultrasound to quantify muscles in community-dwelling older adults (≥60 years) or a clinical population were evaluated. Four out of 13 reliability studies investigated both intra-rater and inter-rater reliability. Intraclass correlation coefficient (ICC) scores for reliability ranged from -0.26 to 1.00. The highest ICC scores were found for the vastus lateralis, rectus femoris, upper arm anterior, and the trunk (ICC = 0.72 to 1.000). All included validity studies found ICC scores ranging from 0.92 to 0.999. Two studies describing the validity of ultrasound to predict lean body mass showed good validity as compared with dual-energy X-ray absorptiometry (r = 0.92 to 0.96). This systematic review shows that ultrasound is a reliable and valid tool for the assessment of muscle size in older adults. More high-quality research is required to confirm these findings in both clinical and healthy populations. Furthermore, ultrasound assessment of small muscles needs further evaluation. Ultrasound to predict lean body mass is feasible; however, future research is required to validate prediction equations in older adults with varying function and health.
Topics: Age Factors; Aged; Aged, 80 and over; Body Composition; Humans; Muscle, Skeletal; Organ Size; Reproducibility of Results; Sarcopenia; Ultrasonography
PubMed: 28703496
DOI: 10.1002/jcsm.12210 -
JAMA Pediatrics Dec 2019Because children in a preverbal stage of development are unable to voice their feelings, they completely depend on their caregiving team for the interpretation and...
IMPORTANCE
Because children in a preverbal stage of development are unable to voice their feelings, they completely depend on their caregiving team for the interpretation and management of their pain and discomfort. Thus, accurately validated scales to assess pain and sedation levels are crucial.
OBJECTIVE
To provide clinicians a complete overview on the validity and reliability of the existing pain and sedation scales for different target populations (preterm infants, term infants, and toddlers) and in different clinical contexts.
EVIDENCE REVIEW
BIOSIS Previews, Cumulative Index to Nursing and Allied Health Literature, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Embase, MEDLINE, PsycCRITIQUES, PsycINFO, PSYNDEXplus Literature and Audiovisual Media, and PSYNDEXplus Tests were the databases screened from their inception to August 2018. All studies examining the validity or reliability of a given pain or sedation scale for patients in a preverbal stage of development were included in this systematic review. Those scales that were tested for at least construct validity, internal consistency, and interrater reliability were subsequently scored using the consensus-based standards for the selection of health measurement instruments (COSMIN) checklist.
FINDINGS
In total, 89 validation articles comprising 65 scales were included. Fifty-seven scales (88%) were useful for assessing pain, 13 scales (20%) for assessing sedation, and 4 scales (6%) for assessing both conditions. Forty-two (65%) were behavioral scales, and 23 (35%) were multidimensional scales. Eleven scales (17%) were validated for infants on mechanical ventilation. Thirty-seven scales (57%) were validated for preterm infants, 24 scales (37%) for term and preterm infants, 7 scales (11%) for term-born children, 7 scales (11%) for preterm infants, term infants, and toddlers, and 17 scales (26%) for term infants and toddlers. Twenty-eight scales (43%) considered construct validity, internal consistency, and interrater reliability.
CONCLUSIONS AND RELEVANCE
Clinicians should consider using scales that are validated for at least construct validity, internal consistency, and interrater reliability, combining this information with the population of interest and the construct the scale is intended to measure.
Topics: Child; Child, Preschool; Conscious Sedation; Humans; Infant; Infant, Newborn; Infant, Premature; Pain
PubMed: 31609437
DOI: 10.1001/jamapediatrics.2019.3351 -
BMJ (Clinical Research Ed.) May 2016To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population. (Review)
Review
OBJECTIVE
To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population.
DESIGN
Systematic review.
DATA SOURCES
Medline and Embase until June 2013.
ELIGIBILITY CRITERIA FOR STUDY SELECTION
Studies describing the development or external validation of a multivariable model for predicting CVD risk in the general population.
RESULTS
9965 references were screened, of which 212 articles were included in the review, describing the development of 363 prediction models and 473 external validations. Most models were developed in Europe (n=167, 46%), predicted risk of fatal or non-fatal coronary heart disease (n=118, 33%) over a 10 year period (n=209, 58%). The most common predictors were smoking (n=325, 90%) and age (n=321, 88%), and most models were sex specific (n=250, 69%). Substantial heterogeneity in predictor and outcome definitions was observed between models, and important clinical and methodological information were often missing. The prediction horizon was not specified for 49 models (13%), and for 92 (25%) crucial information was missing to enable the model to be used for individual risk prediction. Only 132 developed models (36%) were externally validated and only 70 (19%) by independent investigators. Model performance was heterogeneous and measures such as discrimination and calibration were reported for only 65% and 58% of the external validations, respectively.
CONCLUSIONS
There is an excess of models predicting incident CVD in the general population. The usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies. Rather than developing yet another similar CVD risk prediction model, in this era of large datasets, future research should focus on externally validating and comparing head-to-head promising CVD risk models that already exist, on tailoring or even combining these models to local settings, and investigating whether these models can be extended by addition of new predictors.
Topics: Cardiovascular Diseases; Female; Humans; Male; Models, Theoretical; Predictive Value of Tests; Risk Assessment; Risk Factors
PubMed: 27184143
DOI: 10.1136/bmj.i2416 -
Clinical Microbiology and Infection :... Aug 2021Only clinically validated HPV assays can be accepted in cervical cancer screening. (Meta-Analysis)
Meta-Analysis
BACKGROUND
Only clinically validated HPV assays can be accepted in cervical cancer screening.
OBJECTIVES
To update the list of high-risk HPV assays that fulfil the 2009 international validation criteria (Meijer-2009).
DATA SOURCES
PubMed/Medline, Embase, Scopus, references from selected studies; published in January 2014 to August 2020.
STUDY ELIGIBILITY CRITERIA
HPV test validation studies and primary screening studies, involving testing with an index HPV test and a comparator HPV test with reporting of disease outcome (occurrence of histologically confirmed cervical precancer; CIN2+).
PARTICIPANTS
Women participating in cervical cancer screening.
INTERVENTIONS
Testing with an index and a comparator HPV test of clinician-collected cervical specimens and assessment of disease outcome (
validation guidelines, or tests with consistent previous validations. METHODS
Assessment of relative clinical accuracy (including non-inferiority statistics index vs comparator assay) and test reproducibility in individual studies; random effects meta-analyses of the relative clinical sensitivity and specificity of index vs comparator tests.
RESULTS
Seven hrHPV DNA tests consistently fulfilled all validation criteria in multiple studies using predefined test positivity cut-offs (Abbott RealTime High Risk HPV, Anyplex II HPV HR Detection, BD Onclarity HPV Assay, Cobas 4800 HPV Test, HPV-Risk Assay, PapilloCheck HPV-Screening Test and Xpert HPV). Another assay (Alinity m HR HPV Assay) was fully validated in one validation study. The newer Cobas 6800 HPV Test, was validated in two studies against Cobas 4800. Other tests partially fulfilled the international validation criteria (Cervista HPV HR Test, EUROArray HPV, Hybribio's 14 High-Risk HPV, LMNX Genotyping Kit GP HPV, MALDI-TOF, RIATOL qPCR and a number of other in-house developed assays) since the non-inferior accuracy was reached after a posteriori cut-off optimization, inconsistent accuracy findings in different studies, and/or insufficient reproducibility assessment. The APTIMA HPV Assay targeting E6/E7 mRNA of hrHPV was fully validated in one formal validation study and showed slightly lower pooled sensitivity but higher specificity than the standard comparator tests in seven screening studies. However, the current international validation criteria relate to DNA assays. The additional requirement for longitudinal performance data required for non-DNA based HPV assays was not assessed in this review.
CONCLUSIONS
Eleven hrHPV DNA assays fulfil all requirements for use in cervical cancer screening using clinician-collected specimens.
Topics: Alphapapillomavirus; Early Detection of Cancer; Female; Genotyping Techniques; Humans; Papillomaviridae; Papillomavirus Infections; Reproducibility of Results; Sensitivity and Specificity; Uterine Cervical Neoplasms
PubMed: 33975008
DOI: 10.1016/j.cmi.2021.04.031 -
Neuropsychobiology 2023The utility of heart rate variability (HRV) for characterizing psychological stress is primarily impacted by methodological considerations such as study populations,... (Review)
Review
The utility of heart rate variability (HRV) for characterizing psychological stress is primarily impacted by methodological considerations such as study populations, experienced versus induced stress, and method of stress assessment. Here, we review studies on the associations between HRV and psychological stress, examining the nature of stress, ways stress was assessed, and HRV metrics used. The review was performed according to the PRISMA guidelines on select databases. Studies that examined the HRV-stress relationship via repeated measurements and validated psychometric instruments were included (n = 15). Participant numbers and ages ranged between 10 and 403 subjects and 18 and 60 years, respectively. Both experimental (n = 9) and real-life stress (n = 6) have been explored. While RMSSD was the most reported HRV metric (n = 10) significantly associated with stress, other metrics, including LF/HF (n = 7) and HF power (n = 6) were also reported. Various linear and nonlinear HRV metrics have been utilized, with nonlinear metrics used less often. The most frequently used psychometric instrument was the State-Trait Anxiety Inventory (n = 10), though various other instruments have been reported. In conclusion, HRV is a valid measure of the psychological stress response. Standard stress induction and assessment protocols combined with validated HRV measures in different domains will improve the validity of findings.
Topics: Humans; Adult; Heart Rate; Stress, Psychological
PubMed: 37290411
DOI: 10.1159/000530376 -
Psychology and Psychotherapy Mar 2021Current models of psychosis posit a developmental and maintaining role for anxiety, supported by robust empirical evidence. Given the central role for anxiety in... (Review)
Review
PURPOSE
Current models of psychosis posit a developmental and maintaining role for anxiety, supported by robust empirical evidence. Given the central role for anxiety in psychosis, valid and reliable assessment is necessary. This systematic review is the first to critically appraise measures of anxiety applied to psychosis samples.
METHODS
Web of Science, MEDLINE, PsycINFO, EMBASE, and CINAHL were systematically searched for studies evaluating psychometric properties of instruments measuring anxiety in samples with non-affective psychosis diagnoses. Psychometric properties were extracted and rated according to established criteria. The COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist was used to assess the methodological quality of studies.
RESULTS
Of 4,344 records identified, 11 studies were identified as eligible for inclusion, by two independent raters with high reliability. Reported psychometric properties for 17 different instruments ranged from unacceptable to excellent. The Scale of Anxiety Evaluation in Schizophrenia was assessed most extensively and demonstrated consistently good psychometric properties. The Beck Anxiety Index, Depression Anxiety Stress Scale, DSM-based Generalised Anxiety Disorder Symptoms Severity Scale, Liebowitz Social Anxiety Scale, Obsessive-Compulsive Inventory, Psychological Stress Index, Perseverative Thinking Questionnaire, and Yale-Brown Obsessive Compulsive Scale demonstrated adequate reliability and/or validity on the limited properties reported. Methodological quality was largely poor according to the requirements of the COSMIN checklist.
CONCLUSIONS
The instruments listed are recommended as at least adequate for the assessment of anxiety in psychosis on the basis of these preliminary data. Further validation of existing instruments designed to measure anxiety in people with psychosis is strongly recommended.
PRACTITIONER POINTS
Anxiety has a developmental and maintaining role in psychosis; therefore, we should routinely screen for symptoms of anxiety when working with people with psychosis spectrum disorders. Studies included in this review reported reliable and valid measures of anxiety for people with psychosis; however, the methodological quality of most studies was poor. We recommend the BAI, DASS, or SAES for general screening, and the DGSS, LSAS, OCI, PSI, PTQ, and Y-BOCS to assess symptoms associated with specific anxiety disorders and anxiety-related processes.
Topics: Anxiety; Anxiety Disorders; Humans; Psychometrics; Psychotic Disorders; Reproducibility of Results
PubMed: 31880406
DOI: 10.1111/papt.12265 -
Global Mental Health (Cambridge,... 2023Implementing culturally sensitive psychometric measures of depression may be an effective strategy to improve acceptance, response rate, and reliability of psychological... (Review)
Review
BACKGROUND
Implementing culturally sensitive psychometric measures of depression may be an effective strategy to improve acceptance, response rate, and reliability of psychological assessment among Indigenous populations. However, the psychometric properties of depression scales after cultural adaptation remain unclear.
METHODS
We screened the Ovid Medline, PubMed, Embase, Global Health, PsycInfo, and CINAHL databases through three levels of search terms: Depression, Psychometrics, and Indigenous, following the PRISMA guidelines. We assessed metrics for reliability (including Cronbach's alpha), validity (including fit indices), and clinical utility (including predictive value).
RESULTS
Across 31 studies included the review, 13 different depression scales were adapted through language or content modification. Sample populations included Indigenous from the Americas, Asia, Africa, and Oceania. Most cultural adaptations had strong psychometric properties; however, few and inconsistent properties were reported. Where available, alphas, inter-rater and test-retest reliability, construct validity, and incremental validity often indicated increased cultural sensitivity of adapted scales. There were mixed results for clinical utility, criterion validity, cross-cultural validity, sensitivity, specificity, area under the receiver operating characteristic curve, predictive value, and likelihood ratio.
CONCLUSIONS
Modifications to increase cultural relevance have the potential to improve fit and acceptance of a scale by the Indigenous population, however, these changes may decrease specificity and negative predictive value. There is an urgent need for suitable tools that are useful and reliable for identifying Indigenous individuals for clinical treatment of depression. This awaits future work for optimal specificity and validated cut-off points that take into account the high prevalence of depression in these populations.
PubMed: 37854390
DOI: 10.1017/gmh.2023.52 -
Frontiers in Immunology 2023To identify the risk factors associated with prognosis in patients with hepatocellular carcinoma (HCC) treated with immune checkpoint inhibitors (ICI) via meta-analysis.... (Meta-Analysis)
Meta-Analysis
Development and validation of prognostic risk prediction models for hepatocellular carcinoma patients treated with immune checkpoint inhibitors based on a systematic review and meta-analysis of 47 cohorts.
OBJECTIVE
To identify the risk factors associated with prognosis in patients with hepatocellular carcinoma (HCC) treated with immune checkpoint inhibitors (ICI) via meta-analysis. And to construct prediction models to aid in the prediction and improvement of prognosis.
METHODS
We searched PubMed, Embase, Web of Science and Cochrane Library for relevant studies from inception to March 29, 2023. After completing literature screening and data extraction, we performed meta-analysis, sensitivity analysis, and subgroup analysis to identify risk factors associated with OS and PFS. Using the pooled hazard ratio value for each risk factor, we constructed prediction models, which were then validated using datasets from 19 centers in Japan and two centers in China, comprising a total of 204 patients.
RESULTS
A total of 47 studies, involving a total of 7649 ICI-treated HCC patients, were included in the meta-analysis. After analyzing 18 risk factors, we identified AFP, ALBI, NLR, ECOG performance status, Child-Pugh stage, BCLC stage, tumor number, vascular invasion and combination therapy as predictors for OS prediction model, while AFP, ALBI, NLR, ECOG performance status, Child-Pugh stage, BCLC stage, tumor number and vascular invasion were selected as predictors for PFS model. To validate the models, we scored two independent cohorts of patients using both prediction models. Our models demonstrated good performance in these cohorts. In addition, in the pooled cohort of 204 patients, Our models also showed good performance with area under the curve (AUC) values of 0.712, 0.753, and 0.822 for the OS prediction model at 1-year, 2-year, and 3-year follow-up points, respectively, and AUC values of 0.575, 0.749 and 0.691 for the PFS prediction model Additionally, the calibration curve, decision curve analysis, and Kaplan-Meier curves in the pooled cohort all supported the validity of both models.
CONCLUSION
Based on the meta-analysis, we successfully constructed the OS and PFS prediction models for ICI-treated HCC patients. We also validated the models externally and observed good discrimination and calibration. The model's selected indicators are easily obtainable, making them suitable for further application in clinical practice.
Topics: Humans; Carcinoma, Hepatocellular; Prognosis; Immune Checkpoint Inhibitors; Liver Neoplasms; alpha-Fetoproteins
PubMed: 37520554
DOI: 10.3389/fimmu.2023.1215745