-
Journal of Evaluation in Clinical... Oct 2023One of the criticisms of the operational/diagnostic criteria, generalised since DSM-III, has been that they were shaped solely to achieve the best inter-peer reliability... (Review)
Review
BACKGROUND
One of the criticisms of the operational/diagnostic criteria, generalised since DSM-III, has been that they were shaped solely to achieve the best inter-peer reliability with no considerations for validity. This does not fully reflect reality since throughout the development of the criteria, there was an effort to define and fulfil some validity requirements. However, despite several attempts to create alternative diagnostic systems, there is still a widespread misunderstanding of the epistemological foundations that support this paradigm.
METHODS
In this article, we intend to analyse the epistemological context in which the operational criteria (OC) emerged and some of the validation processes they have undergone since their conception.
RESULTS
On the epistemological basis of these operational criteria (OC) the influence of Hempel has been widely discussed. However, the group from St. Louis and, also the DSM-III editors, never openly acknowledged his role and his contribution and revealed other influences such as other medical specialties (that used and validated several OC in the diagnosis of their diseases). On the other hand, contrary to what has often been mentioned there has been a continuous attempt to validate the OC since their conception. In the implementation and development of the operational paradigm, a more instrumental trend was followed, focused on utility, but with successive attempts to achieve realistic validity by searching for biological or psychological causality. The methodologies were initially expert-driven and gradually more data-driven and included some variables external to the construct itself, such as familial aggregation, diagnostic consistency over time, prognostic and other psychometric measures.
PubMed: 37859515
DOI: 10.1111/jep.13933 -
International Journal of Language &... 2010Workplace-based learning is a critical component of professional preparation in speech pathology. A validated assessment of this learning is seen to be 'the gold... (Review)
Review
BACKGROUND
Workplace-based learning is a critical component of professional preparation in speech pathology. A validated assessment of this learning is seen to be 'the gold standard', but it is difficult to develop because of design and validation issues. These issues include the role and nature of judgement in assessment, challenges in measuring quality, and the relationship between assessment and learning. Valid assessment of workplace-based performance needs to capture the development of competence over time and account for both occupation specific and generic competencies.
AIMS
This paper reviews important conceptual issues in the design of valid and reliable workplace-based assessments of competence including assessment content, process, impact on learning, measurement issues, and validation strategies. It then goes on to share what has been learned about quality assessment and validation of a workplace-based performance assessment using competency-based ratings. The outcomes of a four-year national development and validation of an assessment tool are described.
METHODS & PROCEDURES
A literature review of issues in conceptualizing, designing, and validating workplace-based assessments was conducted. Key factors to consider in the design of a new tool were identified and built into the cycle of design, trialling, and data analysis in the validation stages of the development process.
OUTCOMES & RESULTS
This paper provides an accessible overview of factors to consider in the design and validation of workplace-based assessment tools. It presents strategies used in the development and national validation of a tool COMPASS, used in an every speech pathology programme in Australia, New Zealand, and Singapore. The paper also describes Rasch analysis, a model-based statistical approach which is useful for establishing validity and reliability of assessment tools.
CONCLUSIONS & IMPLICATIONS
Through careful attention to conceptual and design issues in the development and trialling of workplace-based assessments, it has been possible to develop the world's first valid and reliable national assessment tool for the assessment of performance in speech pathology.
Topics: Educational Measurement; Humans; Speech-Language Pathology
PubMed: 19424886
DOI: 10.3109/13682820902745461 -
Annales de Chirurgie Plastique Et... Apr 2021Simulation models are increasingly important for skill acquisition during microsurgery training. Prosthetics, living and non-living biological models have been proposed...
BACKGROUND
Simulation models are increasingly important for skill acquisition during microsurgery training. Prosthetics, living and non-living biological models have been proposed in the literature in the optics of recreating real-life scenarios in a controlled environment. This study aims to validate and prove the reusability of a novel non-living biological model: the porcine placenta.
METHODS
A prospective comparative study was carried out to assess face and content validities of the proposed model, as well as the reusability and quality of the Thiel-embalming method. Participants were asked answer a questionnaire for each anastomosis they performed on porcine placental vessels of ≤2mm (small) and 2-4mm (large). Scores were classified according to different subgroups, either small or large vessels and first or second sessions. Reliability analysis of the questionnaire was carried out using Cronbach's α, to ensure an α>0.7. Median scores for each question were analyzed using boxplots and compared amongst each subgroup using a non-parametric independent Mann-Whitney U test.
RESULTS
With nine participants, the Cronbach's α for each category of question was 0.867, 0.778, 0.720 and 0.593. Statistical differences were found between responses of small and large vessels on 5/10 questions, where large vessels reported higher validity. No statistical differences were found between scores of the first and second sessions.
CONCLUSION
By evaluating face and content validity, the Thiel-embalmed porcine placenta has proven its suitability as a microsurgery model, especially for vessels of larger caliber. Qualities that distinguish this model is its reliable reusability, its low cost-effectiveness, and its ethical acceptability.
Topics: Animals; Cadaver; Embalming; Female; Humans; Placenta; Pregnancy; Prospective Studies; Reproducibility of Results; Swine
PubMed: 33388177
DOI: 10.1016/j.anplas.2020.12.001 -
BioMed Research International 2019The Angioedema Activity Score (AAS) is recommended by the EAACI/GALEN/EDF/WAO guidelines for urticaria as the standard measure for assessing disease activity in patients... (Clinical Trial)
Clinical Trial
The Angioedema Activity Score (AAS) is recommended by the EAACI/GALEN/EDF/WAO guidelines for urticaria as the standard measure for assessing disease activity in patients with recurrent angioedema (RAE). To date, it has been translated into 80 languages for use in 52 countries, but it has not been formally validated in Asian patient populations. As RAE may be different in Asian and non-Asian patients, it is important to validate and characterize the reliability of tools to assess RAE disease activity in Asian patients. This study proposed to demonstrate the validity and reliability of the AAS in Asian patients. Accordingly, this study aimed to generate and validate the Thai version of the AAS and to characterize its reliability in Asian patients, specifically in Thailand. A structured translation was conducted with approval from the original authors. The Patient Global Assessment of Disease Activity (PGA-DA) was used as an instrument to compare with the Thai version of the AAS. In total, 86 patients with RAE participated in the study. Seventy-six (88%) patients had RAE with chronic spontaneous urticaria. The Thai AAS was found to be a valid and reliable instrument, with high convergent and known-groups validities, excellent internal consistency, and good test-retest reliability. The validity and reliability of the AAS for assessing RAE disease activity in Asian patients have been demonstrated by our study, making it the first to do so. This will help promote the use of the AAS, in clinical trials and practice, in Asia. It will also facilitate the comparison of disease activity in patients with RAE inside and outside Asia in future studies. However, a limitation of this study was its small number of patients.
Topics: Adolescent; Adult; Aged; Angioedema; Asian People; Female; Humans; Male; Middle Aged; Surveys and Questionnaires; Thailand; Translating; Urticaria
PubMed: 31781652
DOI: 10.1155/2019/9157895 -
Acta Psychiatrica Scandinavica Aug 2014To examine the empirical literature on diagnostic validators in borderline personality and bipolar illness. (Review)
Review
OBJECTIVE
To examine the empirical literature on diagnostic validators in borderline personality and bipolar illness.
METHOD
Using principles of evidence-based medicine, the highest levels of evidence were emphasized in interpretation of similarities or differences between bipolar illness and borderline personality on the five standard diagnostic validators in psychiatric nosology: symptoms, course, genetics, treatment response, and neurobiology.
RESULTS
Bipolar illness and borderline personality were found to be similar in the nosological validator of symptoms of mood lability and impulsivity, but differed notably on all other diagnostic validators, especially the course validator of past sexual abuse and the genetic validator of a bipolar family history. They also differ notably in the symptom validator of parasuicidal self-harm. Treatment response and neurobiological differences were also present and consistent.
CONCLUSION
This review of the literature indicates that these two conditions, bipolar illness and borderline personality, are different and can be distinguished. The much stronger biological and genetic evidence for bipolar illness in particular suggests that the two conditions can be reasonably seen as different kinds of clinical entities, namely a biological disease versus a psychosocially caused clinical picture. If this interpretation is correct, similarities between the two conditions, such as mood lability and impulsivity, are superficial, while differences are profound. Further, true comorbidity may be much less common than often presumed.
Topics: Bipolar Disorder; Borderline Personality Disorder; Humans
PubMed: 24571137
DOI: 10.1111/acps.12257 -
International Journal of Epidemiology Aug 2020Information bias is common in epidemiology and can substantially diminish the validity of study results. Validation studies, in which an investigator compares the...
Information bias is common in epidemiology and can substantially diminish the validity of study results. Validation studies, in which an investigator compares the accuracy of a measure with a gold standard measure, are an important way to understand and mitigate this bias. More attention is being paid to the importance of validation studies in recent years, yet they remain rare in epidemiologic research and, in our experience, they remain poorly understood. Many epidemiologists have not had any experience with validations studies, either in the classroom or in their work. We present an example of misclassification of a dichotomous exposure to elucidate some important misunderstandings about how to conduct validation studies to generate valid information. We demonstrate that careful attention to the design of validation studies is central to determining how the bias parameters (e.g. sensitivity and specificity or positive and negative predictive values) can be used in quantitative bias analyses to appropriately correct for misclassification. Whether sampling is done based on the true gold standard measure, the misclassified measure or at random will determine which parameters are valid and the precision of those estimates. Whether or not the validation is done stratified by other key variables (e.g. by the exposure) will also determine the validity of those estimates. We also present sample questions that can be used to teach these concepts. Increasing the presence of validation studies in the classroom could have a positive impact on their use and improve the validity of estimates of effect in epidemiologic research.
Topics: Bias; Epidemiologic Studies; Humans; Predictive Value of Tests; Sensitivity and Specificity
PubMed: 32617564
DOI: 10.1093/ije/dyaa090 -
Journal of Inflammation Research 2023A diabetic foot ulcer (DFU) is a serious, long-term condition associated with a significant risk of disability and mortality. However, research on its biomarkers is...
BACKGROUND
A diabetic foot ulcer (DFU) is a serious, long-term condition associated with a significant risk of disability and mortality. However, research on its biomarkers is still limited. This study utilizes bioinformatics and machine learning methods to identify immune-related biomarkers for DFU and validates them through external datasets and animal experiments.
METHODS
This study used bioinformatics and machine learning to analyze microarray data from the Gene Expression Omnibus (GEO) database to identify key genes associated with DFU. Animal experiments were conducted to validate these findings. This research employs the datasets GSE68183 and GSE80178 retrieved from the GEO database as the training dataset for building a gene machine learning model, and after conducting differential analysis on the data, this study used package glmnet and package e1071 to construct LASSO and SVM-RFE machine learning models, respectively. Subsequently, we validated the model using the training set and validation set (GSE134431). We conducted enrichment analysis, including GSEA and GSVA, on the model genes. We also performed immune functional analysis and immune-related analysis on the model genes. Finally, we conducted immunohistochemistry (IHC) validation on the model genes.
RESULTS
This study identifies GSTM5 as a potential immune-related key target in DFU using machine learning and bioinformatics methods. Subsequent validation through external datasets and IHC experiments also confirms GSTM5 as a critical biomarker for DFU. The gene may be associated with T cells regulatory (Tregs) and T cells follicular helper, and it influences the NF-κB, GnRH, and MAPK signaling pathway.
CONCLUSION
This study identified and validated GSTM5 as a biomarker for DFU. This finding may potentially provide a target for immune therapy for DFU.
PubMed: 38145013
DOI: 10.2147/JIR.S442388 -
Frontiers in Psychology 2023Internalized stigma among people living with HIV has been linked to a range of negative consequences. The current study describes the development and validation of a...
INTRODUCTION
Internalized stigma among people living with HIV has been linked to a range of negative consequences. The current study describes the development and validation of a contextually appropriate internalized HIV-related Stigma Scale for people living with HIV in Thailand.
METHODS
The study was carried out in two stages from 2018 to 2019: developing items based on the findings of focus group discussions and pilot testing the original list of items and validating the instrument. In the cross-sectional survey stage, a sample of 400 people living with HIV was used to validate the test items in accordance with their psychometric characteristics.
RESULTS
The study's outcome was the 22-item Thai Internalized HIV-related Stigma Scale (Thai-IHSS). The exploratory factor analysis showed that the Thai-IHSS consisted of four components: negative thoughts toward self (5 items), anticipated negative thoughts (7 items), effects of negative thought toward self (6 items), and effects of negative thoughts toward family and access to healthcare services (4 items).
DISCUSSION
The Thai-IHSS had acceptable concurrent, convergent, and congruent validity according to the findings. Additionally, the 8-item Thai-IHSS brief, which included two items for each component, was detailed. The Thai-IHSS is valid and reliable for use in Thailand and other countries with comparable sociocultural environments.
PubMed: 37008845
DOI: 10.3389/fpsyg.2023.1134648 -
The Cochrane Database of Systematic... Sep 2023Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system that affects millions of people worldwide. The disease course varies greatly... (Review)
Review
BACKGROUND
Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system that affects millions of people worldwide. The disease course varies greatly across individuals and many disease-modifying treatments with different safety and efficacy profiles have been developed recently. Prognostic models evaluated and shown to be valid in different settings have the potential to support people with MS and their physicians during the decision-making process for treatment or disease/life management, allow stratified and more precise interpretation of interventional trials, and provide insights into disease mechanisms. Many researchers have turned to prognostic models to help predict clinical outcomes in people with MS; however, to our knowledge, no widely accepted prognostic model for MS is being used in clinical practice yet.
OBJECTIVES
To identify and summarise multivariable prognostic models, and their validation studies for quantifying the risk of clinical disease progression, worsening, and activity in adults with MS.
SEARCH METHODS
We searched MEDLINE, Embase, and the Cochrane Database of Systematic Reviews from January 1996 until July 2021. We also screened the reference lists of included studies and relevant reviews, and references citing the included studies.
SELECTION CRITERIA
We included all statistically developed multivariable prognostic models aiming to predict clinical disease progression, worsening, and activity, as measured by disability, relapse, conversion to definite MS, conversion to progressive MS, or a composite of these in adult individuals with MS. We also included any studies evaluating the performance of (i.e. validating) these models. There were no restrictions based on language, data source, timing of prognostication, or timing of outcome.
DATA COLLECTION AND ANALYSIS
Pairs of review authors independently screened titles/abstracts and full texts, extracted data using a piloted form based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS), assessed risk of bias using the Prediction Model Risk Of Bias Assessment Tool (PROBAST), and assessed reporting deficiencies based on the checklist items in Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD). The characteristics of the included models and their validations are described narratively. We planned to meta-analyse the discrimination and calibration of models with at least three external validations outside the model development study but no model met this criterion. We summarised between-study heterogeneity narratively but again could not perform the planned meta-regression.
MAIN RESULTS
We included 57 studies, from which we identified 75 model developments, 15 external validations corresponding to only 12 (16%) of the models, and six author-reported validations. Only two models were externally validated multiple times. None of the identified external validations were performed by researchers independent of those that developed the model. The outcome was related to disease progression in 39 (41%), relapses in 8 (8%), conversion to definite MS in 17 (18%), and conversion to progressive MS in 27 (28%) of the 96 models or validations. The disease and treatment-related characteristics of included participants, and definitions of considered predictors and outcome, were highly heterogeneous amongst the studies. Based on the publication year, we observed an increase in the percent of participants on treatment, diversification of the diagnostic criteria used, an increase in consideration of biomarkers or treatment as predictors, and increased use of machine learning methods over time. Usability and reproducibility All identified models contained at least one predictor requiring the skills of a medical specialist for measurement or assessment. Most of the models (44; 59%) contained predictors that require specialist equipment likely to be absent from primary care or standard hospital settings. Over half (52%) of the developed models were not accompanied by model coefficients, tools, or instructions, which hinders their application, independent validation or reproduction. The data used in model developments were made publicly available or reported to be available on request only in a few studies (two and six, respectively). Risk of bias We rated all but one of the model developments or validations as having high overall risk of bias. The main reason for this was the statistical methods used for the development or evaluation of prognostic models; we rated all but two of the included model developments or validations as having high risk of bias in the analysis domain. None of the model developments that were externally validated or these models' external validations had low risk of bias. There were concerns related to applicability of the models to our research question in over one-third (38%) of the models or their validations. Reporting deficiencies Reporting was poor overall and there was no observable increase in the quality of reporting over time. The items that were unclearly reported or not reported at all for most of the included models or validations were related to sample size justification, blinding of outcome assessors, details of the full model or how to obtain predictions from it, amount of missing data, and treatments received by the participants. Reporting of preferred model performance measures of discrimination and calibration was suboptimal.
AUTHORS' CONCLUSIONS
The current evidence is not sufficient for recommending the use of any of the published prognostic prediction models for people with MS in clinical routine today due to lack of independent external validations. The MS prognostic research community should adhere to the current reporting and methodological guidelines and conduct many more state-of-the-art external validation studies for the existing or newly developed models.
Topics: Adult; Humans; Prognosis; Multiple Sclerosis; Reproducibility of Results; Systematic Reviews as Topic; Disease Progression
PubMed: 37681561
DOI: 10.1002/14651858.CD013606.pub2 -
Neuroscience and Biobehavioral Reviews May 2023The past 30 years of IBNS has included research attempting to treat the cognitive and behavioral deficits observed in people with psychiatric conditions. Early work... (Review)
Review
The past 30 years of IBNS has included research attempting to treat the cognitive and behavioral deficits observed in people with psychiatric conditions. Early work utilized drugs identified from tests thought to be cognition-relevant, however the high failure rate crossing the translational-species barrier led to focus on developing valid cross-species translational tests. The face, predictive, and neurobiological validities used to assess animal models of psychiatry can be used to validate these tests. Clinical sensitivity is another important aspect however, for if the clinical population targeted for treatment does not exhibit task deficits, then why develop treatments? This review covers some work validating cross-species translational tests and suggests future directions. Also covered is the contribution IBNS made to fostering such research and my role in IBNS, making it more available to all including fostering mentor/mentee programs plus spearheading diversity and inclusivity initiatives. All science needs support and IBNS has supported research recreating the behavioral abnormalities that define psychiatric conditions with the aim to improve the lives of people with such conditions.
Topics: Animals; Mental Disorders; Cognition; Disease Models, Animal; Neurobiology
PubMed: 36889561
DOI: 10.1016/j.neubiorev.2023.105119