-
Perspectives on Medical Education Dec 2018Ongoing monitoring of cohort demographic variation is an essential part of quality assurance in medical education assessments, yet the methods employed to explore...
INTRODUCTION
Ongoing monitoring of cohort demographic variation is an essential part of quality assurance in medical education assessments, yet the methods employed to explore possible underlying causes of demographic variation in performance are limited. Focussing on properties of the vignette text in single-best-answer multiple-choice questions (MCQs), we explore here the viability of conducting analyses of text properties and their relationship to candidate performance. We suggest that such analyses could become routine parts of assessment evaluation and provide an additional, equality-based measure of an assessment's quality and fairness.
METHODS
We describe how a corpus of vignettes can be compiled, followed by examples of using Microsoft Word's native readability statistics calculator and the koRpus text analysis package for the R statistical analysis environment for estimating the following properties of the question text: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (Grade), word count, sentence count, and average words per sentence (WpS). We then provide examples of how these properties can be combined with equality and diversity variables, and the process automated to provide ongoing monitoring.
CONCLUSIONS
Given the monitoring of demographic differences in assessment for assurance of equality, the ability to easily include textual analysis of question vignettes provides a useful tool for exploring possible causes of demographic variations in performance where they occur. It also provides another means of evaluating assessment quality and fairness with respect to demographic characteristics. Microsoft Word provides data comparable to the specialized koRpus package, suggesting routine use of word processing software for writing items and assessing their properties is viable with minimal burden, but that automation for ongoing monitoring also provides an additional means of standardizing MCQ assessment items, and eliminating or controlling textual variables as a possible contributor to differential attainment between subgroups.
Topics: Cohort Studies; Communication Barriers; Comprehension; Educational Measurement; Female; Humans; Male; Quality Improvement; Test Taking Skills; Writing
PubMed: 30353285
DOI: 10.1007/s40037-018-0478-x -
A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2.Systematic Reviews Oct 2021AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another... (Review)
Review
BACKGROUND
AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions.
METHODS
Three reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet's agreement coefficient (AC for unweighted analysis and AC for weighted analysis).
RESULTS
Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher.
CONCLUSION
ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.
Topics: Bias; Humans; Reproducibility of Results; Systematic Reviews as Topic
PubMed: 34696810
DOI: 10.1186/s13643-021-01819-x -
Journal of Radiation Research Mar 2023After the Fukushima Daiichi Nuclear Power Plant (FDNPP) accident, individual exposure doses to residents have been assessed by many municipalities, governments and... (Review)
Review
External exposure assessment in the Fukushima accident area for governmental policy planning in Japan; Part 2. Matters to be attended for assessments of external exposure.
After the Fukushima Daiichi Nuclear Power Plant (FDNPP) accident, individual exposure doses to residents have been assessed by many municipalities, governments and research institutes. Various methods including measurements with personal dosimeters and simulations have been used for this evaluation depending on purposes, but the information of assessments and methods has not been systematically organized. A comprehensive review of the knowledge and experiences of individual exposure doses assessments accumulated so far and understanding the characteristics of the assessment methods will be very useful for radiation protection and risk communication, following to governmental policy planning. We reviewed the efforts made by the Japanese government and research institutes to assess radiation doses to residents after the FDNPS accident in Part 1. On the other hand, each method of assessing individual exposure doses includes uncertainties and points to be considered for the appropriate assessment. These knowledge and experiences are important for the assessment implementation and applying the assessment results to the governmental policy planning, and are summarized in Part 2 of this article.
Topics: Fukushima Nuclear Accident; Japan; Radiation Dosimeters; Radiation Protection; Radiation Monitoring; Nuclear Power Plants; Radiation Dosage
PubMed: 36610718
DOI: 10.1093/jrr/rrac088 -
Integrated Environmental Assessment and... Jun 2022Assessing the persistence of chemicals in the environment is a key element in existing regulatory frameworks to protect human health and ecosystems. Persistence in the... (Review)
Review
Assessing the persistence of chemicals in the environment is a key element in existing regulatory frameworks to protect human health and ecosystems. Persistence in the environment depends on many fate processes, including abiotic and biotic transformations and physical partitioning, which depend on substances' physicochemical properties and environmental conditions. A main challenge in persistence assessment is that existing frameworks rely on simplistic and reductionist evaluation schemes that may lead substances to be falsely assessed as persistent or the other way around-to be falsely assessed as nonpersistent. Those evaluation schemes typically assess persistence against degradation half-lives determined in single-compartment simulation tests or against degradation levels measured in stringent screening tests. Most of the available test methods, however, do not apply to all types of substances, especially substances that are poorly soluble, complex in composition, highly sorptive, or volatile. In addition, the currently applied half-life criteria are derived mainly from a few legacy persistent organic pollutants, which do not represent the large diversity of substances entering the environment. Persistence assessment would undoubtedly benefit from the development of more flexible and holistic evaluation schemes including new concepts and methods. A weight-of-evidence (WoE) approach incorporating multiple influencing factors is needed to account for chemical fate and transformation in the whole environment so as to assess overall persistence. The present paper's aim is to begin to develop an integrated assessment framework that combines multimedia approaches to organize and interpret data using a clear WoE approach to allow for a more consistent, transparent, and thorough assessment of persistence. Integr Environ Assess Manag 2022;18:868-887. © 2021 ExxonMobil Biomedical Sciences, Inc. Integrated Environmental Assessment and Management published by Wiley Periodicals LLC on behalf of Society of Environmental Toxicology & Chemistry (SETAC).
Topics: Ecosystem; Ecotoxicology; Environmental Monitoring; Environmental Pollution; Humans; Risk Assessment
PubMed: 34730270
DOI: 10.1002/ieam.4548 -
Bulletin of the World Health... Nov 2020To explore how primary care organizations assess and subsequently act upon the social determinants of noncommunicable diseases in their local populations. (Review)
Review
OBJECTIVE
To explore how primary care organizations assess and subsequently act upon the social determinants of noncommunicable diseases in their local populations.
METHODS
For this systematic review we searched the online databases of PubMed®, MEDLINE®, Embase® and the Health Management Information Consortium from inception to 28 June 2019, along with hand-searching of references. Studies of any design that examined a primary care organization assessing social determinants of noncommunicable diseases were included. For quality assessment we used Cochrane's tool for assessing risk of bias in non-randomized studies of interventions. We used narrative data synthesis to appraise the extent to which the assessments gathered data on the domains of the World Health Organization social determinants of health framework.
FINDINGS
We identified 666 studies of which 17 were included in the review. All studies used descriptive study designs. Clinic-based and household surveys and interviews were more commonly used to assess local social determinants than population-level data. We found no examples of organizations that assessed sociopolitical drivers of noncommunicable diseases; all focused on sociodemographic factors or circumstances of daily living. Nevertheless, the resulting actions to address social determinants ranged from individual-level interventions to population-wide measures and introducing representation of primary care organizations on system-level policy and planning committees.
CONCLUSION
Our findings may help policy-makers to consider suitable approaches for assessing and addressing social determinants of health in their domestic context. More rigorous observational and experimental evidence is needed to ascertain whether measuring social determinants leads to interventions which mitigate unmet social needs and reduce health disparities.
Topics: Humans; Noncommunicable Diseases; Primary Health Care; Social Determinants of Health
PubMed: 33177772
DOI: 10.2471/BLT.19.248278 -
Brain Sciences Mar 2021Aphasia assessment tools have primarily focused on classical aphasia type and severity, with minimal incorporation of recent findings that suggest a significant role of...
Aphasia assessment tools have primarily focused on classical aphasia type and severity, with minimal incorporation of recent findings that suggest a significant role of executive control operations in language generation. Assessment of the interface between language and executive functions is needed to improve detection of spontaneous speech difficulties. In this study we develop a new (BELS), a brief tool specifically designed to assess core language and executive functions shown to be involved in spontaneous generation of language. Similar to other measures of aphasia, the BELS assesses articulation and core language skills (repetition, naming and comprehension). Unique additions to the BELS include assessments of spontaneous connected speech, word fluency (phonemic/semantic) and sentence completion (verbal initiation, inhibition and selection). One-hundred and eight healthy controls and 136 stroke patients were recruited. Confirmatory factor analysis was used to determine construct validity and logistic regression was used to evaluate the discriminative validity, informing the final version of the BELS. The results showed that the BELS is sensitive for articulation and nominal language deficits, and it measures executive aspects of spontaneous language generation, which is a hallmark of frontal dynamic aphasia. The results have encouraging theoretical and practical implications.
PubMed: 33802073
DOI: 10.3390/brainsci11030353 -
Clinical & Translational Oncology :... May 2022Radiation-induced toxicity (RIT) is usually assessed by inspection and palpation. Due to their subjective and unquantitative nature, objective methods are required. This...
PURPOSE
Radiation-induced toxicity (RIT) is usually assessed by inspection and palpation. Due to their subjective and unquantitative nature, objective methods are required. This study aimed to determine whether a quantitative tool is able to assess RIT and establish an underlying BED-response relationship in breast cancer.
METHODS
Patients following seven different breast radiation protocols were recruited to this study for RIT assessment with qualitative and quantitative examination. The biologically equivalent dose (BED) was used to directly compare different radiation regimens. RIT was subjectively evaluated by physicians using the Radiation Therapy Oncology Group (RTOG) late toxicity scores. Simultaneously an objective multiprobe device was also used to quantitatively assess late RIT in terms of erythema, hyperpigmentation, elasticity and skin hydration.
RESULTS
In 194 patients, in terms of the objective measurements, treated breasts showed higher erythema and hyperpigmentation and lower elasticity and hydration than untreated breasts (p < 0.001, p < 0.001, p < 0.001, p = 0.019, respectively). As the BED increased, Δerythema and Δpigmentation gradually increased as well (p = 0.006 and p = 0.002, respectively). Regarding the clinical assessment, the increase in BED resulted in a higher RTOG toxicity grade (p < 0.001). Quantitative assessments were consistent with RTOG scores. As the RTOG toxicity grade increased, the erythema and pigmentation values increased, and the elasticity index decreased (p < 0.001, p = 0.016, p = 0.005, respectively).
CONCLUSIONS
The multiprobe device can be a sensitive and simple tool for research purpose and quantitatively assessing RIT in patients undergoing radiotherapy for breast cancer. Physician-assessed toxicity scores and objective measurements revealed that the BED was positively associated with the severity of RIT.
Topics: Breast; Breast Neoplasms; Erythema; Female; Humans; Hyperpigmentation; Radiation Injuries; Skin
PubMed: 34792726
DOI: 10.1007/s12094-021-02729-z -
Animals : An Open Access Journal From... Apr 2018Naturalness is considered important for animals, and is one criterion for assessing how we care for them. However, it is a vague and ambiguous term, which needs...
Naturalness is considered important for animals, and is one criterion for assessing how we care for them. However, it is a vague and ambiguous term, which needs definition and assessments suitable for scientific and ethical questions. This paper makes a start on that aim. This paper differentiates the term from other related concepts, such as species-typical behaviour and wellbeing. It identifies contingent ways in which naturalness might be used, as: (i) prompts for further welfare assessment; (ii) a plausible hypothesis for what safeguards wellbeing; (iii) a threshold for what is acceptable; (iv) constraints on what improvements are unacceptable; and (v) demarcating what is not morally wrong, because of a lack of human agency. It then suggests an approach to evaluating animals' behaviour that is quantitative, is based on reality, and which assesses naturalness by degrees. It proposes classing unaffected wild populations as natural by definition. Where animals might have been affected by humans, they should be compared to the closest population(s) of unaffected animals. This approach could allow us both to assess naturalness scientifically, and to make practical decisions about the behaviour of domestic animals.
PubMed: 29621140
DOI: 10.3390/ani8040053