-
Brain, Behavior, and Immunity Jan 2019Life stress is central to many contemporary theories of human health and behavior. Despite this fact, numerous conceptual and measurement issues remain unresolved. The...
Life stress is central to many contemporary theories of human health and behavior. Despite this fact, numerous conceptual and measurement issues remain unresolved. The present article explores these topics by first summarizing several key definitional and conceptual matters that are important for life stress research. Second, I introduce stressnology, defined herein as the fictitiously named, but otherwise very real and problematic approach to studying life stress exposure that involves measuring only the superficial contours of this very complex construct. Finally, I review some recent methodological advancements that have the potential to move us past primitive approaches to conceptualizing and assessing life stress. Ultimately, although the influence that life stress has on human health and behavior is profound, our understanding of this construct-and how it affects wellbeing, functioning, and development-is still very limited. Using state-of-the-art instruments for assessing life stress exposure, especially across the entire life course, should therefore be a top scientific and clinical priority.
Topics: Benchmarking; Humans; Life Change Events; Stress, Physiological; Stress, Psychological
PubMed: 30236597
DOI: 10.1016/j.bbi.2018.08.011 -
NeuroImage Aug 2022The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access...
The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access neuroimaging datasets, specifically designed benchmarks are rare in the field. In this article, we first describe how benchmarks in computer science and biomedical imaging have fostered methodological progress in machine learning. Second, we identify the special characteristics of neuroimaging data and outline what researchers have to ensure when establishing a neuroimaging benchmark, how datasets should be composed and how adequate evaluation criteria can be chosen. Based on lessons learned from machine learning benchmarks, we argue for an extended evaluation procedure that, next to applying suitable performance metrics, focuses on scientifically relevant aspects such as explainability, robustness, uncertainty, computational efficiency and code quality. Lastly, we envision a collaborative neuroimaging benchmarking platform that combines the discussed aspects in a collaborative and agile framework, allowing researchers across disciplines to work together on the key predictive problems of the field of neuroimaging and psychiatry.
Topics: Benchmarking; Humans; Machine Learning; Neuroimaging; Psychiatry
PubMed: 35561945
DOI: 10.1016/j.neuroimage.2022.119298 -
International Wound Journal Dec 2017Multidisciplinary wound centres are currently facing an increase in both the incidence of wounds and the complexity of care. This has resulted in rising costs and... (Comparative Study)
Comparative Study Review
Multidisciplinary wound centres are currently facing an increase in both the incidence of wounds and the complexity of care. This has resulted in rising costs and increased interest in the effectiveness of treatments. Little evidence is available regarding optimal wound centre organisation and effectiveness; therefore, measuring the quality of wound centres has become more important. This study aims to assess the evidence concerning quality by describing the state of the art of wound centres and organisational effectiveness by developing indicators of quality and by assessing their suitability in a pilot study. A multi-method approach was used: a literature review performed resulted in the development of an indicator list that was consequently subjected to expert review, and a benchmark study was completed comparing eight wound centres in the Netherlands. We thus provide a description of the relevant state-of-the-art aspects of wound centre organisation, which were multidisciplinary collaborations and standardisation of the organisation of care. In literature, significant patient-related effects were observed in improved healing rates and decreased costs. A total of 48 indicators were selected. The indicator list was tested by a benchmark study pilot. In practice, the outcome indicators were especially difficult to generate. Six indicators regarding structure, three regarding process and five regarding outcome proved feasible to measure and improve quality of wound centres.
Topics: Benchmarking; Health Facilities; Humans; Quality Indicators, Health Care; Skin Ulcer
PubMed: 28612454
DOI: 10.1111/iwj.12768 -
Briefings in Bioinformatics May 2023Since the 1980s, dozens of computational methods have addressed the problem of predicting RNA secondary structure. Among them are those that follow standard optimization... (Review)
Review
Since the 1980s, dozens of computational methods have addressed the problem of predicting RNA secondary structure. Among them are those that follow standard optimization approaches and, more recently, machine learning (ML) algorithms. The former were repeatedly benchmarked on various datasets. The latter, on the other hand, have not yet undergone extensive analysis that could suggest to the user which algorithm best fits the problem to be solved. In this review, we compare 15 methods that predict the secondary structure of RNA, of which 6 are based on deep learning (DL), 3 on shallow learning (SL) and 6 control methods on non-ML approaches. We discuss the ML strategies implemented and perform three experiments in which we evaluate the prediction of (I) representatives of the RNA equivalence classes, (II) selected Rfam sequences and (III) RNAs from new Rfam families. We show that DL-based algorithms (such as SPOT-RNA and UFold) can outperform SL and traditional methods if the data distribution is similar in the training and testing set. However, when predicting 2D structures for new RNA families, the advantage of DL is no longer clear, and its performance is inferior or equal to that of SL and non-ML methods.
Topics: Humans; RNA; Machine Learning; Algorithms; Benchmarking
PubMed: 37096592
DOI: 10.1093/bib/bbad153 -
Proteins Sep 2020Protein docking is essential for structural characterization of protein interactions. Besides providing the structure of protein complexes, modeling of proteins and...
Protein docking is essential for structural characterization of protein interactions. Besides providing the structure of protein complexes, modeling of proteins and their complexes is important for understanding the fundamental principles and specific aspects of protein interactions. The accuracy of protein modeling, in general, is still less than that of the experimental approaches. Thus, it is important to investigate the applicability of docking techniques to modeled proteins. We present new comprehensive benchmark sets of protein models for the development and validation of protein docking, as well as a systematic assessment of free and template-based docking techniques on these sets. As opposed to previous studies, the benchmark sets reflect the real case modeling/docking scenario where the accuracy of the models is assessed by the modeling procedure, without reference to the native structure (which would be unknown in practical applications). We also expanded the analysis to include docking of protein pairs where proteins have different structural accuracy. The results show that, in general, the template-based docking is less sensitive to the structural inaccuracies of the models than the free docking. The near-native docking poses generated by the template-based approach, typically, also have higher ranks than those produces by the free docking (although the free docking is indispensable in modeling the multiplicity of protein interactions in a crowded cellular environment). The results show that docking techniques are applicable to protein models in a broad range of modeling accuracy. The study provides clear guidelines for practical applications of docking to protein models.
Topics: Amino Acid Sequence; Benchmarking; Binding Sites; Databases, Protein; Molecular Docking Simulation; Protein Binding; Protein Structure, Secondary; Proteins; Software
PubMed: 32170770
DOI: 10.1002/prot.25889 -
BMC Medical Education Oct 2020Benchmarking across and within universities is a common tool to evaluate performance of a program and maintain accreditation requirements. While teaching remains a...
BACKGROUND
Benchmarking across and within universities is a common tool to evaluate performance of a program and maintain accreditation requirements. While teaching remains a primary responsibility of many academics, academic research performance is a major contributor towards career advancement and standards in the medical laboratory science profession. While anecdotal evidence suggests academics are active contributors to the evidence base of the profession, there is a high variability in research output in relation to institution and level of appointment. The aim of the study was to benchmark the research track record of Australian medical laboratory science academics and provide insight into how research productivity informs the level of appointment of academics across their career pathway.
METHODS
A bibliographic analysis of Australian medical laboratory science faculty websites and corresponding Scopus citation database profiles was conducted. A description of current research track record and relationships with holding a doctorate, academic appointment level, research and teaching interests, and institutional characteristics were explored. Quantitative data and frequencies were analysed using IBM SPSS version 26 to benchmark research track records by academic appointment level.
RESULTS
There were 124 academics identified from 13 universities who had a teaching and research position in an undergraduate medical laboratory science program in Australia. Academics at the level of lecturer or higher typically held a doctorate (89%). Holding a doctorate strongly influenced the number of publications. The top 20% of researchers authored around half of the overall publications. The majority of academics did not have alignment of their major research and teaching areas however, alignment had no influence on number of publications. There was, however, an inconsistent relationship between metropolitan or regional university location and the number of publications.
CONCLUSION
Data from this study provides academics with benchmarks for the research track record required at each level of appointment. When drawing conclusions on academic progression, promotion and tenure through research track record it would be mindful to assess each on a case by case basis. Institution (metropolitan versus regional) and research interest appears to influence publication number, h-index and citation scores.
Topics: Australia; Benchmarking; Bibliometrics; Faculty; Humans; Medical Laboratory Science
PubMed: 33059641
DOI: 10.1186/s12909-020-02298-9 -
Scientific Data Jun 2019Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in...
Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in digital clinical data available for analysis, but progress in machine learning for healthcare research has been difficult to measure because of the absence of publicly available benchmark data sets. To address this problem, we propose four clinical prediction benchmarks using data derived from the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database. These tasks cover a range of clinical problems including modeling risk of mortality, forecasting length of stay, detecting physiologic decline, and phenotype classification. We propose strong linear and neural baselines for all four tasks and evaluate the effect of deep supervision, multitask training and data-specific architectural modifications on the performance of neural models.
Topics: Benchmarking; Data Mining; Databases, Factual; Electronic Health Records; Humans; Machine Learning
PubMed: 31209213
DOI: 10.1038/s41597-019-0103-9 -
Journal of Chemical Information and... Jan 2023Force fields (FFs) for molecular simulation have been under development for more than half a century. As with any predictive model, rigorous testing and comparisons of... (Review)
Review
Force fields (FFs) for molecular simulation have been under development for more than half a century. As with any predictive model, rigorous testing and comparisons of models critically depends on the availability of standardized data sets and benchmarks. While such benchmarks are rather common in the fields of quantum chemistry, this is not the case for empirical FFs. That is, few benchmarks are reused to evaluate FFs, and development teams rather use their own training and test sets. Here we present an overview of currently available tests and benchmarks for computational chemistry, focusing on organic compounds, including halogens and common ions, as FFs for these are the most common ones. We argue that many of the benchmark data sets from quantum chemistry can in fact be reused for evaluating FFs, but new gas phase data is still needed for compounds containing phosphorus and sulfur in different valence states. In addition, more nonequilibrium interaction energies and forces, as well as molecular properties such as electrostatic potentials around compounds, would be beneficial. For the condensed phases there is a large body of experimental data available, and tools to utilize these data in an automated fashion are under development. If FF developers, as well as researchers in artificial intelligence, would adopt a number of these data sets, it would become easier to compare the relative strengths and weaknesses of different models and to, eventually, restore the balance in the force.
Topics: Benchmarking; Artificial Intelligence; Computer Simulation; Ions
PubMed: 36630710
DOI: 10.1021/acs.jcim.2c01127 -
Microbial Genomics Oct 2022Culture-independent metagenomic detection of microbial species has the potential to provide rapid and precise real-time diagnostic results. However, it is potentially...
Culture-independent metagenomic detection of microbial species has the potential to provide rapid and precise real-time diagnostic results. However, it is potentially limited by sequencing and taxonomic classification errors. We use simulated and real-world data to benchmark rates of species misclassification using 100 reference genomes for each of the ten common bloodstream pathogens and six frequent blood-culture contaminants (=1568, only 68 genomes were available for ). Simulating both with and without sequencing error for both the Illumina and Oxford Nanopore platforms, we evaluated commonly used classification tools including Kraken2, Bracken and Centrifuge, utilizing mini (8 GB) and standard (30-50 GB) databases. Bracken with the standard database performed best, the median percentage of reads across both sequencing platforms identified correctly to the species level was 97.8% (IQR 92.7:99.0) [range 5:100]. For Kraken2 with a mini database, a commonly used combination, median species-level identification was 86.4% (IQR 50.5:93.7) [range 4.3:100]. Classification performance varied by species, with being more challenging to classify correctly (probability of reads being assigned to the correct species: 56.1-96.0%, varying by tool used). Human read misclassification was negligible. By filtering out shorter Nanopore reads we found performance similar or superior to Illumina sequencing, despite higher sequencing error rates. Misclassification was more common when the misclassified species had a higher average nucleotide identity to the true species. Our findings highlight taxonomic misclassification of sequencing data occurs and varies by sequencing and analysis workflow. To account for 'bioinformatic contamination' we present a contamination catalogue that can be used in metagenomic pipelines to ensure accurate results that can support clinical decision making.
Topics: Humans; Nanopores; Benchmarking; Metagenomics; High-Throughput Nucleotide Sequencing; Nucleotides
PubMed: 36269282
DOI: 10.1099/mgen.0.000886 -
Health Services Research Aug 2021
Topics: Accountable Care Organizations; Benchmarking; Efficiency, Organizational; Health Expenditures; Humans; Reimbursement, Incentive; United States
PubMed: 34105147
DOI: 10.1111/1475-6773.13689