-
Cell Aug 2019Metagenomic sequencing is revolutionizing the detection and characterization of microbial species, and a wide variety of software tools are available to perform... (Review)
Review
Metagenomic sequencing is revolutionizing the detection and characterization of microbial species, and a wide variety of software tools are available to perform taxonomic classification of these data. The fast pace of development of these tools and the complexity of metagenomic data make it important that researchers are able to benchmark their performance. Here, we review current approaches for metagenomic analysis and evaluate the performance of 20 metagenomic classifiers using simulated and experimental datasets. We describe the key metrics used to assess performance, offer a framework for the comparison of additional classifiers, and discuss the future of metagenomic data analysis.
Topics: Bacteria; Benchmarking; Databases, Genetic; Fungi; Metagenome; Metagenomics; Phylogeny; Polymerase Chain Reaction; Sequence Analysis, DNA; Software; Viruses
PubMed: 31398336
DOI: 10.1016/j.cell.2019.07.010 -
Nature Communications Jan 2023A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA)...
A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA) data processing. However, how the combination of a DIA software tool and a spectral library impacts the outcome of DIA proteomics and phosphoproteomics data analysis has been rarely investigated using benchmark data that mimics biological complexity. In this study, we create DIA benchmark data sets simulating the regulation of thousands of proteins in a complex background, which are collected on both an Orbitrap and a timsTOF instruments. We evaluate four commonly used software suites (DIA-NN, Spectronaut, MaxDIA and Skyline) combined with seven different spectral libraries in global proteome analysis. Moreover, we assess their performances in analyzing phosphopeptide standards and TNF-α-induced phosphoproteome regulation. Our study provides a practical guidance on how to construct a robust data analysis pipeline for different proteomics studies implementing the DIA technique.
Topics: Proteomics; Benchmarking; Workflow; Mass Spectrometry; Software; Proteome
PubMed: 36609502
DOI: 10.1038/s41467-022-35740-1 -
Nature Biotechnology Dec 2022Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of...
Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of highest-grossing individual products.
Topics: Biological Products; Benchmarking; Drug Industry; Drug Approval
PubMed: 36471135
DOI: 10.1038/s41587-022-01582-x -
Annual Review of Biophysics May 2017Binding free energy calculations based on molecular simulations provide predicted affinities for biomolecular complexes. These calculations begin with a detailed... (Review)
Review
Binding free energy calculations based on molecular simulations provide predicted affinities for biomolecular complexes. These calculations begin with a detailed description of a system, including its chemical composition and the interactions among its components. Simulations of the system are then used to compute thermodynamic information, such as binding affinities. Because of their promise for guiding molecular design, these calculations have recently begun to see widespread applications in early-stage drug discovery. However, many hurdles remain in making them a robust and reliable tool. In this review, we highlight key challenges of these calculations, discuss some examples of these challenges, and call for the designation of standard community benchmark test systems that will help the research community generate and evaluate progress. In our view, progress will require careful assessment and evaluation of new methods, force fields, and modeling innovations on well-characterized benchmark systems, and we lay out our vision for how this can be achieved.
Topics: Benchmarking; Bridged-Ring Compounds; Computer Simulation; Drug Discovery; Imidazoles; Ligands; Models, Molecular; Muramidase; Protein Binding; Proteins; Software; Thermodynamics
PubMed: 28399632
DOI: 10.1146/annurev-biophys-070816-033654 -
BMC Health Services Research Apr 2017Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on... (Review)
Review
BACKGROUND
Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics.
METHODS
We searched PubMed and EMBASE for articles published in English in the last 10 years. Eligible articles described a project stating benchmarking as its objective and involving a specialty hospital or specific patient category; or those dealing with the methodology or evaluation of benchmarking.
RESULTS
Of 1,817 articles identified in total, 24 were included in the study. Articles were categorized into: pathway benchmarking, institutional benchmarking, articles on benchmark methodology or -evaluation and benchmarking using a patient registry. There was a large degree of variability:(1) study designs were mostly descriptive and retrospective; (2) not all studies generated and showed data in sufficient detail; and (3) there was variety in whether a benchmarking model was just described or if quality improvement as a consequence of the benchmark was reported upon. Most of the studies that described a benchmark model described the use of benchmarking partners from the same industry category, sometimes from all over the world.
CONCLUSIONS
Benchmarking seems to be more developed in eye hospitals, emergency departments and oncology specialty hospitals. Some studies showed promising improvement effects. However, the majority of the articles lacked a structured design, and did not report on benchmark outcomes. In order to evaluate the effectiveness of benchmarking to improve quality in specialty hospitals, robust and structured designs are needed including a follow up to check whether the benchmark study has led to improvements.
Topics: Benchmarking; Emergency Service, Hospital; Hospitals, Special; Humans; Models, Theoretical; Quality Improvement; Retrospective Studies
PubMed: 28372574
DOI: 10.1186/s12913-017-2154-y -
International Journal of Surgery... Mar 2023Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and... (Review)
Review
INTRODUCTION
Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and evolution of benchmarking and its application in surgery.
METHODS
The literature about benchmarking was reviewed through an ever-narrowing search strategy, commencing from the concept, definition, and evolution of benchmarking to the application of benchmarking and experiences of benchmarking in surgery. PubMed, Web of Science, Embase, and Science Direct databases were searched until 20 September 2022, in the English language according to the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines.
RESULTS
In the first phase of the literature search, the development of benchmarking was identified. The definitions of benchmarking evolved from a surveying term to a novel quality-improvement tool to assess the best achievable results in surgery. In the second phase, a total of 23 studies were identified about benchmarking in surgery, including esophagectomy, hepatic surgery, pancreatic surgery, rectum resection, and bariatric surgery. All studies were multicenter analyses from national, international, or global expert centers. Most studies (87.0%) adopted the definition that benchmark was the 75th percentile of the median values of centers. Performance metrics to define benchmarks were clinically relevant intraoperative and postoperative outcome indicators.
CONCLUSION
Benchmarking in surgery is a novel quality-improvement tool to define and measure the best achievable results, establishing a meaningful reference to evaluate surgical performance.
Topics: Humans; Benchmarking; Postoperative Complications; Esophagectomy; Bariatric Surgery; Multicenter Studies as Topic
PubMed: 37093075
DOI: 10.1097/JS9.0000000000000212 -
Sensors (Basel, Switzerland) Dec 2021Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such... (Review)
Review
Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such supervised approaches is that the training and testing data are drawn from the same underlying data distribution. This may not always be the case, and in such cases, the performance of the model is degraded. Domain adaptation aims to overcome the domain shift between the source domain used for training and the target domain data used for testing. Unsupervised domain adaptation deals with situations where the network is trained on labeled data from the source domain and unlabeled data from the target domain with the goal of performing well on the target domain data at the time of deployment. In this study, we overview seven state-of-the-art unsupervised domain adaptation models based on deep learning and benchmark their performance on three new domain adaptation datasets created from publicly available aerial datasets. We believe this is the first study on benchmarking domain adaptation methods for aerial data. In addition to reporting classification performance for the different domain adaptation models, we present t-SNE visualizations that illustrate the benefits of the adaptation process.
Topics: Adaptation, Physiological; Benchmarking
PubMed: 34884072
DOI: 10.3390/s21238070 -
The Journal of Thoracic and... May 2016
Topics: Benchmarking; Dose-Response Relationship, Drug; Humans; Risk Assessment
PubMed: 27085614
DOI: 10.1016/j.jtcvs.2016.03.005 -
American Journal of Transplantation :... Sep 2018
Topics: Benchmarking; Humans; Kidney Transplantation; Living Donors; Tissue and Organ Procurement
PubMed: 29791069
DOI: 10.1111/ajt.14947 -
Sensors (Basel, Switzerland) Aug 2021Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives...
Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives which can be exploited to improve the robustness of anomaly detection. In this work, we build upon the deep support vector data description algorithm and address multi-perspective anomaly detection using three different fusion techniques, i.e., early fusion, late fusion, and late fusion with multiple decoders. We employ different augmentation techniques with a denoising process to deal with scarce one-class data, which further improves the performance (ROC AUC =80%). Furthermore, we introduce the dices dataset, which consists of over 2000 grayscale images of falling dices from multiple perspectives, with 5% of the images containing rare anomalies (e.g., drill holes, sawing, or scratches). We evaluate our approach on the new dices dataset using images from two different perspectives and also benchmark on the standard MNIST dataset. Extensive experiments demonstrate that our proposed multi-perspective approach exceeds the state-of-the-art single-perspective anomaly detection on both the MNIST and dices datasets. To the best of our knowledge, this is the first work that focuses on addressing multi-perspective anomaly detection in images by jointly using different perspectives together with one single objective function for anomaly detection.
Topics: Algorithms; Benchmarking
PubMed: 34450753
DOI: 10.3390/s21165311