-
Cell Aug 2019Metagenomic sequencing is revolutionizing the detection and characterization of microbial species, and a wide variety of software tools are available to perform... (Review)
Review
Metagenomic sequencing is revolutionizing the detection and characterization of microbial species, and a wide variety of software tools are available to perform taxonomic classification of these data. The fast pace of development of these tools and the complexity of metagenomic data make it important that researchers are able to benchmark their performance. Here, we review current approaches for metagenomic analysis and evaluate the performance of 20 metagenomic classifiers using simulated and experimental datasets. We describe the key metrics used to assess performance, offer a framework for the comparison of additional classifiers, and discuss the future of metagenomic data analysis.
Topics: Bacteria; Benchmarking; Databases, Genetic; Fungi; Metagenome; Metagenomics; Phylogeny; Polymerase Chain Reaction; Sequence Analysis, DNA; Software; Viruses
PubMed: 31398336
DOI: 10.1016/j.cell.2019.07.010 -
Nature Communications Jan 2023A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA)...
A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA) data processing. However, how the combination of a DIA software tool and a spectral library impacts the outcome of DIA proteomics and phosphoproteomics data analysis has been rarely investigated using benchmark data that mimics biological complexity. In this study, we create DIA benchmark data sets simulating the regulation of thousands of proteins in a complex background, which are collected on both an Orbitrap and a timsTOF instruments. We evaluate four commonly used software suites (DIA-NN, Spectronaut, MaxDIA and Skyline) combined with seven different spectral libraries in global proteome analysis. Moreover, we assess their performances in analyzing phosphopeptide standards and TNF-α-induced phosphoproteome regulation. Our study provides a practical guidance on how to construct a robust data analysis pipeline for different proteomics studies implementing the DIA technique.
Topics: Proteomics; Benchmarking; Workflow; Mass Spectrometry; Software; Proteome
PubMed: 36609502
DOI: 10.1038/s41467-022-35740-1 -
Cell Systems Feb 2021In single-cell RNA sequencing (scRNA-seq), doublets form when two cells are encapsulated into one reaction volume. The existence of doublets, which appear to be-but are...
In single-cell RNA sequencing (scRNA-seq), doublets form when two cells are encapsulated into one reaction volume. The existence of doublets, which appear to be-but are not-real cells, is a key confounder in scRNA-seq data analysis. Computational methods have been developed to detect doublets in scRNA-seq data; however, the scRNA-seq field lacks a comprehensive benchmarking of these methods, making it difficult for researchers to choose an appropriate method for specific analyses. We conducted a systematic benchmark study of nine cutting-edge computational doublet-detection methods. Our study included 16 real datasets, which contained experimentally annotated doublets, and 112 realistic synthetic datasets. We compared doublet-detection methods regarding detection accuracy under various experimental settings, impacts on downstream analyses, and computational efficiencies. Our results show that existing methods exhibited diverse performance and distinct advantages in different aspects. Overall, the DoubletFinder method has the best detection accuracy, and the cxds method has the highest computational efficiency. A record of this paper's transparent peer review process is included in the Supplemental Information.
Topics: Benchmarking; Humans; RNA-Seq; Single-Cell Analysis
PubMed: 33338399
DOI: 10.1016/j.cels.2020.11.008 -
Nature Biotechnology Dec 2022Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of...
Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of highest-grossing individual products.
Topics: Biological Products; Benchmarking; Drug Industry; Drug Approval
PubMed: 36471135
DOI: 10.1038/s41587-022-01582-x -
Sensors (Basel, Switzerland) Dec 2021Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such... (Review)
Review
Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such supervised approaches is that the training and testing data are drawn from the same underlying data distribution. This may not always be the case, and in such cases, the performance of the model is degraded. Domain adaptation aims to overcome the domain shift between the source domain used for training and the target domain data used for testing. Unsupervised domain adaptation deals with situations where the network is trained on labeled data from the source domain and unlabeled data from the target domain with the goal of performing well on the target domain data at the time of deployment. In this study, we overview seven state-of-the-art unsupervised domain adaptation models based on deep learning and benchmark their performance on three new domain adaptation datasets created from publicly available aerial datasets. We believe this is the first study on benchmarking domain adaptation methods for aerial data. In addition to reporting classification performance for the different domain adaptation models, we present t-SNE visualizations that illustrate the benefits of the adaptation process.
Topics: Adaptation, Physiological; Benchmarking
PubMed: 34884072
DOI: 10.3390/s21238070 -
International Journal of Surgery... Mar 2023Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and... (Review)
Review
INTRODUCTION
Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and evolution of benchmarking and its application in surgery.
METHODS
The literature about benchmarking was reviewed through an ever-narrowing search strategy, commencing from the concept, definition, and evolution of benchmarking to the application of benchmarking and experiences of benchmarking in surgery. PubMed, Web of Science, Embase, and Science Direct databases were searched until 20 September 2022, in the English language according to the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines.
RESULTS
In the first phase of the literature search, the development of benchmarking was identified. The definitions of benchmarking evolved from a surveying term to a novel quality-improvement tool to assess the best achievable results in surgery. In the second phase, a total of 23 studies were identified about benchmarking in surgery, including esophagectomy, hepatic surgery, pancreatic surgery, rectum resection, and bariatric surgery. All studies were multicenter analyses from national, international, or global expert centers. Most studies (87.0%) adopted the definition that benchmark was the 75th percentile of the median values of centers. Performance metrics to define benchmarks were clinically relevant intraoperative and postoperative outcome indicators.
CONCLUSION
Benchmarking in surgery is a novel quality-improvement tool to define and measure the best achievable results, establishing a meaningful reference to evaluate surgical performance.
Topics: Humans; Benchmarking; Postoperative Complications; Esophagectomy; Bariatric Surgery; Multicenter Studies as Topic
PubMed: 37093075
DOI: 10.1097/JS9.0000000000000212 -
Sensors (Basel, Switzerland) Aug 2021Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives...
Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives which can be exploited to improve the robustness of anomaly detection. In this work, we build upon the deep support vector data description algorithm and address multi-perspective anomaly detection using three different fusion techniques, i.e., early fusion, late fusion, and late fusion with multiple decoders. We employ different augmentation techniques with a denoising process to deal with scarce one-class data, which further improves the performance (ROC AUC =80%). Furthermore, we introduce the dices dataset, which consists of over 2000 grayscale images of falling dices from multiple perspectives, with 5% of the images containing rare anomalies (e.g., drill holes, sawing, or scratches). We evaluate our approach on the new dices dataset using images from two different perspectives and also benchmark on the standard MNIST dataset. Extensive experiments demonstrate that our proposed multi-perspective approach exceeds the state-of-the-art single-perspective anomaly detection on both the MNIST and dices datasets. To the best of our knowledge, this is the first work that focuses on addressing multi-perspective anomaly detection in images by jointly using different perspectives together with one single objective function for anomaly detection.
Topics: Algorithms; Benchmarking
PubMed: 34450753
DOI: 10.3390/s21165311 -
Journal of Hepatology Feb 2022The concept of benchmarking is established in the field of transplant surgery; however, benchmark values for donation after circulatory death (DCD) liver transplantation...
BACKGROUND & AIMS
The concept of benchmarking is established in the field of transplant surgery; however, benchmark values for donation after circulatory death (DCD) liver transplantation are not available. Thus, we aimed to identify the best possible outcomes in DCD liver transplantation and to propose outcome reference values.
METHODS
Based on 2,219 controlled DCD liver transplantations, collected from 17 centres in North America and Europe, we identified 1,012 low-risk, primary, adult liver transplantations with a laboratory MELD score of ≤20 points, receiving a DCD liver with a total donor warm ischemia time of ≤30 minutes and asystolic donor warm ischemia time of ≤15 minutes. Clinically relevant outcomes were selected and complications were reported according to the Clavien-Dindo-Grading and the comprehensive complication index (CCI). Corresponding benchmark cut-offs were based on median values of each centre, where the 75-percentile was considered.
RESULTS
Benchmark cases represented between 19.7% and 75% of DCD transplantations in participating centres. The 1-year retransplant and mortality rates were 4.5% and 8.4% in the benchmark group, respectively. Within the first year of follow-up, 51.1% of recipients developed at least 1 major complication (≥Clavien-Dindo-Grade III). Benchmark cut-offs were ≤3 days and ≤16 days for ICU and hospital stay, ≤66% for severe recipient complications (≥Grade III), ≤16.8% for ischemic cholangiopathy, and ≤38.9 CCI points 1 year after transplant. Comparisons with higher risk groups showed more complications and impaired graft survival outside the benchmark cut-offs. Organ perfusion techniques reduced the complications to values below benchmark cut-offs, despite higher graft risk.
CONCLUSIONS
Despite excellent 1-year survival, morbidity in benchmark cases remains high. Benchmark cut-offs targeting morbidity parameters offer a valid tool to assess the protective value of new preservation technologies in higher risk groups and to provide a valid comparator cohort for future clinical trials.
LAY SUMMARY
The best possible outcomes after liver transplantation of grafts donated after circulatory death (DCD) were defined using the concept of benchmarking. These were based on 2,219 liver transplantations following controlled DCD donation in 17 centres worldwide. Donor and recipient combinations with higher risk had significantly worse outcomes. However, the use of novel organ perfusion technology helped high-risk patients achieve similar outcomes as the benchmark cohort.
Topics: Aged; Area Under Curve; Benchmarking; Cohort Studies; Female; Humans; Kaplan-Meier Estimate; Liver Transplantation; Male; Middle Aged; Organ Dysfunction Scores; Outcome Assessment, Health Care; Postoperative Complications; Proportional Hazards Models; ROC Curve; Shock; Tissue and Organ Procurement
PubMed: 34655663
DOI: 10.1016/j.jhep.2021.10.004 -
Therapeutic Innovation & Regulatory... Jul 2022Little to no data exist quantifying and benchmarking the magnitude of protocol deviation experience.
BACKGROUND
Little to no data exist quantifying and benchmarking the magnitude of protocol deviation experience.
METHODS
Nearly two-dozen companies provided the Tufts Center for the Study of Drug Development (Tufts CSDD) with data on the design and the performance of 187 protocols.
RESULTS
The results of this working group study show that phase II and III protocols have a mean total of 75 and 119 protocol deviations, respectively, involving nearly one-third of all patients enrolled in each clinical trial. Oncology clinical trials have the highest relative mean number of protocol deviations affecting more than 40% of patients enrolled in each trial. The number of endpoints, the number of procedures per visit, and the number of countries were modestly positively associated with and predictive of, the incidence of deviations per protocol. A strong positive relationship was shown between the number of investigative sites and the number of protocol deviations.
CONCLUSION
The results of this initial study provide useful measures that sponsor companies can use to benchmark their own protocol deviation experience, identify factors most associated with protocol deviations, and determine whether remediation is warranted.
Topics: Benchmarking; Humans
PubMed: 35378712
DOI: 10.1007/s43441-022-00401-4 -
Genome Biology Dec 2019Insufficient performance of optimization-based approaches for the fitting of mathematical models is still a major bottleneck in systems biology. In this article, the... (Review)
Review
Insufficient performance of optimization-based approaches for the fitting of mathematical models is still a major bottleneck in systems biology. In this article, the reasons and methodological challenges are summarized as well as their impact in benchmark studies. Important aspects for achieving an increased level of evidence for benchmark results are discussed. Based on general guidelines for benchmarking in computational biology, a collection of tailored guidelines is presented for performing informative and unbiased benchmarking of optimization-based fitting approaches. Comprehensive benchmark studies based on these recommendations are urgently required for the establishment of a robust and reliable methodology for the systems biology community.
Topics: Benchmarking; Models, Biological; Systems Biology
PubMed: 31842943
DOI: 10.1186/s13059-019-1887-9