-
Nature Communications Jan 2023A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA)...
A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA) data processing. However, how the combination of a DIA software tool and a spectral library impacts the outcome of DIA proteomics and phosphoproteomics data analysis has been rarely investigated using benchmark data that mimics biological complexity. In this study, we create DIA benchmark data sets simulating the regulation of thousands of proteins in a complex background, which are collected on both an Orbitrap and a timsTOF instruments. We evaluate four commonly used software suites (DIA-NN, Spectronaut, MaxDIA and Skyline) combined with seven different spectral libraries in global proteome analysis. Moreover, we assess their performances in analyzing phosphopeptide standards and TNF-α-induced phosphoproteome regulation. Our study provides a practical guidance on how to construct a robust data analysis pipeline for different proteomics studies implementing the DIA technique.
Topics: Proteomics; Benchmarking; Workflow; Mass Spectrometry; Software; Proteome
PubMed: 36609502
DOI: 10.1038/s41467-022-35740-1 -
Nature Biotechnology Dec 2022Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of...
Monoclonal antibodies as a group continue to lead biopharmaceuticals in numbers of approvals and sales, although COVID-19 vaccines shot to the top of the list of highest-grossing individual products.
Topics: Biological Products; Benchmarking; Drug Industry; Drug Approval
PubMed: 36471135
DOI: 10.1038/s41587-022-01582-x -
BMC Health Services Research Apr 2017Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on... (Review)
Review
BACKGROUND
Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics.
METHODS
We searched PubMed and EMBASE for articles published in English in the last 10 years. Eligible articles described a project stating benchmarking as its objective and involving a specialty hospital or specific patient category; or those dealing with the methodology or evaluation of benchmarking.
RESULTS
Of 1,817 articles identified in total, 24 were included in the study. Articles were categorized into: pathway benchmarking, institutional benchmarking, articles on benchmark methodology or -evaluation and benchmarking using a patient registry. There was a large degree of variability:(1) study designs were mostly descriptive and retrospective; (2) not all studies generated and showed data in sufficient detail; and (3) there was variety in whether a benchmarking model was just described or if quality improvement as a consequence of the benchmark was reported upon. Most of the studies that described a benchmark model described the use of benchmarking partners from the same industry category, sometimes from all over the world.
CONCLUSIONS
Benchmarking seems to be more developed in eye hospitals, emergency departments and oncology specialty hospitals. Some studies showed promising improvement effects. However, the majority of the articles lacked a structured design, and did not report on benchmark outcomes. In order to evaluate the effectiveness of benchmarking to improve quality in specialty hospitals, robust and structured designs are needed including a follow up to check whether the benchmark study has led to improvements.
Topics: Benchmarking; Emergency Service, Hospital; Hospitals, Special; Humans; Models, Theoretical; Quality Improvement; Retrospective Studies
PubMed: 28372574
DOI: 10.1186/s12913-017-2154-y -
International Journal of Surgery... Mar 2023Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and... (Review)
Review
INTRODUCTION
Benchmarking, a novel measuring tool for outcome comparisons, is a recent concept in surgery. The objectives of this review are to examine the concept, definition, and evolution of benchmarking and its application in surgery.
METHODS
The literature about benchmarking was reviewed through an ever-narrowing search strategy, commencing from the concept, definition, and evolution of benchmarking to the application of benchmarking and experiences of benchmarking in surgery. PubMed, Web of Science, Embase, and Science Direct databases were searched until 20 September 2022, in the English language according to the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines.
RESULTS
In the first phase of the literature search, the development of benchmarking was identified. The definitions of benchmarking evolved from a surveying term to a novel quality-improvement tool to assess the best achievable results in surgery. In the second phase, a total of 23 studies were identified about benchmarking in surgery, including esophagectomy, hepatic surgery, pancreatic surgery, rectum resection, and bariatric surgery. All studies were multicenter analyses from national, international, or global expert centers. Most studies (87.0%) adopted the definition that benchmark was the 75th percentile of the median values of centers. Performance metrics to define benchmarks were clinically relevant intraoperative and postoperative outcome indicators.
CONCLUSION
Benchmarking in surgery is a novel quality-improvement tool to define and measure the best achievable results, establishing a meaningful reference to evaluate surgical performance.
Topics: Humans; Benchmarking; Postoperative Complications; Esophagectomy; Bariatric Surgery; Multicenter Studies as Topic
PubMed: 37093075
DOI: 10.1097/JS9.0000000000000212 -
Annual Review of Biophysics May 2017Binding free energy calculations based on molecular simulations provide predicted affinities for biomolecular complexes. These calculations begin with a detailed... (Review)
Review
Binding free energy calculations based on molecular simulations provide predicted affinities for biomolecular complexes. These calculations begin with a detailed description of a system, including its chemical composition and the interactions among its components. Simulations of the system are then used to compute thermodynamic information, such as binding affinities. Because of their promise for guiding molecular design, these calculations have recently begun to see widespread applications in early-stage drug discovery. However, many hurdles remain in making them a robust and reliable tool. In this review, we highlight key challenges of these calculations, discuss some examples of these challenges, and call for the designation of standard community benchmark test systems that will help the research community generate and evaluate progress. In our view, progress will require careful assessment and evaluation of new methods, force fields, and modeling innovations on well-characterized benchmark systems, and we lay out our vision for how this can be achieved.
Topics: Benchmarking; Bridged-Ring Compounds; Computer Simulation; Drug Discovery; Imidazoles; Ligands; Models, Molecular; Muramidase; Protein Binding; Proteins; Software; Thermodynamics
PubMed: 28399632
DOI: 10.1146/annurev-biophys-070816-033654 -
Sensors (Basel, Switzerland) Aug 2021Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives...
Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives which can be exploited to improve the robustness of anomaly detection. In this work, we build upon the deep support vector data description algorithm and address multi-perspective anomaly detection using three different fusion techniques, i.e., early fusion, late fusion, and late fusion with multiple decoders. We employ different augmentation techniques with a denoising process to deal with scarce one-class data, which further improves the performance (ROC AUC =80%). Furthermore, we introduce the dices dataset, which consists of over 2000 grayscale images of falling dices from multiple perspectives, with 5% of the images containing rare anomalies (e.g., drill holes, sawing, or scratches). We evaluate our approach on the new dices dataset using images from two different perspectives and also benchmark on the standard MNIST dataset. Extensive experiments demonstrate that our proposed multi-perspective approach exceeds the state-of-the-art single-perspective anomaly detection on both the MNIST and dices datasets. To the best of our knowledge, this is the first work that focuses on addressing multi-perspective anomaly detection in images by jointly using different perspectives together with one single objective function for anomaly detection.
Topics: Algorithms; Benchmarking
PubMed: 34450753
DOI: 10.3390/s21165311 -
Bioinformatics (Oxford, England) Nov 2023Biomedical entity linking (BEL) is the task of grounding entity mentions to a knowledge base (KB). It plays a vital role in information extraction pipelines for the life... (Review)
Review
MOTIVATION
Biomedical entity linking (BEL) is the task of grounding entity mentions to a knowledge base (KB). It plays a vital role in information extraction pipelines for the life sciences literature. We review recent work in the field and find that, as the task is absent from existing benchmarks for biomedical text mining, different studies adopt different experimental setups making comparisons based on published numbers problematic. Furthermore, neural systems are tested primarily on instances linked to the broad coverage KB UMLS, leaving their performance to more specialized ones, e.g. genes or variants, understudied.
RESULTS
We therefore developed BELB, a biomedical entity linking benchmark, providing access in a unified format to 11 corpora linked to 7 KBs and spanning six entity types: gene, disease, chemical, species, cell line, and variant. BELB greatly reduces preprocessing overhead in testing BEL systems on multiple corpora offering a standardized testbed for reproducible experiments. Using BELB, we perform an extensive evaluation of six rule-based entity-specific systems and three recent neural approaches leveraging pre-trained language models. Our results reveal a mixed picture showing that neural approaches fail to perform consistently across entity types, highlighting the need of further studies towards entity-agnostic models.
AVAILABILITY AND IMPLEMENTATION
The source code of BELB is available at: https://github.com/sg-wbi/belb. The code to reproduce our experiments can be found at: https://github.com/sg-wbi/belb-exp.
Topics: Benchmarking; Data Mining; Software; Language; Natural Language Processing
PubMed: 37975879
DOI: 10.1093/bioinformatics/btad698 -
Sensors (Basel, Switzerland) Dec 2021Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such... (Review)
Review
Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such supervised approaches is that the training and testing data are drawn from the same underlying data distribution. This may not always be the case, and in such cases, the performance of the model is degraded. Domain adaptation aims to overcome the domain shift between the source domain used for training and the target domain data used for testing. Unsupervised domain adaptation deals with situations where the network is trained on labeled data from the source domain and unlabeled data from the target domain with the goal of performing well on the target domain data at the time of deployment. In this study, we overview seven state-of-the-art unsupervised domain adaptation models based on deep learning and benchmark their performance on three new domain adaptation datasets created from publicly available aerial datasets. We believe this is the first study on benchmarking domain adaptation methods for aerial data. In addition to reporting classification performance for the different domain adaptation models, we present t-SNE visualizations that illustrate the benefits of the adaptation process.
Topics: Adaptation, Physiological; Benchmarking
PubMed: 34884072
DOI: 10.3390/s21238070 -
BMC Bioinformatics Dec 2023Biclustering is increasingly used in biomedical data analysis, recommendation tasks, and text mining domains, with hundreds of biclustering algorithms proposed. When...
BACKGROUND
Biclustering is increasingly used in biomedical data analysis, recommendation tasks, and text mining domains, with hundreds of biclustering algorithms proposed. When assessing the performance of these algorithms, more than real datasets are required as they do not offer a solid ground truth. Synthetic data surpass this limitation by producing reference solutions to be compared with the found patterns. However, generating synthetic datasets is challenging since the generated data must ensure reproducibility, pattern representativity, and real data resemblance.
RESULTS
We propose G-Bic, a dataset generator conceived to produce synthetic benchmarks for the normative assessment of biclustering algorithms. Beyond expanding on aspects of pattern coherence, data quality, and positioning properties, it further handles specificities related to mixed-type datasets and time-series data.G-Bic has the flexibility to replicate real data regularities from diverse domains. We provide the default configurations to generate reproducible benchmarks to evaluate and compare diverse aspects of biclustering algorithms. Additionally, we discuss empirical strategies to simulate the properties of real data.
CONCLUSION
G-Bic is a parametrizable generator for biclustering analysis, offering a solid means to assess biclustering solutions according to internal and external metrics robustly.
Topics: Gene Expression Profiling; Reproducibility of Results; Benchmarking; Cluster Analysis; Algorithms
PubMed: 38053078
DOI: 10.1186/s12859-023-05587-4 -
Scientific Reports Aug 2021The low biomass of respiratory samples makes it difficult to accurately characterise the microbial community composition. PCR conditions and contaminating microbial DNA...
The low biomass of respiratory samples makes it difficult to accurately characterise the microbial community composition. PCR conditions and contaminating microbial DNA can alter the biological profile. The objective of this study was to benchmark the currently available laboratory protocols to accurately analyse the microbial community of low biomass samples. To study the effect of PCR conditions on the microbial community profile, we amplified the 16S rRNA gene of respiratory samples using various bacterial loads and different number of PCR cycles. Libraries were purified by gel electrophoresis or AMPure XP and sequenced by V2 or V3 MiSeq reagent kits by Illumina sequencing. The positive control was diluted in different solvents. PCR conditions had no significant influence on the microbial community profile of low biomass samples. Purification methods and MiSeq reagent kits provided nearly similar microbiota profiles (paired Bray-Curtis dissimilarity median: 0.03 and 0.05, respectively). While profiles of positive controls were significantly influenced by the type of dilution solvent, the theoretical profile of the Zymo mock was most accurately analysed when the Zymo mock was diluted in elution buffer (difference compared to the theoretical Zymo mock: 21.6% for elution buffer, 29.2% for Milli-Q, and 79.6% for DNA/RNA shield). Microbiota profiles of DNA blanks formed a distinct cluster compared to low biomass samples, demonstrating that low biomass samples can accurately be distinguished from DNA blanks. In summary, to accurately characterise the microbial community composition we recommend 1. amplification of the obtained microbial DNA with 30 PCR cycles, 2. purifying amplicon pools by two consecutive AMPure XP steps and 3. sequence the pooled amplicons by V3 MiSeq reagent kit. The benchmarked standardized laboratory workflow presented here ensures comparability of results within and between low biomass microbiome studies.
Topics: Benchmarking; Biomass; Humans; Metagenomics; Microbiota; Polymerase Chain Reaction; RNA, Ribosomal, 16S; Reagent Kits, Diagnostic; Respiratory Mucosa; Saliva
PubMed: 34433845
DOI: 10.1038/s41598-021-96556-5