-
Briefings in Bioinformatics Jul 2018CRISPR (Clustered Regularly Interspaced Short Palindromic Repeats)-based gene editing has been widely implemented in various cell types and organisms. A major challenge...
CRISPR (Clustered Regularly Interspaced Short Palindromic Repeats)-based gene editing has been widely implemented in various cell types and organisms. A major challenge in the effective application of the CRISPR system is the need to design highly efficient single-guide RNA (sgRNA) with minimal off-target cleavage. Several tools are available for sgRNA design, while limited tools were compared. In our opinion, benchmarking the performance of the available tools and indicating their applicable scenarios are important issues. Moreover, whether the reported sgRNA design rules are reproducible across different sgRNA libraries, cell types and organisms remains unclear. In our study, a systematic and unbiased benchmark of the sgRNA predicting efficacy was performed on nine representative on-target design tools, based on six benchmark data sets covering five different cell types. The benchmark study presented here provides novel quantitative insights into the available CRISPR tools.
Topics: Benchmarking; CRISPR-Cas Systems; Computer Simulation; Gene Editing; Humans; RNA, Guide, CRISPR-Cas Systems
PubMed: 28203699
DOI: 10.1093/bib/bbx001 -
Journal of Nursing Scholarship : An... 2001To examine the usefulness of three types of benchmarking for interpreting patient outcome data.
PURPOSE
To examine the usefulness of three types of benchmarking for interpreting patient outcome data.
DESIGN
This study was part of a multiyear, multihospital longitudinal survey of 10 patient outcomes. The patient outcome used for this methodologic presentation was central line infections (CLI). The sample included eight hospitals in an integrated healthcare system, with a range in size from 144 to 861 beds. The unit of analysis for CLI was the number of line days, with the CLI rate defined as the number of infections per 1,000 patient-line days per month.
METHODS
Data on each outcome were collected at the unit level according to standardized protocols. Results were submitted via standardized electronic forms to a central data management center. Data for this presentation were analyzed using a Bayesian hierarchical Poisson model. Results are presented for each hospital and the system as a whole.
FINDINGS
In comparison to published benchmarks, hospital performances were mixed with regard to CLI. Five of the 8 hospitals exceeded 2.2 infections per 1,000 patient-line days. When benchmarks were established for each hospital using 95% credible intervals, hospitals did reasonably well with only isolated months reaching or going beyond the benchmark limits. When the entire system was used to establish benchmarks with the 95% credible intervals, the hospitals that reached or exceeded the benchmark limits remained the same, but some hospitals had CLI rates more frequently in the upper 50% of the benchmarking limits.
CONCLUSIONS
Benchmarking of quality indicators can be accomplished in a variety of ways as a means to quantify patient care and identify areas needing attention and improvement. Hospital-specific and system-wide benchmarks provide relevant feedback for improving performance at individual hospitals.
Topics: Benchmarking; Catheterization, Central Venous; Cross Infection; Health Services Research; Humans; Longitudinal Studies; Outcome Assessment, Health Care; Quality Indicators, Health Care; Quality of Health Care; Systems Analysis; Total Quality Management; United States
PubMed: 11419316
DOI: 10.1111/j.1547-5069.2001.00185.x -
Microbial Genomics Oct 2022Culture-independent metagenomic detection of microbial species has the potential to provide rapid and precise real-time diagnostic results. However, it is potentially...
Culture-independent metagenomic detection of microbial species has the potential to provide rapid and precise real-time diagnostic results. However, it is potentially limited by sequencing and taxonomic classification errors. We use simulated and real-world data to benchmark rates of species misclassification using 100 reference genomes for each of the ten common bloodstream pathogens and six frequent blood-culture contaminants (=1568, only 68 genomes were available for ). Simulating both with and without sequencing error for both the Illumina and Oxford Nanopore platforms, we evaluated commonly used classification tools including Kraken2, Bracken and Centrifuge, utilizing mini (8 GB) and standard (30-50 GB) databases. Bracken with the standard database performed best, the median percentage of reads across both sequencing platforms identified correctly to the species level was 97.8% (IQR 92.7:99.0) [range 5:100]. For Kraken2 with a mini database, a commonly used combination, median species-level identification was 86.4% (IQR 50.5:93.7) [range 4.3:100]. Classification performance varied by species, with being more challenging to classify correctly (probability of reads being assigned to the correct species: 56.1-96.0%, varying by tool used). Human read misclassification was negligible. By filtering out shorter Nanopore reads we found performance similar or superior to Illumina sequencing, despite higher sequencing error rates. Misclassification was more common when the misclassified species had a higher average nucleotide identity to the true species. Our findings highlight taxonomic misclassification of sequencing data occurs and varies by sequencing and analysis workflow. To account for 'bioinformatic contamination' we present a contamination catalogue that can be used in metagenomic pipelines to ensure accurate results that can support clinical decision making.
Topics: Humans; Nanopores; Benchmarking; Metagenomics; High-Throughput Nucleotide Sequencing; Nucleotides
PubMed: 36269282
DOI: 10.1099/mgen.0.000886 -
Bioinformatics (Oxford, England) Jan 2020Secondary structure prediction accuracy (SSPA) in the QuanTest benchmark can be used to measure accuracy of a multiple sequence alignment. SSPA correlates well with the...
MOTIVATION
Secondary structure prediction accuracy (SSPA) in the QuanTest benchmark can be used to measure accuracy of a multiple sequence alignment. SSPA correlates well with the sum-of-pairs score, if the results are averaged over many alignments but not on an alignment-by-alignment basis. This is due to a sub-optimal selection of reference and non-reference sequences in QuanTest.
RESULTS
We develop an improved strategy for selecting reference and non-reference sequences for a new benchmark, QuanTest2. In QuanTest2, SSPA and SP correlate better on an alignment-by-alignment basis than in QuanTest. Guide-trees for QuanTest2 are more balanced with respect to reference sequences than in QuanTest. QuanTest2 scores correlate well with other well-established benchmarks.
AVAILABILITY AND IMPLEMENTATION
QuanTest2 is available at http://bioinf.ucd.ie/quantest2.tar, comprises of reference and non-reference sequence sets and a scoring script.
SUPPLEMENTARY INFORMATION
Supplementary data are available at Bioinformatics online.
Topics: Algorithms; Benchmarking; Protein Structure, Secondary; Sequence Alignment; Software
PubMed: 31292629
DOI: 10.1093/bioinformatics/btz552 -
PloS One 2020Benchmarking RNA-seq differential expression analysis methods using spike-in and simulated RNA-seq data has often yielded inconsistent results. The spike-in data, which... (Comparative Study)
Comparative Study
Benchmarking RNA-seq differential expression analysis methods using spike-in and simulated RNA-seq data has often yielded inconsistent results. The spike-in data, which were generated from the same bulk RNA sample, only represent technical variability, making the test results less reliable. We compared the performance of 12 differential expression analysis methods for RNA-seq data, including recent variants in widely used software packages, using both RNA spike-in and simulation data for negative binomial (NB) model. Performance of edgeR, DESeq2, and ROTS was particularly different between the two benchmark tests. Then, each method was tested under most extensive simulation conditions especially demonstrating the large impacts of proportion, dispersion, and balance of differentially expressed (DE) genes. DESeq2, a robust version of edgeR (edgeR.rb), voom with TMM normalization (voom.tmm) and sample weights (voom.sw) showed an overall good performance regardless of presence of outliers and proportion of DE genes. The performance of RNA-seq DE gene analysis methods substantially depended on the benchmark used. Based on the simulation results, suitable methods were suggested under various test conditions.
Topics: Benchmarking; Computer Simulation; Gene Expression Profiling; Humans; RNA; RNA-Seq; Sequence Analysis, RNA; Software
PubMed: 32353015
DOI: 10.1371/journal.pone.0232271 -
The Plant Genome Jun 2023Structural variations (SVs) are larger polymorphisms (> 50 bp in length), which consist of insertions, deletions, inversions, duplications, and translocations. They...
Structural variations (SVs) are larger polymorphisms (> 50 bp in length), which consist of insertions, deletions, inversions, duplications, and translocations. They can have a strong impact on agronomical traits and play an important role in environmental adaptation. The development of long-read sequencing technologies, including Oxford Nanopore, allows for comprehensive SV discovery and characterization even in complex polyploid crop genomes. However, many of the SV discovery pipeline benchmarks do not include complex plant genome datasets. In this study, we benchmarked insertion and deletion detection by popular long-read alignment-based SV detection tools for crop plant genomes. We used real and simulated Oxford Nanopore reads for two crops, allotetraploid Brassica napus (oilseed rape) and diploid Solanum lycopersicum (tomato), and evaluated several read aligners and SV callers across 5×, 10×, and 20× coverages typically used in re-sequencing studies. We further validated our findings using maize and soybean datasets. Our benchmarks provide a useful guide for designing Oxford Nanopore re-sequencing projects and SV discovery pipelines for crop plants.
Topics: Sequence Analysis, DNA; Benchmarking; Nanopores; High-Throughput Nucleotide Sequencing; Genome, Plant
PubMed: 36988043
DOI: 10.1002/tpg2.20314 -
Physical Therapy Dec 2021Academic physical therapy has no universal metrics by which educational programs can measure outcomes, limiting their ability to benchmark to their own historical...
OBJECTIVE
Academic physical therapy has no universal metrics by which educational programs can measure outcomes, limiting their ability to benchmark to their own historical performance, to peer institutions, or to other health care professions. The PT-Graduation Questionnaire (GQ) survey, adapted from the Association of American Medical Colleges' GQ, addresses this gap by offering both inter-professional insight and fine-scale assessment of physical therapist education. This study reports the first wave of findings from an ongoing multi-site trial of the PT-GQ among diverse academic physical therapy programs, including (1) benchmarks for academic physical therapy, and (2) a comparison of the physical therapist student experience to medical education benchmarks.
METHODS
Thirty-four doctor of physical therapy (DPT) programs (13.2% nationwide sample) administered the online survey to DPT graduates during the 2019 to 2020 academic year. PT-GQ and Association of American Medical Colleges data were contrasted via Welch's unequal-variance t test and Hedges g (effect size).
RESULTS
A total of 1025 respondents participated in the study (response rate: 63.9%). The average survey duration was 31.8 minutes. Overall educational satisfaction was comparable with medicine, and respondents identified areas of curricular strength (eg, anatomy) and weakness (eg, pharmacology). DPT respondents provided higher ratings of faculty professionalism than medicine, lower rates of student mistreatment, and a lesser impact of within-program diversity on their training. One-third of respondents were less than "satisfied" with student mental health services. DPT respondents reported significantly higher exhaustion but lower disengagement than medical students, along with lower tolerance for ambiguity. Of DPT respondents who reported educational debt, one-third reported debt exceeding $150,000, the threshold above which the DPT degree loses economic power.
CONCLUSIONS
These academic benchmarks, using the PT-GQ, provided insight into physical therapist education and identified differences between physical therapist and medical student perceptions.
IMPACT
This ongoing trial will establish a comprehensive set of benchmarks to better understand academic physical therapy outcomes.
Topics: Benchmarking; Humans; Physical Therapy Specialty; Program Evaluation; Surveys and Questionnaires
PubMed: 34723335
DOI: 10.1093/ptj/pzab229 -
Journal of Chemical Information and... Mar 2019De novo design seeks to generate molecules with required property profiles by virtual design-make-test cycles. With the emergence of deep learning and neural generative...
De novo design seeks to generate molecules with required property profiles by virtual design-make-test cycles. With the emergence of deep learning and neural generative models in many application areas, models for molecular design based on neural networks appeared recently and show promising results. However, the new models have not been profiled on consistent tasks, and comparative studies to well-established algorithms have only seldom been performed. To standardize the assessment of both classical and neural models for de novo molecular design, we propose an evaluation framework, GuacaMol, based on a suite of standardized benchmarks. The benchmark tasks encompass measuring the fidelity of the models to reproduce the property distribution of the training sets, the ability to generate novel molecules, the exploration and exploitation of chemical space, and a variety of single and multiobjective optimization tasks. The benchmarking open-source Python code and a leaderboard can be found on https://benevolent.ai/guacamol .
Topics: Benchmarking; Deep Learning; Drug Design; Isomerism; Models, Molecular; Molecular Structure; Monte Carlo Method; Pharmaceutical Preparations; Quantitative Structure-Activity Relationship
PubMed: 30887799
DOI: 10.1021/acs.jcim.8b00839 -
Journal of Cellular Biochemistry Jun 2015Genomic regions represent features such as gene annotations, transcription factor binding sites and epigenetic modifications. Performing various genomic operations such...
Genomic regions represent features such as gene annotations, transcription factor binding sites and epigenetic modifications. Performing various genomic operations such as identifying overlapping/non-overlapping regions or nearest gene annotations are common research needs. The data can be saved in a database system for easy management, however, there is no comprehensive database built-in algorithm at present to identify overlapping regions. Therefore I have developed a novel region-mapping (RegMap) SQL-based algorithm to perform genomic operations and have benchmarked the performance of different databases. Benchmarking identified that PostgreSQL extracts overlapping regions much faster than MySQL. Insertion and data uploads in PostgreSQL were also better, although general searching capability of both databases was almost equivalent. In addition, using the algorithm pair-wise, overlaps of >1000 datasets of transcription factor binding sites and histone marks, collected from previous publications, were reported and it was found that HNF4G significantly co-locates with cohesin subunit STAG1 (SA1).Inc.
Topics: Algorithms; Benchmarking; Genome; Genomics
PubMed: 25560631
DOI: 10.1002/jcb.25049 -
BMC Bioinformatics Apr 2023Elucidating compound mechanism of action (MoA) is beneficial to drug discovery, but in practice often represents a significant challenge. Causal Reasoning approaches aim...
BACKGROUND
Elucidating compound mechanism of action (MoA) is beneficial to drug discovery, but in practice often represents a significant challenge. Causal Reasoning approaches aim to address this situation by inferring dysregulated signalling proteins using transcriptomics data and biological networks; however, a comprehensive benchmarking of such approaches has not yet been reported. Here we benchmarked four causal reasoning algorithms (SigNet, CausalR, CausalR ScanR and CARNIVAL) with four networks (the smaller Omnipath network vs. 3 larger MetaBase™ networks), using LINCS L1000 and CMap microarray data, and assessed to what extent each factor dictated the successful recovery of direct targets and compound-associated signalling pathways in a benchmark dataset comprising 269 compounds. We additionally examined impact on performance in terms of the functions and roles of protein targets and their connectivity bias in the prior knowledge networks.
RESULTS
According to statistical analysis (negative binomial model), the combination of algorithm and network most significantly dictated the performance of causal reasoning algorithms, with the SigNet recovering the greatest number of direct targets. With respect to the recovery of signalling pathways, CARNIVAL with the Omnipath network was able to recover the most informative pathways containing compound targets, based on the Reactome pathway hierarchy. Additionally, CARNIVAL, SigNet and CausalR ScanR all outperformed baseline gene expression pathway enrichment results. We found no significant difference in performance between L1000 data or microarray data, even when limited to just 978 'landmark' genes. Notably, all causal reasoning algorithms also outperformed pathway recovery based on input DEGs, despite these often being used for pathway enrichment. Causal reasoning methods performance was somewhat correlated with connectivity and biological role of the targets.
CONCLUSIONS
Overall, we conclude that causal reasoning performs well at recovering signalling proteins related to compound MoA upstream from gene expression changes by leveraging prior knowledge networks, and that the choice of network and algorithm has a profound impact on the performance of causal reasoning algorithms. Based on the analyses presented here this is true for both microarray-based gene expression data as well as those based on the L1000 platform.
Topics: Benchmarking; Gene Expression Profiling; Algorithms; Microarray Analysis; Gene Expression; Gene Regulatory Networks
PubMed: 37072707
DOI: 10.1186/s12859-023-05277-1