SciDataReportR 21.14.0
Plot and formatted-table functions that show p-value stars now support
IncludeTrendSymbol = TRUE, which adds†for nominal trends (0.05 < p <= 0.10) without changing significance thresholds or defaults.Stability refits and common-method benchmarks now resolve automatic workers from a conservative memory estimate (the smaller of 50% physical RAM and 75% currently available RAM), retain a worker/memory audit in settings, and run pairwise-memory methods serially by default. Benchmark screens no longer retain every full-data candidate model unless
keep_models = TRUE; selected confirmation models remain projectable. The default HDBSCAN benchmark set is reduced to three representative settings.SOM + Mclust automatic maps now supplement the k-informed 20-nodes-per- planned-profile target with a mixture-support floor. New
min_nodes_per_parameter = 3uses Mclust’s free-parameter count after SOM preprocessing to size automatic maps, while explicit dimensions are never changed. Fit and map-screen tables report mixture nodes, parameters, nodes-per-parameter, and overparameterized candidates, which remain visible but are ineligible for a composite recommendation.Automatic SOM + Mclust sizing now also applies the map-health ceiling
min_participants_per_node = 10. NewSOMMapSupport()reports the competing parameter-support and occupancy constraints before fitting. Candidate tables and fit-review points identify supported, under-supported, and over-parameterized mixtures. Finalized fits can usemap_fromto reuse an exploratory map’s design while recalculating preprocessing on their finalization data.SOM + Mclust stability now supports an opt-in participant-deletion curve via
stability_fractions(default0.90). It retains all-participant ARI/P05 as the primary final-partition result while adding VI, NMI, and Fowlkes–Mallows companions; exact memory-safe co-membership cohesion and cross-mixing summaries; split/merge diagnostics; and frozen-scale matched phenotype-profile correlation/RMSE recovery. New stability-panel plots and documentation make these explanatory diagnostics available without excluding boundary participants or substituting them for ARI.CreateClusterModel_SOM_MClust()now uses one deterministic, seeded full-cohort batch SOM fit. Experimental consensus-SOM and median-only fitting controls have been removed. New-model provenance records the initial codebook, batch mode, epochs, grid, topology, neighbourhood, preprocessing, and node-level mixture; existing consensus-era objects remain projectable through their frozen fitted components.SOM + Mclust automatic map resolution is now k-informed rather than driven solely by training-cohort size. Exploratory fits use
max(k_range)and finalized fits usefinal_k, targeting approximately 20 nodes per maximum planned phenotype unless dimensions are supplied explicitly. The map screen now evaluates 10/20/30-nodes-per-maximum-profile candidates plus 3x3 and 10x10 anchors, reporting expected nodes per requested profile. Grid design, boundary prevalence, and all-/non-boundary stability diagnostics are retained in review output so map selection remains a reviewer decision.SOM + Mclust adds
assignment = "hard"or"soft"(soft-3 default). All models retain hard BMU-node labels, proximity-weighted soft class probabilities, entropy, margin, and boundary diagnostics in training and projection output. Stability retains all-participant ARI as primary and now reports reference-defined non-boundary ARI/Jaccard as secondary sensitivity diagnostics without excluding or relabeling any participant.CreateClusterModel_SOM_MClust()gainslpa_bootstrap_n, defaulting to0. tidyLPA’s mclust backend previously ran a bootstrap likelihood-ratio test andMclustBootstrap(100 resamples each) for every candidate fit, which dominated fitting time (about 10 times slower per fit). BLRT columns areNAand left out of the fit plot unlesslpa_bootstrap_nis set; stability refits never run the bootstraps.CreateClusterModel_MClust()(and the PCA + Mclust pipeline) never reported these results, so it now always skips them; its candidate fits are correspondingly faster.New
PlotClusterPrevalence()draws a cluster-by-feature heatmap of the percentage of participants with each binary feature (for example, a marker beyond a clinical cutoff, a diagnosis, or a medication), with the whole cohort alongside for reference.New
PlotClusterSplits()follows participants from k to k + 1 clusters for each model family of an exploratoryCreateClusterModel_SOM_MClust()model (or any table of assignments) as an alluvial plot, with a transition table reporting whether each added cluster splits an existing one (ParentPurity,SplitCluster,SplitRetained) or reshuffles participants (lowerParentPurity).CreateClusterModel_SOM_MClust()fit tables now summarize participants’ posterior probabilities for every candidate (PosteriorMean,PosteriorMedian,PropLowPosteriorbelowlow_prob_threshold), and"Posterior"can be weighted in the composite rank index. Subsample stability adds core metrics computed among participants confidently assigned in the reference fit (StabilityCoreProportion,StabilityARI_Core,StabilityJaccard_CoreMean,StabilityJaccard_CoreMin); these legacy descriptive outputs do not replace all-participant ARI as the primary final-partition reproducibility result.CreateClusterModel_SOM_MClust()uses node-based phenotype definitions. Participant assignment is either the best-matching node’s Mclust class ("hard") or a proximity-weighted combination of nearby node posteriors ("soft"); it never recalibrates a separate participant-level mixture.CreateClusterModel_SOM_MClust()gainsstability_candidates = "best_per_k", which runs subsample stability on the best candidate at each number of profiles so reproducibility can inform the choice of k. Stability refits now fit only the shortlisted candidates.CreateClusterModel_SOM_MClust()stability is much faster: each resample now trains one SOM and fits every candidate on its codebook, instead of retraining the SOM for every candidate. The newstability_top_nlimits stability to the top candidates by the composite index (eligible first), so a large candidate grid can be screened with resampling. Assessed candidates are recorded inStability$settings$candidates_assessed, and every candidate’s node assignment is returned inModelInfo_MClust$node_assignments.RevalueData()no longer fails with “missing value where TRUE/FALSE needed” when the codebook omits the optionalRecode,Code,Type, orLabelcolumns; absent optional columns are treated as blank, as documented.
SciDataReportR 21.12.0
MakeCoxRegressionTable()now screens event-predictor pairs with ordinary or survey-weighted Cox proportional hazards models and returns labelledgttables, tidy hazard ratios, FDR correction, concordance, and proportional hazards diagnostics compatible withPlotForestFromTable().SOM + Mclust
SOM_Distanceis now the Euclidean distance to the best-matching unit, in Z-score units. It was previously kohonen’s squared distance. Distance flags are unchanged because they use quantiles; distance means, SDs, and z-scores change scale. Models fitted with earlier versions still project on their original squared scale.SOM + Mclust training and projection now decompose each participant’s SOM distance into per-variable residuals (
SOMFit$residuals,ProjectionFit$residuals), record the largest contributor (Top_Distance_Variable,Top_Distance_Share), and summarize variable contributions.ProjectionFit$variable_driftand its plot compare each variable’s misfit in the projected cohort with training, showing which measures drive projection drift.ProjectionFit$summaryfor SOM + Mclust addsexcess_high_distance(share beyond the training high-distance cutoff minus the share expected without drift) andcluster_occupancy_js_divergence(phenotype prevalence drift).CreateClusterModel_SOM_MClust()gainslpa_min_node_nto leave sparse or empty SOM nodes out of the mixture fit (they are classified afterwards by the fitted model),min_cluster_propto recommend only candidates whose smallest profile holds a minimum share of participants, andcriteria_weightsto weight the composite rank index directly or through an Analytic Hierarchy Process pairwise-comparison matrix with its consistency ratio. The fit table addsMinProfileParticipantN,MinProfileParticipantProportion, andEligible, and the recommendation is now described as a composite rank index.CreateClusterModel_SOM_MClust()exploratory runs no longer stop with “This tidyProfile has no data attached” after fitting every candidate. A model/kcombination that mclust cannot estimate (tidyLPA returns fit indices without class assignments) is now recorded as failed inModelInfo_MClust$diagnostics$lpa_fit_diagnosticsand left out of the candidate table.CreateClusterModel_SOM_MClust(method = "finalize")withstability_resamples > 0now records the user-specifiedfinal_kandfinal_modelinSpecification$selected, reports a “User-specified” recommendation, and leaves the AHP columnsNAinstead of scoring the single candidate.SOM + Mclust training fits now label poorly represented participants
"Poor fit to training structure", matchingProjectCluster(), so training and projectedProjection_Fit_Classvalues can be compared directly.ProbFit$individualfrom SOM + Mclust training and projection no longer carries the matched node’s codebook vector under theZ_variable names; those values remain inProbFit$node.
SciDataReportR 21.2.0
Clustering stability now retains 90% of participants per full-pipeline subsample refit by default, so it measures whether assignments persist after removing a random 10% of participants. The retained fraction is recorded as
Stability$settings$resample_fraction = 0.90.Mclust candidate plots now use consistent descriptive model legends, such as
Model 6: varying variance, varying covariance, derived from one shared model registry. The clustering documentation now explains each supported variance and covariance structure.
SciDataReportR 21.0.2
PlotVolcanoEffects()can now color both raw-p and FDR volcano plots by an analyte category supplied as aVariable/Categorydata frame or named vector. Unmapped analytes remain visible in grey.PlotVolcanoEffects(LabelMode = "raw")now explicitly labels analytes that pass the raw p-value threshold. The existing"significant"value remains a compatible alias, while"fdr"continues to use adjusted p-values.
SciDataReportR 21.0.1
Mclust, PCA + Mclust, MCA + Mclust, and SOM + Mclust now share tidyLPA’s mclust backend and explore all supported covariance models by default: models 1 (EEI), 2 (VVI), 3 (EEE), and 6 (VVV). Models 4 and 5 remain unavailable because they require OpenMx.
MakeComparisonTable()now displays adjusted Type II ANCOVA p-values when the grouping variable has a non-syntactic name requiring backticks in model formulas. Previously the model fit and effect size succeeded, but the p-value row lookup missed the backtick-quoted term and rendered a blank cell.MakeUnivariateRegressionTable()no longer emits tidyselect deprecation warnings while building its report tables, and biomarker-performance source code now passes the package portability check for non-ASCII characters.
SciDataReportR 20.24.0
Clustering API is standardized around fitted pipelines and one projector
Stability now measures participant-overlap refit reproducibility. Each replicate refits the complete clustering pipeline on an 80% participant subsample and compares those participants’ refitted assignments with their full-data reference assignments by
.row_id. Projection is no longer part of stability. ARI is the primary partition metric, model seeds vary by replicate, and SOM grid dimensions remain fixed to the reference fit.Breaking unreleased clustering-output cleanup. Finalized fits and projections now return
DataWithClusters(formerlydf_with_clusters) andClusterVariableName(formerlyClusterName). The mistypedModelInfo_Mclust, publiccomplete_rows,CandidateAudit, and injected.scidr_rowidcolumns have been removed and replaced by a consistently labeled.row_idin mergeable row-level outputs.ModelInfo_MClustis the canonical Mclust-specific layer;ModelInforemains the generic compatibility alias.SOM + Mclust now reports
MinProfileNodeN/MaxProfileNodeNalongside their explicit proportions, andBLRTStatistic/BLRTPValue. The candidate plot includes an unselected BLRT p-value panel with a neutral 0.05 reference line.Clustering constructors now use
CreateClusterModel_<reduction>_<method>(), for exampleCreateClusterModel_PCA_MClust()andCreateClusterModel_SOM_HDBSCAN(). Fitted objects use matchingPipeline_<reduction>_<method>classes.ProjectCluster(object, new_df)is the single public projection interface. It dispatches from the fitted pipeline class and retains each method’s frozen preprocessing, reduction, support checks, and projection diagnostics.CreateSOMClusterModel(),ProjectSOMCluster(),Pipeline_SOMClust(), andProject_SOMClust()remain for established SOM compatibility. The first two emit lifecycle deprecation warnings and delegate to the canonical API.
The SciData palette is now the default categorical color scheme
Every plot the package produces now draws categorical color from
SciDataPalette(). Previously each function chose its own: some fell back to ggplot2’s default hue palette, others hardcoded a tableau-style vector, anhcl.colors()name, or apaletteerstring. Clusters, groups, cohorts, and category levels now look like one system across the whole package.Only categorical color changed. Continuous and diverging scales (correlation and effect-size heatmaps, the p-value scales) are untouched, as are the meaning-carrying scales: PASS/WARNING/FAIL status, the signed significance ladders, RCI classification, volcano significance tiers, the grey used for missing data, and the grey reserved for the density-based
Noisecluster.Fixes a latent failure.
CreatePCATable()andCreateZScorePlot()shared a 20-color vector that went straight intoscale_color_manual(), so both failed withInsufficient values in manual scaleonce a plot had more than 20 variable categories. Package plots now extend the palette instead of erroring, however many categories a variable has.Labels drawn inside filled shapes now pick black or white by background luminance.
PlotCategoricalDistributions()draws its labels inside the bars, and the palette’s darker colors would have made them unreadable.Palette arguments keep their existing signatures.
PlotClusterBoxplot(Palette),PlotSplitViolin(color_palette),PlotSpiderChart(Palette)andPlot2GroupStats(palette)all still accept exactly what they accepted before; only the default changed.PlotSpiderChart()andPlot2GroupStats()now default toNULL, meaning “use the package palette” - passing"Dark 3"or"pals::alphabet"explicitly works as it always did.SciDataPalette()itself is unchanged, including erroring when asked for more than its 34 colors.
Cluster occupancy figures removed from the clustering pipelines
The
occupancybar chart is no longer attached toModelInfo$plotsbyCreateClusterModel_MClust(),CreateClusterModel_KMeans(),CreateClusterModel_PCA_MClust(),CreateClusterModel_PCA_KMeans(),CreateClusterModel_HDBSCAN(),CreateClusterModel_Gower_PAM(),CreateClusterModel_LatentClass(),CreateClusterModel_MCA_MClust(), orCreateSOMClusterModel(). Cluster sizes are already reported in the fit tables and inProbFit$individual, so the figure added a panel without adding information.PlotClusterOccupancy()is unchanged and still exported, so the plot can still be drawn directly from a model’sDataWithClusters.The training-versus-projected occupancy comparison in the
Project*functions is unaffected: it compares two distributions rather than describing one, which is a projection-quality check rather than a restatement of the cluster sizes.
Symmetric p-value matrices are corrected once per pair
ApplyFDRCorrection()treated a symmetric matrix - the shape produced whenever a variable set is correlated against itself, asPlotCorrelationsHeatmap()does whenoutcome_varsis leftNULL- as a family ofn * (n - 1)tests, when onlyn * (n - 1) / 2were run. Newsymmetricargument, defaulting to"auto", detects the case and corrects each pair once, mirroring the adjusted values back so the matrix stays symmetric. Passsymmetric = FALSEfor the old behavior orsymmetric = TRUEto require it.Benjamini-Hochberg is invariant to exact duplication, so
method = "fdr"results - the default, and what every plotting function uses - are unchanged."bonferroni","holm"and"hochberg"were coming out exactly twice as large as they should have been, and are now correct.The diagonal of a symmetric matrix holds self-comparisons that were never tested. It is now excluded from the family and returned as
NA; setinclude_diagonal = TRUEto keep the old behavior. This was the damaging case: where the diagonal carried real values, a self-correlation p-value of 0 entered the family as the most significant test in it and pulled every off-diagonal result down, making findings look stronger than they were.
Misspelled variables are now an error instead of a silent result change
The matrix and heatmap functions used to handle an unknown variable name five different ways.
PlotCorrelationsHeatmap(),PlotAnovaRelationshipsMatrix(),PlotChiSqCovar(),PlotMiningMatrix(),PlotNumInteractionEffectsMatrix()andPlotCatInteractionEffectsMatrix()dropped the name silently, returning a smaller results matrix with no warning that anything had gone missing;PlotPhiHeatmap()andPlotPointCorrelationsHeatmap()failed with base R’s"undefined columns selected", which never named the offending variable; andPlotDirectionalHeatmaps()andPlotSpiderChart()each raised their own differently worded error. All of them now stop with one message that names both the variable and the argument it was supplied to, for exampleVariables not found indata: NOPE (supplied topredictor_vars).A misspelled covariate was the more serious case.
PlotCorrelationsHeatmap(),PlotChiSqCovar(),PlotAnovaRelationshipsMatrix(),PlotMiningMatrix()andPlotNumInteractionEffectsMatrix()silently ignored a covariate that did not match a column, so the function computed and reported unadjusted statistics while the caller believed they were adjusted. Unknown covariates now error.Validation happens against the data frame as supplied, before ordinal handling renames anything, so variables legitimately removed by
TreatOrdinalAsare unaffected. Auto-detected variable sets (variables = NULL) still work as before, andPlotCorrelationsHeatmap()still reports covariates dropped by ordinal handling throughCovariatesMissing.New internal helpers
ScidrValidateVariables()andScidrValidateVariable()provide the single message format, replacing theunique(intersect(as.character(vars), names(Data)))idiom that caused the silent dropping.
Significance stars agree across the package
Six separate star implementations disagreed on whether the cut points were inclusive: a p-value of exactly 0.001 earned
***inMultivariableRegressionTable()andPlotInteractionEffectsMatrix()but only**inPlotSplitViolin(), while every plot caption stated the strict<form regardless. All of them now route through oneScidrPValueStars()using inclusive upper bounds (p <= 0.05is*), which matchesrstatix::add_significance()— already used byPlotPhiHeatmap(),PlotPointCorrelationsHeatmap()andPlotAnovaRelationshipsMatrix()— and is verified against it at every boundary.Captions and legend labels now state
<=to match the thresholds actually applied. This affectsgeom_starcaption(),MakePairwiseHeatmap()captions,PlotMiningMatrix()legend labels, and thePlot2GroupStats()shape legend (whosecut(right = TRUE)breaks were always inclusive despite reading<).
Other fixes
PlotDirectionalHeatmaps()had the deprecatedyVarsin formal position 3, soPlotDirectionalHeatmaps(df, vars, TRUE)boundTRUEtoyVarsrather thanRelabel. All deprecated formals are now trailing, as elsewhere in the package.PlotDirectionalHeatmaps()’sOrdinalargument was documented as “reserved for future use” and as not affecting the computed tiles, but it is passed toPlotPointCorrelationsHeatmap()and does change how ordinal variables are treated in the binary~continuous block. The documentation now describes what it actually does.
SciDataReportR 20.23.0
Clustering pipelines no longer return a flat
plotslist. Figures are stored beside the object they describe, following the layoutCreateSOMClusterModel()already used:fit_plotreviews candidates,ModelInfo$plotsdescribes the selected solution,ModelInfo$FitDiagnostics$plotsdescribes how individual training cases sit inside it,ProbFit$plotsdescribes membership confidence,Stability$plotsdescribes bootstrap reproducibility, andProjectionFit$plotsdescribes projected cases against the frozen training reference. The previousplotslist mixed aliases, duplicated entries under two names, and stored interactive widgets alongsideggplotobjects; it has been removed.Every method now carries figures appropriate to that method rather than a shared lowest common denominator. K-means and Gower/PAM gain per-participant silhouette profiles, elbow and average-silhouette curves; Mclust gains BIC, ICL, entropy, and classification-uncertainty maps; HDBSCAN gains density-grid review and per-cluster persistence; latent class analysis gains item response profiles; and the PCA/MCA pipelines lead with scree and loadings. All methods gain a frozen two-dimensional review map, cluster centre profiles, and a distance histogram/boxplot/ECDF triad against a frozen high-distance cutoff.
Project*Cluster()results gainProjectionFit, which triages every projected case against the frozen training reference intoGood fit,Uncertain membership,Poor fit to training structure, orPotential novel phenotype, and addsProjection_Fit_ClasstoDataWithClusters. This extends the triageProjectSOMCluster()already performed to all clustering methods.CreateSOMClusterModel()andProjectSOMCluster()keep their existing figures unchanged. The SOM model gainsModelInfo_SOM$plots$occupancyandStability$plots; the projection exposes its diagnostics asProjectionFit(still also available asSOMProj) and gainssom_grid_map.CreateSOMClusterModel(method = "finalize", stability_resamples > 0)now refits through the exploratory path by a direct recursive call rather than evaluating a reconstructed call in the caller’s frame, which failed when arguments were supplied as local variables.Bootstrap stability resamples now preserve observed factor levels, and the MCA pipeline freezes its reduction on observed categories only. Previously every MCA stability replicate could fail when a resample dropped a rare category, and a declared-but-unobserved factor level made projection fail outright.
ProjectCluster()now warns and treats categories absent from the training model as incomplete cases instead of erroring.New exported figure helpers:
PlotClusterOccupancy(),PlotClusterMap(),PlotClusterSilhouette(),PlotClusterCentreHeatmap(),PlotClusterCentreProfile(),PlotClusterComposition(), andPlotClusterDiagnostic().PlotProjectionDiagnostics()has been renamed toPlotClusterDiagnostic().
SciDataReportR 20.15.0
-
MakeComparisonTable()now reports absolute Cohen’s d for two-group parametric comparisons, including covariate-adjusted d from estimated marginal means, and Cohen’s f for multi-group omnibus comparisons. Effect-size captions identify the scales present, provide qualified magnitude guides, and warn when d and f should not be compared numerically.
SciDataReportR 20.12.0
-
PlotVolcanoEffects()now reports more detail in itsResultsTableand point tooltips. Continuous outcomes gainR(zero-order Pearson correlation) andAdjustedR(covariate-adjusted partial correlation). Two-group categorical outcomes gainGroup1Level,Group2Level,Group1Mean, andGroup2Mean(raw predictor means within each group). These values also appear in theTooltipcolumn so they show up when the plot is passed toplotly::ggplotly(tooltip = "text"). - New
FreezeTableHeader()wraps agtsummarytable (or data frame) so its header row stays frozen while scrolling long tables in HTML Quarto/R Markdown output. -
MakeComparisonTable()now uses safe internal names for its pairwise columns. Previously contrast labels such as"3 - 1"produced column names likepw_X3...1, whose trailing...1collided with tidyverse name-repair and broke downstream tibble round-trips (for examplegtsummary::as_kable_extra(), and thereforeFreezeTableHeader()). Displayed contrast headers are unchanged. -
MakeUnivariateRegressionTable()now extracts regression results directly from model coefficient tables and formats the final display withgt, avoiding the slow per-modelgtsummary::tbl_regression()path. Fitted model objects are now skipped by default; setReturnModels = TRUEto include them inModelSummaries. -
ReadSciData()now usesdata.table::fread()for ordinary delimited files when available, which substantially speeds up large.csv,.tsv, and.txtimports. Setfast_delimited = FALSEto force the previousreadrpath.
SciDataReportR 20.5.0
-
UnivariateRegressionTable()was renamed toMakeUnivariateRegressionTable()andplotForestFromTable()was renamed toPlotForestFromTable()to match the package’s naming conventions. The old names remain available as backwards-compatible synonyms (soft-deprecated). -
MakeUnivariateRegressionTable()now returns aResultselement: a tidy dataframe of estimates, confidence intervals, and p-values, one row per term. -
PlotForestFromTable()now plots from theResultsdataframe, and also accepts a (filtered)Resultsdataframe directly. Objects created by older package versions still plot via a fallback extraction.
SciDataReportR 17.0.0
Dependency stability
- Internalized the minimal half-violin geom used by
PlotSplitViolin()so the split-violin workflow no longer depends on the archivedgghalvespackage.
SciDataReportR 16.25.0
Documentation and infrastructure positioning
- Repositioned the package as scientific workflow infrastructure for reproducible life science data reporting.
- Expanded the README with workflow families, a visualization gallery, implemented methods, function dependency chains, and future heatmap/volcano plot roadmap items.
- Reorganized pkgdown navigation and reference topics around workflows, visualization functions, projection chains, and reusable infrastructure.
- Clarified the downstream relationship between
PlotCorrelationsHeatmap(),add_r_and_stars(), andgeom_starcaption(). - Added a Quarto getting-started article based on the R/Medicine workflow narrative.
Workflow-oriented public names
- Added canonical workflow-oriented names for reusable objects, models, projections, plots, and metadata workflows.
- Added
CreatePCAObject(),CreateMCAObject(),PlotZScore(),PlotPathway_KT(),MakeDataDictionary(),ProjectZScore(),ProjectSOMCluster(),CreateZScoreObject(),CreateMScoreObject(),CreateNormativeTScoreModel(), andCreateSOMClusterModel(). - Preserved backward compatibility: the older public names remain exported as wrappers and do not emit lifecycle warnings.
- Documented
PlotPathway_KT()as a SciDataReportR visualization that is expected to move to a future metabolomics-focused package.