Skip to contents

Fit HDBSCAN clusters on a frozen self-organizing map

Source: R/SOMHDBSCAN.R
CreateClusterModel_SOM_HDBSCAN.Rd

Trains the same frozen SOM used by CreateClusterModel_SOM_MClust(), then applies HDBSCAN to its node codebook. Participants inherit the cluster (or noise) label of their best-matching node. This is a node-based phenotype model: projected participants are mapped to the original nodes and are not refit with HDBSCAN.

Usage

CreateClusterModel_SOM_HDBSCAN(
  data,
  variables = NULL,
  method = c("exploratory", "finalize"),
  minPts_range = 2:10,
  cluster_selection_epsilon_range = c(0, 0.05, 0.1),
  final_minPts = NULL,
  final_cluster_selection_epsilon = NULL,
  ClusterVariableName = "Cluster",
  seed_som = 934521L,
  seed_hdbscan = 93421L,
  stability_resamples = 0L,
  stability_seed = seed_hdbscan + 1L,
  stability_progress = FALSE,
  stability_cores = NULL,
  ...
)

Arguments

data

Data frame used to train the SOM and node-level HDBSCAN model.

variables

Numeric variables used for SOM training.

method

Either "exploratory" or "finalize".

minPts_range

Candidate HDBSCAN minimum-point settings for SOM nodes.

cluster_selection_epsilon_range

Candidate HDBSCAN extraction epsilon settings.

final_minPts

Optional finalized HDBSCAN minimum-point setting.

final_cluster_selection_epsilon

Optional finalized extraction epsilon.

ClusterVariableName

Name of the appended cluster column.

seed_som

Seed used for SOM training.

seed_hdbscan

Seed retained in the model specification.

stability_resamples

Number of 90% participant subsample refits used for final-model stability. Subsamples are drawn without replacement and reuse the reference model's resolved SOM grid dimensions.

stability_seed

Seed controlling participant subsampling.

stability_progress

Whether to print subsample progress messages.

stability_cores

Number of workers for stability refits. NULL uses a conservative CPU/memory cap: the smaller of 50% physical RAM and 75% of available RAM. Explicit values warn when their estimated memory exceeds this budget.

...

Additional arguments passed to CreateClusterModel_SOM_MClust().

Value

A Pipeline_SOM_HDBSCAN object containing frozen SOM and HDBSCAN node models, assignments, diagnostics, and a projection specification. Persistence and minimum node-cluster size are higher-is-better, noise proportion is lower-is-better, and extracted class count is data-derived.

Stability output

Stability assesses internal reproducibility by full-pipeline 90% participant subsampling without replacement. For each replicate, 90% of complete participants are selected once, all preprocessing, any reduction (PCA, MCA, or SOM), and the selected clustering method are refit. Most pipelines compare assignments for sampled participants. SOM + Mclust instead maps the complete reference cohort onto every refit and compares the resulting full participant partition. Stability$settings$comparison_scope records the policy used. It is an internal sensitivity analysis, not independent-cohort validation.

Stability$settings records the analysis provenance:

  • resamples: requested number of 90% subsample refits.

  • seed: seed used to derive reproducible sampling and model seeds.

  • refit_scope: whether agreement is calculated directly in the refitted sample or after refit projection onto the reference cohort.

  • comparison_scope: the participants entering replicate agreement metrics; most pipelines use "sampled_participants", while SOM + Mclust uses "all_reference_participants".

  • resample_type: "subsample_without_replacement" for the primary stability analysis.

  • resample_fraction: the retained participant fraction, 0.90.

  • resample_fractions: every requested retention fraction. The default is 0.90; additional values create an opt-in deletion-stability curve without replacing the 90% primary screen.

  • coassignment_limit: maximum number of complete training participants (2,000) for which the full pairwise co-assignment matrix is calculated.

  • noise_policy: whether the method has noise labels. For ordinary methods it is "all clusters included"; HDBSCAN variants retain noise in ARI but exclude it from phenotype-level Jaccard recovery.

  • requested_cores, resolved_cores, and worker_policy: the requested and resolved refit workers and the reason for the resolution. Automatic resolution is capped by CPU, refit count, and the smaller of 50% physical RAM and 75% currently available RAM. Pairwise-memory methods are run serially automatically.

  • estimated_bytes_per_worker, estimated_total_bytes, physical_memory_bytes, available_memory_bytes, and safe_memory_budget_bytes: the conservative memory audit used to resolve automatic workers. An explicit worker request is preserved (within the CPU/scheduler cap) but warns if it exceeds the estimated safe budget.

Stability$replicates has one row per requested refit. Model and Classes identify the selected candidate (for HDBSCAN, Classes is the data-derived extracted count); Replicate is its sequence number; Status is "success" or a failure status; SamplingSeed and ModelSeed identify the participant draw and refit initialization policy; and Error contains the error message for an unsuccessful refit. ARI is the primary successful-refit metric: it is invariant to cluster numbering, higher is better, and it can be negative when agreement is worse than chance.

Stability$cluster_recovery has one row for each reference Cluster in each successful Replicate. Jaccard is the recovery of that reference cluster after matching it to the subsample cluster with the largest Jaccard overlap; higher is better and one is exact recovery. Model and Classes again identify the fitted candidate.

Stability$summary combines successful subsample refits: StabilitySuccessRate is successful refits divided by requested refits, an operational reliability measure that does not enter the reproducibility score. StabilityARI_Mean and StabilityARI_P05 are the mean and fifth percentile of ARI. StabilityJaccard_Mean and StabilityJaccard_Min are respectively the mean cluster-level recovery and the minimum of the cluster-specific mean recoveries across replicates. ReproducibilityScore does not include success rate. For most pipelines it is the mean of finite StabilityARI_Mean and StabilityJaccard_Mean values; SOM + Mclust uses StabilityARI_Mean directly because it selects on agreement of the final participant partition.

SOM + Mclust additionally returns StabilityARI_NonBoundary, StabilityARI_NonBoundary_P05, and non-boundary Jaccard summaries. These are secondary diagnostics calculated only on the fixed reference-model mask with SoftMargin below its threshold or SoftEntropy above its threshold excluded. The primary all-participant ARI always retains these boundary participants. boundary_summary reports their count and proportion overall and within each reference cluster. Earlier posterior-based Core fields are retained for compatibility as descriptive diagnostics; they are not the boundary-aware sensitivity measure.

Stability$failures repeats the replicate columns for unsuccessful refits, making fit failures auditable without mixing them with successful metrics.

Stability$participant_inclusion is one row per complete reference participant. .row_id identifies the merge-safe input row, Cluster its reference assignment, SuccessfulRefits the number of usable refits, and InclusionProbability the proportion of those refits in which the participant returned to that cluster's label-matched subsample cluster. Model and Classes identify the candidate. Higher inclusion is better.

Stability$cluster_inclusion summarizes inclusion within each reference Cluster: MeanInclusion, P05Inclusion, and MinInclusion are the mean, fifth percentile, and minimum participant inclusion probabilities; Model and Classes identify the candidate. Higher values indicate that all, not only the average, of a cluster is recovered consistently.

Stability$coassignment is available only when the complete training cohort has at most coassignment_limit participants. Each candidate entry has a status of "available", "skipped", or "not_available"; reason explains a non-available result; matrix is the pairwise probability that two complete reference participants are assigned together across successful subsample refits; and .row_id maps matrix rows and columns to the same identifiers returned in DataWithClusters and ProbFit$individual. Higher matrix values mean more consistent pairwise co-membership. Where more than one candidate is summarized, entries are named by its Model_Classes key. The matrix is diagnostic only and is never used for selection.

SOM + Mclust additionally returns memory-safe final-partition diagnostics for every cohort size. co_membership_by_cluster reports the proportion of within-reference-cluster participant pairs retained together (Cohesion) and the probability of co-assignment with outsiders (CrossMixing). co_membership_between_clusters reports cross-reference-cluster co-assignment probabilities. split_merge_reference and split_merge_refit identify dominant sources, shares, and effective numbers of groups, while profile_recovery reports matched-cluster correlation and RMSE for mean clustering-variable profiles on the frozen reference scale. These explain the nature of membership changes; they do not supersede ARI.

deletion_curve summarizes ARI, VI, NMI, and Fowlkes–Mallows agreement at each requested participant-retention fraction. It is useful for describing sensitivity to increasing cohort perturbation, not for selecting the least demanding fraction.

Stability$plots contains cluster_recovery (per-cluster Jaccard), partition_metrics (the ARI distribution), and cluster_inclusion; it also contains a co-assignment heatmap when the matrix is available. These diagnostics complement rather than replace ARI and Jaccard: none can turn a poorly reproducible cluster into a stable phenotype.

Metric sources: Hubert and Arabie (1985) define ARI; Jaccard (1901) defines the overlap coefficient; and Monti et al. (2003) describe resampling-based consensus co-assignment.