feat: add clustering support across models and notebooks - #841
Open
constanzaru wants to merge 27 commits into
Open
feat: add clustering support across models and notebooks#841constanzaru wants to merge 27 commits into
constanzaru wants to merge 27 commits into
Conversation
Refactor ModelJob to delegate task-specific execution to registered TaskExecutor components. Move the supervised execution flow into SupervisedTaskExecutor to keep it compatible with cross-validation changes, and introduce ClusteringTaskExecutor for target-free clustering runs. Based on supervised ModelJob changes by Andrés Salazar.
Extract shared sklearn preprocessing and persistence into SklearnBaseModel. Add SklearnLikeClusterer as the adapter for clustering algorithms and introduce KMeans and DBSCAN wrappers for ClusteringTask.
Adds a converter that fits any registered ClusteringModel, appends a cluster-label column to the dataset, and saves a JSON report with metrics, cluster sizes, profiles, and algorithm specific attributes.
…rage Extended BaseExplorer and ExplorerJob so explorers can declare whether they need the report produced by a converter, which is then loaded and injected at runtime. Converter reports are now stored under notebook/<id>/converters/ instead of datasets/<uuid>/converters/.
Adds ClusteringProfileExplorer, which reads the converter report to summarize a clustering run without requiring column selection. The ClusteringProfileVisualizer renders cluster sizes, evaluation metrics, and the distinctive feature profile of each cluster.
…thm validation Algorithm-specific explorers (HDBSCAN, Agglomerative) raise a descriptive error if the last converter ran a different algorithm. Column selection is validated against the converter scope. Converter report now includes feature_columns and algorithm_key to support both validations.
Clustering trains and evaluates over the entire dataset instead of a train/validation/test partition. Extend the existing Metric.split enum with a FULL value instead of introducing a parallel metrics path, so the runs API, live updates, and model comparison scoring keep working unmodified for tasks with no split.
Clustering sessions train on the full dataset with no validation split, so there is no goal metric to score Optuna trials against. Automatically maximizing an internal metric (Silhouette, Calinski-Harabasz, Davies-Bouldin) also tends to reward degenerate clusterings, since there is no external check on whether the result is meaningful. Hyperparameter optimization is therefore intentionally unsupported for clustering at this stage.
Clustering sessions have no train/validation/test split, so RunResults and LiveMetricsChart now detect this from the session config and adjust accordingly: the split/level selectors are hidden in favor of a single global-evaluation view, and the Predictions tab is hidden (clustering models don't implement predict()) while Explainability and Hyperparameters are left to their existing empty-state handling, matching how any other task without available components already behaves.
Restrict the clustering converter to numeric columns and reject silhouette explorer selections that include columns outside the fitted model's scope. Require an explorer's requires_converter_class to match the most recently finished converter, not just any past match. Always report DBSCAN's noise point count, including zero. Guard the notebook search panel against non-string component metadata.
…to be fixed on units workflow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces end-to-end clustering support in DashAI, covering both the Models and Notebooks workflows.
In the Models module, clustering is added as a first-class unsupervised task. Clustering sessions do not require target columns or train/validation/test partitions. Models are trained using the selected input features and evaluated with internal clustering metrics over the complete dataset.
In the Notebooks module, a configurable Clustering converter is introduced. It dynamically exposes the parameters of the selected algorithm, appends the generated cluster labels to the dataset, and persists a report containing metrics, cluster sizes, feature profiles, and algorithm-specific fitted attributes. Specialized clustering explorers consume this report to generate visual analyses.
The existing modeling architecture was also refactored to separate supervised and unsupervised contracts.
ModelJobnow delegates task-specific execution to a compatible task executor instead of assuming that every model requires targets and train/validation/test splits.Type of Change
Changes (by file)
Below are the most relevant files modified or added by this PR.
Task and model architecture
DashAI/back/tasks/base_task.py: Extended task metadata to declare whether a task requires target columns and which dataset split strategy it supports.DashAI/back/tasks/supervised_task.py: Added the common contract for tasks trained with input and target columns.DashAI/back/tasks/unsupervised_task.py: Added the base contract for tasks that do not require target columns or dataset splits.DashAI/back/tasks/clustering_task.py: Added the clustering task, including numeric input requirements, scoring profiles, and multilingual metadata.DashAI/back/tasks/classification_task.py,regression_task.py, andtranslation_task.py: Updated existing tasks to inherit from the new supervised task contract.DashAI/back/models/base_model.py: Refactored the base model contract so it no longer assumes supervised training orX/yevaluation.DashAI/back/models/supervised_model.py: Moved supervised metric calculation and persistence fromBaseModelinto a dedicated supervised model abstraction.DashAI/back/models/clustering_model.py: Added the clustering model contract, runtime model discovery, label retrieval, and post-fit attribute reporting.DashAI/back/models/model_factory.py: Adapted model creation so it can instantiate both supervised and clustering models.Model job execution
DashAI/back/job/model_job.py: Refactored model jobs to delegate training and evaluation to the task executor associated with the selected task.DashAI/back/job/model_job_context.py: Added a shared runtime context containing the run, session, dataset, task, registry, database session, and configuration.DashAI/back/job/task_executors/base_task_executor.py: Added the common interface for task-specific model execution.DashAI/back/job/task_executors/supervised_task_executor.py: Preserved the existing supervised workflow, including dataset splits, optimization, training, and metric calculation.DashAI/back/job/task_executors/clustering_task_executor.py: Added input-only clustering execution, complete-dataset evaluation, and persistence of clustering metrics.Clustering models
DashAI/back/models/scikit_learn/sklearn_like_clusterer.py: Added a reusable adapter between the DashAI clustering contract and scikit-learn estimators.DashAI/back/models/scikit_learn/kmeans_clustering.py: Added the scikit-learn K-Means model and its fitted centroid and inertia metadata.DashAI/back/models/scikit_learn/dbscan_clustering.py: Added the scikit-learn DBSCAN model, including noise-point metadata.DashAI/back/models/scikit_learn/agglomerative_clustering.py: Added hierarchical agglomerative clustering and linkage information for dendrogram generation.DashAI/back/models/scikit_learn/gaussian_mixture_clustering.py: Added Gaussian Mixture clustering with configurable covariance and component parameters.DashAI/back/models/scikit_learn/hdbscan_clustering.py: Added HDBSCAN clustering and cluster persistence metadata.DashAI/back/models/scikit_learn/spectral_clustering.py: Added Spectral Clustering with configurable affinity and label-assignment parameters.DashAI/back/models/faiss/faiss_base_model.py: Added common FAISS data conversion and model serialization utilities.DashAI/back/models/faiss/faiss_like_clusterer.py: Connected the FAISS infrastructure with the DashAI clustering model contract.DashAI/back/models/faiss/faiss_kmeans_clustering.py: Added FAISS K-Means for accelerated clustering of large numeric datasets.DashAI/back/models/faiss/faiss_dbscan_clustering.py: Added DBSCAN using FAISS-accelerated neighborhood searches.Clustering metrics
DashAI/back/metrics/clustering_metric.py: Added the base clustering metric contract and common input validation, including noise-point filtering.DashAI/back/metrics/clustering/silhouette.py: Added the Silhouette score.DashAI/back/metrics/clustering/calinski_harabasz.py: Added the Calinski–Harabasz score.DashAI/back/metrics/clustering/davies_bouldin.py: Added the Davies–Bouldin score.DashAI/back/core/enums/metrics.py: Added theFULLmetric split for evaluations performed over the complete dataset.DashAI/back/api/api_v1/endpoints/runs.py: Added full-dataset metrics to run responses, score calculation, and run reset behavior.Clustering converter
DashAI/back/converters/base_converter.py: Added optional converter reports for exposing information generated during execution.DashAI/back/converters/category/clustering.py: Added the clustering converter category.DashAI/back/converters/clustering/clustering.py: Added a generic clustering converter with dynamic algorithm selection, numeric feature validation, cluster-label generation, unique output column names, metrics, cluster profiles, and algorithm-specific reports.DashAI/back/converters/converter_report.py: Added helpers to save and load converter reports as JSON files.DashAI/back/job/converter_job.py: Updated converter execution to persist reports after a successful conversion.Clustering explorers
DashAI/back/exploration/base_explorer.py: Added optional runtime context and metadata for explorers that require converter reports.DashAI/back/exploration/clustering_explorer.py: Added the common base class for clustering explorers.DashAI/back/job/explorer_job.py: Added converter-report loading and validation of the required converter and clustering algorithm.DashAI/back/exploration/explorers/clustering_profile.py: Added a structured summary of the clustering algorithm, metrics, cluster sizes, and feature profiles.DashAI/back/exploration/explorers/clustering_scatter.py: Added two-dimensional cluster visualization using PCA or t-SNE.DashAI/back/exploration/explorers/clustering_heatmap.py: Added a heatmap comparing standardized feature values across clusters.DashAI/back/exploration/explorers/silhouette_plot.py: Added per-cluster Silhouette visualization.DashAI/back/exploration/explorers/cluster_distribution.py: Added feature-distribution comparisons between clusters.DashAI/back/exploration/explorers/dendrogram.py: Added hierarchical clustering dendrogram visualization for Agglomerative Clustering.DashAI/back/exploration/explorers/cluster_stability.py: Added cluster persistence visualization for HDBSCAN.DashAI/back/initial_components.py: Registered the clustering task, models, metrics, converter, task executors, and explorers.Models frontend
DashAI/front/src/components/models/CreateSessionSteps.jsx: Adapted session creation to task-specific target and split requirements.DashAI/front/src/components/models/modelSession/PrepareDatasetStep.jsx: Added support for tasks without output columns or dataset partitions.DashAI/front/src/components/models/modelSession/DivideDatasetColumns.jsx: Updated column selection so output columns are hidden for unsupervised tasks.DashAI/front/src/components/models/AddModelDialog.jsx: Disabled hyperparameter optimization options for clustering sessions.DashAI/front/src/components/models/ModelCenterContent.jsx: Added supervised and unsupervised task information to the task-selection interface.DashAI/front/src/components/threeSectionLayout/OptionBox.jsx: Added task-type badges to selection cards.DashAI/front/src/components/models/LiveMetricsChart.jsx: Adapted live metric visualization to sessions without train, validation, or test splits.DashAI/front/src/components/models/ModelComparisonTable.jsx: Added support for comparing full-dataset metrics.DashAI/front/src/components/models/RunResults.jsx: Adapted run results to clustering sessions and full-dataset evaluation.DashAI/front/src/components/models/SessionVisualization.jsx: Added automatic handling of thefullmetric split in clustering sessions.DashAI/front/src/pages/results/components/ResultsGraphs.jsx: Added graph support for full-dataset metrics.DashAI/front/src/types/run.ts: Addedfull_metricsto the run interface.Dynamic forms and Notebooks frontend
DashAI/front/src/components/shared/FormSchemaRenderFields.jsx: Added support for conditional schemas whose parameter fields depend on the selected algorithm.DashAI/front/src/components/configurableObject/Inputs/SelectInput.jsx: Added descriptions for selectable algorithm options.DashAI/front/src/utils/schema.js: Adapted frontend validation for conditional object schemas.DashAI/front/src/components/notebooks/ColumnSelector.jsx: Adapted scope selection for clustering converters and explorers.DashAI/front/src/components/notebooks/RightBar.jsx: Added clustering-specific explorer validation and user feedback.DashAI/front/src/components/notebooks/explorerCreation/ScopeStepExplorer.jsx: Updated explorer scope handling for the generated cluster column.DashAI/front/src/components/notebooks/explorer/visualizations/ClusteringProfileVisualizer.jsx: Added the frontend visualization for clustering profile results.DashAI/front/src/components/notebooks/explorer/useExplorerResults.jsx: Added support for loading clustering profile results.DashAI/front/src/components/notebooks/explorer/tabs/Results.jsx: Registered the clustering profile visualizer in the explorer results view.Internationalization and dependencies
DashAI/front/src/utils/i18n/locales/*/common.json: Added translations for the full-dataset metric split.DashAI/front/src/utils/i18n/locales/*/experiments.json: Added translations for sessions without targets or dataset splits.DashAI/front/src/utils/i18n/locales/*/models.json: Added translations for clustering sessions and supervised/unsupervised task badges.DashAI/front/src/utils/i18n/locales/*/datasets.json: Added translations for clustering converters and explorers.requirements.txt: Addedfaiss-cpuas a dependency for FAISS-backed clustering models.Tests
tests/back/api/test_components_api.py,tests/back/api/test_jobs.py,tests/back/api/test_predict_api.py, andtests/back/tasks/test_tasks.py: Updated existing tests to reflect the new task hierarchy, task executors, component metadata, and supervised model behavior.Testing (optional)
Run the affected backend tests to verify component registration, task-executor routing, prediction compatibility, and task metadata:
tests/back/api/test_components_api.pytests/back/api/test_jobs.pytests/back/api/test_predict_api.pytests/back/tasks/test_tasks.pyRecommended clustering verification:
Notes (optional)
Design decisions
test partitions.
-1is treated as noise for density-based clustering algorithms.clustering metadata cannot be applied to a modified dataset.
before execution.