How Dynamic Quantum Clustering Transforms Data Visualization
Dynamic Quantum Clustering Data Structure Visualisation
Techniques for Quantum Clustering
Quantum Clustering (QC) uses quantum-physics mathematical and conceptual concepts to cluster data. Since QC clusters by density, regions with more data points form clusters. The 2001 QC technique uses a multivariate Gaussian distribution for each data point.
The quantum-mechanical wave function is formed by adding these distributions. This wave function describes data points' expected positions.
QC uses the time-independent Schrödinger equation to build a possible surface, called the “landscape” of the data set, for which the wave function is stable. High data density locations are intimately associated to ‘low’ places in this landscape. In the first QC strategy, traditional gradient descent moves data points ‘downhill’ in this terrain, causing them to converge into close minima and reveal clusters.
The Evolution of DQC
The 2009 invention of Dynamic Quantum Clustering (DQC) by David Horn and Marvin Weinstein greatly enhanced QC. DQC is a powerful visual approach for massive, high-dimensional data. It uses data density differences in feature space to locate subsets of data with correlations among all measured variables. DQC replaces hypothesis-driven searches with an organic data structure development process.
Core Mechanism: Non-Local Descent and Quantum Evolution The quantum potential landscape QC generated is used by DQC. DQC replaces gradient descent with quantum evolution.
Dynamic Quantum Clustering (DQC) re-represents each data point using a multivariate Gaussian distribution or wave function. This wave function's dynamics inside the potential are calculated using the time-dependent Schrödinger equation. By recalculating this progression over minuscule time increments, a new data point position is projected. The iterative method gives each data point a trajectory until it stabilises and stops travelling.
The Ehrenfest theorem from quantum mechanics predicts that this quantum evolution will be like the data point falling in the potential landscape. This concept of “in expectation” is crucial since the point's movement is not solely determined by the gradient of the potential at its particular location, unlike classical physics. The mobility is regulated by a complex interaction between the potential and the wave function, and the point's wave function spans the landscape.
This causes non-local gradient descent. Lower terrain ‘attracts’ the point; the lower, the more attractive; the farther away, the less desirable. Higher elevations "repel" the point.
Tunnelling beyond Local Minima
The non-locality of Dynamic Quantum Clustering's quantum development allows tunnelling. A data point tunnelling may appear to bypass a potential impediment on its way to a deeper minimum.
Non-convex gradient descent, especially with high-dimensional data (the curse of dimensionality), can trap points in multiple small, local minima that do not represent significant structure, so this capability is crucial. DQC uses non-local gradient descent and tunnelling to solve this longstanding problem.
Computational Strategy: Limited Basis
The quantum evolution approach is hindered by the fact that processing time increases rapidly with data points. Potential development and point evolution require extensive computation.
For large data sets, Dynamic Quantum Clustering (DQC) employs a limited foundation to circumvent this intractability. In DQC, b is substantially less than n, hence b is used as the basis instead of n quantum eigenstates from all n data points. These b basis points are carefully chosen to encompass the entire data set, usually by selecting points as widely apart as possible.
The generated eigenstates represent non-basis points imperfectly but completely reflect the selected basis points. The hyperparameter sigma, or Gaussian width, affects how much information is lost and must be wide enough for the basis to represent the remaining points. This base' size is the data structure's "resolution" (b). Even with powerful processing, the maximum practicable basis size in 2020 is 1,500–2,000 points.
QC uses sigma, but Dynamic Quantum Clustering (DQC) adds two hyperparameters: the time step and the mass of each data point, which controls tunnelling behaviour. Sigma adjustment is necessary to understand new data, but time step and mass can often be set to appropriate defaults.
Dynamic Structure Exploration Visualisation
DQC's dynamic visualisation of computed trajectories for each data point is unique. Each data point moves synchronously along its course in an animated sequence.
Dynamic Quantum Clustering (DQC) analysis yields useful information from the whole path and the points' clustered destination. These animations often display channel structures that flow into a cluster, like riverbeds or lakes in the possible landscape.
Even though displays can only show three dimensions, these channels show high-dimensional action. These trajectories should be visualised in the first three dimensions defined by Principal Component Analysis for maximum information. Remember that the representation is a 3D view at higher-dimensional motion, not a 3D embedding of the trajectories.
Channels can be viewed as regressions where their position may correspond with significant metadata or as subclusters that merge into the main cluster from different directions. A DQC analysis produces a “movie” that explains how and why data points are categorised into “extended structures” or basic clusters.
Wide-ranging uses Dynamic Quantum Clustering (DQC) is ideal for unconventional exploratory analysis, which lets users find unexpected data without modelling. DQC has been found to find substantial, often tiny, data subsets with valuable hidden information.
Biology, finance, physics, engineering, and economics use QC variations like DQC. DQC has worked well on complex datasets from biology, finance, x-ray nano-chemistry, condensed matter, and seismology. Experience shows that complex datasets include intriguing features that typical clustering approaches miss. Dynamic Quantum Clustering (DQC) finds hidden structures.
