Improving on early exaggeration in t -SNE: Early hierarchization better preserves global structure
Résumé
In dimensionality reduction, t-SNE is a local method of neighbor embedding that requires to be carefully initialized in order to preserve the global structure of data to a good extent. In standard t-SNE, the low-dimensional embedding is initialized either randomly or with PCA. Next, gradient descent runs for two successive phases to refine the embedding coordinates iteratively. In the first phase, named early exaggeration, the attractive forces between points are artificially strengthened to prevent the repulsive forces to scatter fragments of the still poorly organized embedding, before the second phase takes over with the genuine gradient, until final convergence. A novel initialization of t-SNE is proposed in this extended work. It proceeds by hierarchizing the data points into a space-partitioning binary tree that yields faithful subsamples of data with 4,8, 16,...2[log2 N], N points; t-SNE runs on these growing subsamples, each obtained embedding initializing the next run. Between two runs, the prototypical point in each tree branch is split into its two children and the embedding is rescaled to account for the increased population. Extended experimental results with 5 repetitions show quantitatively the effectiveness of the method on a variety of artificial and real data sets, while running times get only multiplied by a small constant factor, leaving the computational complexity unchanged. This confirms that early hierarchization can advantageously replace initialization and early exaggeration, making t-SNE a more homogeneous method with fewer meta-parameters. The proposed method is compatible with any method of neighbor embedding (t-SNE, UMAP, etc.) with quadratic, log-linear, or even linear iterations, provided early exaggeration can be disabled and initial coordinates of the embedded data points can be specified.
