In this talk, we will talk about the problem of grouping a collection of unconstrained face images in which the number of subjects is not known. We propose unsupervised clustering algorithms which are based on measuring the affinities between local neighborhoods in the feature space. By learning the local information for each neighborhood, information about the underlying structure is encapsulated. The encapsulation aids in measuring the neighborhood similarity. Extensive experiments show that the proposed methods are superior candidates for clustering unconstrained faces when the number of subjects is unknown. Unlike conventional linkage and density-based methods that are sensitive to the selection operating points, the proposed approach attains more consistent and improved performance. In addition, we will also discuss an extension using the same similarity learning approach for single subject tracking.