How to construct a phylogeny given a set of the DNA sequences is always a popular topic in both biological and statistical research. A new method based on an ancestral mixture model for building a gene tree from Single Nucleotide Polymorphism (SNP) data was developed by Chen and Lindsay (2006, 2010a, 2010b, 2011). The sieve parameter in the model plays the role of time in the evolutionary tree of the sequences. By varying the sieve parameter, one can create a hierarchical tree that estimates the population structure at each fixed backward point in time. A software, called MixtureTree, was developed for this purpose.
In this talk, we will briefly review the model and present an application to the clustering of the mitochondrial sequences using the mixture tree software. Different optimization algorithms, including EM algorithm, and Modal EM will be discussed. Comparison with some existing methods will be presented as well.