RECOGNITION OF A MIXTURE OF MULTIPLE GAUSSIAN PATTERNS

Abstract: In this paper a methodology for the recognition of multiple Gaussian patterns by estimating sufficient parameters of a finite mixture model (FMM) is proposed. Regular methods of FMM identification require initial guess values (IGVs) that may result in high computation time, slow convergence and or even fail to converge if the provided IGVs are far from the optimal solution. The FMM is firstly decomposed into it’s even and odd parts, which are linearised through differential techniques. Secondly the ordinary least squares (OLS) method is employed to estimate the unknown parameters in the linearised models. A Monte Carlo simulation is done to evaluate the performance of the proposed method (PM). It is shown that numerical results of the PM compare well with the simulated values. The study indicates that (i) the PM can be used symbiotically with the regular methods to compute IGVs; and (ii) can be used to estimate a general n-component Gaussian model.


Introduction
Data measurements of real world phenomena are modeled by a mixture of con-Received: April 4, 2016 Published: June 20, 2016 c 2016 Academic Publications, Ltd. url: www.acadpubl.eu§ Correspondence author tinuous density functions (see [1]) , these data may replicate finite mixture models (see [2]) (FMMs) that is a mixture of Gaussian distributions, and the mixture models are capable of approximating any arbitrary distribution (see [1]).The FMM provides a natural representation of the heterogeneity of a finite number of latent classes, it concerns modeling a statistical distribution by a mixture (or weighted sum) of other distributions, that is, Gaussian distributions.A FMM provides a parametric alternative that describes the unknown distribution in terms of a mixture of known distributions (see [3]).The FMMs are also known as, latent class models or unsupervised learning models (see [1], [4], [5]).A mixture model is usually probabilistic in nature and encompasses finite or infinite sub-populations referred to as components.The FMMs are closely related to intrinsic classification models, clustering (see [6]) and numerical taxonomy.Finite mixture models have a wide range of applications in the field of signal processing (see [7]), image processing (see [5], [8], [9]), pattern recognition (see [10], [8]) , machine learning such as modelling, clustering, classification (see [11], [5], [10]) and survival analysis.Mixture models of normal densities with common variance in the univariate case, may be estimated with a continuous distribution.In pattern reconstruction, FMM permit a probabilistic model based approach for unsupervised learning (see [8]).Generally from a statistical perspective FMM is a semi parametric/non parametric estimator of the density (see [7]).Experience suggests that usually only few latent classes are needed to approximate the density function well (see [12]).In practice FMM are flexible extensions to basic parametric models and can generate skewed distributions from symmetric components.Identification of heterogeneous information presents an enormous challenge and requires highly involved computations, but n-component density functions overcome the estimation problems (see [13]).This article is organized as follows.Section 2, describes the motivation of the proposed method.In Section 3, current methods (regular guess value method) of the estimation of parameters for finite mixture distributions containing normal distributions are investigated, a theorem is stated for estimating an general n−component mixture model.Section 4, the proposed method is discussed.In Section 5, a Monte Carlo simulation is considered to assess the performance of the proposed method.In Section 6, results and discussion of the study are presented and conclusions presented in Section 7

Motivation
The estimation of parameters for finite mixture distributions contain normal distributions components (see [10]).In the work by (see [14], [15]), a proposed method (PM), for estimating parameters of a two component mixture with equal prior probabilities, was considered, the two component mixture model was presented as where x is the study variable, σ 2 1 , σ 2 2 are the variances, µ 1 , µ 2 are the mean and λ is the proportion of each component in the mixture model and λ > 0.
The general form of Equation ( 1) is given as From Equation (2), Ψ is a complete collection of distinct parameters occurring in the mixture model and ǫ is the "small" parameter, which characterises smallness of the distance of the two means µ 1 and µ 2 from their absolute mean in the mixture.The goal here is now to estimate the parameters of Ψ.
In the case of an n-component distribution Equation (1) can be written as where The general form of Equation ( 3) for a n-component normal mixture is given as There exist several methods used to compute FMM parameters, that is, Expectation maximization (EM), Newton-Gauss, Bayesian, Markov chain Monte Carlo, Spectral method, Graphical Methods etc. However all these methods use iterative routines (see [2], [13]) where initial guess values must be provided to start the iterative process (see [5]) before computing the required optimal solutions.The challange with initial guess value routines are, (i) slow convergence (see [10], [13]); (ii) High computational time (see [1]) and (iii) may fail to converge if the initialization values are far from the required solution.

Current Methods of Solution
As mentioned in Section 2, the existing methods of solution use iteration procedures to determine the optimal solutions.
Consider the three component Mixture Model given as The function f (x) in Equation ( 5) is decomposed into its even and odd parts and Considering the expansion of f e (x) and f o (x) in series with respect to ǫ (at ǫ = 0).At first approximation with error of order one, Equation (5) yields Equations (8 & 9) From Equation ( 8) the following parameters can be approximated A i and α i .From Equation ( 9) the following parameters can be obtained a i where At second approximation with error of order two, Equation ( 5) yields Equations (10 & 11) where where and By change of parameters, let Z = x 2 Equation (10 & 11) can be written as Equation ( 12) represents the solution of an exact linear differential equation with real roots, having a characteristic equation of the form Thus the solution of Equation ( 14) is λ j,k = −α i where j = 1, 3, 5, where n is the number of mixture models can be used to estimate the parameters A i , α i and a i of Equation (15).Using the first approximation Equations (8 & 9), the goal function for a three component mixture model is given as The approximated goal function of Equation ( 15), G 1 is given by Deriving the partial derivatives of Equation ( 16) with respect to A i , B i , C i and α i .Which generate the following system of equations where It can be observed that the least squares normal Equations (17, 18, 19 and , 20) can not be solved explicitly to give exact solutions.Iterative methods that require IGVs to initiate the iteration process have to be supplied.Thus the justification of developing the proposed method.Theorem 1.A generalised n-component Gaussian mixture model can be approximated using a series about a predetermined origin.
Proof.Considering the general form of an n-component normal mixture model, Equation ( 4) can be generalised as Equation ( 21) can be decomposed into even and odd. where The n-component mixture model odd part is represented by where The working error of the even and odd parts are of order two O(ǫ 2 x 2 ) + O(ǫ 2 ) , the error indicates the closeness of the mean µ i of the mixture model.Extending the order of the error say to order three will only increase the number of parameters to be estimated thus rendering the formulation to be less accurate, that is, the even and odd part with error of order three now becomes Equation (24) and Equation (25) respectively.
The odd part Equation ( 23), was divided by an x term so as to approximate it as an even part Equation (26).In this way the odd part (23), can be estimated as those of the even part Equations (22, 26 and 28) respectively. where Comment 2. Hence The parameters of the even function Equation (22), are The parameters of the odd function Equation (23), are Using the data for the n-component mixture model Equation ( 21) the parameter estimates B i and A i , for i = 1, 2, • • • , n may be obtained using the ordiary least squares (OLS) method, via parametric ODEs.
In order to simplify the computations, let y = x 2 , then f (x) = f ( √ y), for ease of presentation this can be written as f (x) = f ( √ y) = f (y).Thus the even part can be represented as Similarly the odd part is represented by where By letting Equation ( 31) is the solution to the exact linear ordinary differential equation with real roots of order n, Equation (35).
Using the multiple OLS method, the estimated values of η i for i = 1, 2, • • • , n may be obtained using the data set {y i , f e (y i )} for i = 1, 2, 3, • • • , m.The numeric computations of d i dy i f e for i = 1, 2, • • • , n are used to obtain the estimated parameters of η i .The multiple regression is given by Equation (36). where The auxiliary equation for the linear ODE Equation ( 35) is given by Considering the linear roots of the auxiliary Equation (37) thus Thus solving for the roots of Equation (37) yields the parameters of The finite mixture dataset {x i , f 31) can be solved using multiple ordinary least squares (OLS) method to obtain parameter estimates for A i , for i = 0, 1, 2, • • • , m, since Y ji = e −α j y i , where i = 1, 2, • • • , n and j = 1, 2, • • • , m. Substituting for A i , in to Equation (22), n equations are obtained but with 2n unknown parameters, at this stage, the parameters cannot be estimated.From Equation (22 The odd part can be estimated exactly as the even part, since the relation in Equation (28) holds.Thus the methodology of estimation are similar for the odd and even parts. where The even and odd components can be compared using Equation (31 & 32). Consider Let f (y) = f (Y ) and substitute Equation (42) into Equation ( 41) to obtain Considering the dataset for the mixture model {x i , f (x i )}, using the multiple OLS on the dataset {x i , f (x i )} for the mixture model Equation ( 43) the parameter estimates of This completes the proof.
The function now has n equations with n unknowns.Combining the Even and Odd functions Equations (36 & 43) estimating the general unknown parameters A i , µ i , of the model Equation ( 21), the sufficient parameters of the Mixture Model can be obtained, that is, a system of n Equations and n unknowns.

Proposed Method
In this Section Theorem 1 is applied to a three component mixture model.The 3-component univariate normal density mixture model is considered and can be formulated as follows where Ψ = (σ i , µ i , ǫ) for i = 1, 2, 3. (44) A complete collection of distinct parameters for the mixture model is given by Ψ and ǫ is the "small" distance of the means µ i for i = 1, 2, 3. Thus Equation (44) can be written as , where where (46) where The odd part of Equation (47) may be estimated as the even part as where The parameters α i for i = 1, 2, 3 for Equation (46) and Equation (48) can now be estimated using the ordinary least squares method (OLS).Using the change of parameters y = x 2 Equation (46) is the Even Function Transformed. , The transformation of the odd and even parts of the three FMM is given as where (50) where Using the Ordinary Least Squares (OLS) to estimate the parameters A i and B i for i = 1, 2, 3 from the data set {x i , f (x i )} for i = 1, • • • , m. Equations (50 & 51) are solutions to the homogeneous linear ordinary differential equation (ODE) with constant coefficients Equation (52).
Using the multiple OLS method, the estimated valeues of η i for i = 1, 2, 3 maybe obtained using the data set {y i , f e (y i )} for i = 1, 2, 3, • • • , m.
The solutions η i yields the values of α i using the characteristic equation (53) Thus solving Equation (54) will yield: The Finite Mixture dataset {x i , f (x i )} for i = 1, ..., m a new dataset {y i = x 2 i , f (y i ) = f (x 2 i )} for i = 1, ..., m can be constructed.Equation (50) can also be solved using multiple ordinary least squares (OLS) method to obtain parameter estimates for A 1 , A 2 and A 3 , since Y 1i = e −α 1 y i , Y 2i = e −α 2 y i and Y 3i = e −α 3 y i .Substituting for A 1 , A 2 and A 3 , in to Equation (46), three equations are obtained but with six unknown parameters, at this stage, the parameters cannot be estimated.From Equation (46) In the same way for the odd part of the finite Mixture Model function is estimated, Equation (51), since the even by is similar to the odd part, Equation (28).The methodology of estimation are similar. where The even and odd components can be compared using Equation (49 & 57).Let y = x 2 then f (x) = f ( √ y) and, f (y) = f ( √ y). Consider Let f (y) = f (Y ) and substitute Equation (58) into Equation (57) to obtain Considering the dataset for the mixture model {x i , f (x i )}, using the multiple OLS on the dataset {x i , f (x i )} for the mixture model Equation (59) the parameter estimates of B i for i = 1, 2, 3.
The odd function was divided by an x term so as to approximate it as an even function.In this way the equations from the odd function can be compared to those from the even component.The odd function has three equations with six unknowns.Combining the Even and odd functions Equation (50 & 51) estimating the general unknown parameters A i ,µ i , of model (61), the sufficient parameters of the mixture model can be obtained.i.e A system of six Equations and six unknowns.
The estimated parameters A i and µ i , for i = 1, 2, 3 can be used as the IGVs for the unknown parameters in the iteration methods for the Three Mixture Model.

Three Component Mixture Model
The three component mixture model is represented by Equation (61). where From Equation (60) σ i is the standard deviation µ i is the mean λ i is the scaling factor. , A simulation of a three component mixture model Equation (61) with the following individual parameters as ε = 0.1, 408248.This is presented as f (x) = e −3(x−(0.1)(0.1)) 2 + 1.5 e −2.5(x−(0.1)(1.3)) 2 + 2 e −3(x−(0.1)(1)) 2 . (62) Figure 1: Three Component Mixture Model with small variance According to Figure 1, the three component mixture model may appear as one Gaussian distribution.The main idea is to estimate the sufficient parameters that represent the three component mixture model by reducing the parameter dimensions of the original mixture model on subsequent estimation steps (see Section 4).

Results and Discussion
Tables 1 & 2 represent the result of the proposed method for a three component mixture model.In Table 1, the parameters of the three component mixture model are presented, and in Table 2, the sufficient parameters of the three component mixture model are presented, which are the main focus of the estimation problem.The exact values represent the Monte Carlo simulation of the model and the estimate values represent the values obtained using the proposed method.
Considering the results in Section 6, according to Table 1, the values of the exact simulated model of the three mixture model can be estimated using the proposed method (Theorem 1).The sufficient parameters of the three mixture model can also be computed using the proposed method, according to Table 2, the sufficient parameters are accurately estimated with minimal errors.The proposed method thus gives the sufficient parameters for the mixture model that can be used to approximate the mixture model.

Conclusion
In this paper the problem of estimation of the finite mixture model was considered.The problem was solved using the proposed method that does not require initialization or iterative methods for estimating the sufficient parameters of the FMM.It was also shown in the Monte Carlo simulation how the proposed method can be applied to a FMM and generalised for an n-component FMM.All simulations and computations were performed in Mathematica R .Considering the estimates, of the proposed method, it is reasonable to conclude that, the PM produced acceptable results.Nevertheless, it has to be noted that the results of the regular guess value method largely depend on the initial approximations.Hence the PM can be used as an algorithmic approach to computing initial approximations for the iteration methods.

Table 1 :
Parameter values for the Three Mixture Model

Table 2 :
Sufficient Parameter values for the Three Mixture Model