A novel framework for face recognition using robust local representation–based classification

Abstract

Face recognition via representation-based classification is a trending technique in the recent years. However, the recognition performance of the systems using such a technique degrades in an unconstrained environment. In this article, a novel framework is proposed for representation-based face recognition. To deal with the unconstrained environment, a pre-process is used to frontalize face images, and aligned downsampling local binary pattern features of the frontalized images are used for classification. A dimension reduction is then adopted in order to reduce the computation complexity via an optimized projection matrix. The recognition is carried out using an improved robust sparse coding algorithm. Such an algorithm is expected to avoid the overfitting problem. The open-universe test on labeled faces in the wild data sets shows that the recognition rate of the proposed system can reach 95% with a recall rate of 80%, which is best among those representation-based classification face recognition systems.

Introduction

Face recognition (FR), a popular area of research in computer vision and machine learning, has been widely studied for decades.^1,2 The automatic FR systems, including those using representation classification (RC) techniques such as sparse RC (SRC), ³ have already achieved very impressive performance over large-scale images in the constrained environment (cooperative users with controlled indoor illumination).^4,5 It remains a challenge on how to enhance the performance of the SRC-based FR systems in an unconstrained environment. As SRC is an non-deterministic polynomial (NP)-hard combinatorial problem, it is usually relaxed into an l₁-norm-based convex problem, which can be solved with an iterative shrinkage/thresholding algorithms. ⁶ Recently, it is found that a sparser solution can be obtained by solving its non-convex relaxation rather than by solving its convex relaxation. ⁷ However, solving the non-convex relaxation is usually very costly, which limits its applications in real time.

It has been shown that the nonnegative coefficient constrained SRC is more effective.^6,8 This is mainly due to the fact that this approach can avoid overfitting. The algorithms proposed in the Bioucas-Dias and Figueiredo study ⁶ can be expanded to a finite number of terms, which collectively resemble the typical neural network layers. Consequently, a lengthy sequence of iterations can be treated as a deep learning (DL) network with shared layer weights, ⁹ while the rectified linear unit (ReLU) actually corresponds to the nonnegative coefficient constraint. As well known, however, DL is computationally very expensive.

It should be pointed out that approaches using the representation-based classification techniques cannot be directly used for FR in an unconstrained environment. An algorithm of face alignment by sparse and low-rank matrix decomposition (SLMD) was proposed in Wu et al. ¹⁰ However, it is not applicable for large-scale image data set. The conventional modeling of face alignment is based on affine transform with two-dimensional (2D) model face detection and registration.^11,12 Recently, a face frontalization framework was proposed with three-dimensional (3D) reference model in Hassner et al., ¹³ which is more effective in FR and gender estimation. A multi-view FR based on tensor subspace analysis was proposed by Gao and Tian, ¹⁴ but it is very difficult to apply in the uncontrolled environment. The mirror face image information was adopted in Xu et al. ¹⁵ to improve the recognition performance; this improvement is, however, very limited as the multi-view face is not frontalized. A robust sparse coding (RSC) method was proposed by Yang et al., ¹⁶ which is the maximum likelihood estimation (MLE) solution of the sparse coding problem. However, the sparse coding algorithm for RSC, involving iterative re-weighting, is costly for a large-scale data set.

In this article, we propose a 3D model–based robust local nonnegative sparse representation classification (RLNSRC), which is intended to deal with FR in uncontrolled scenarios. The main contributions in this article are given as follows:

• Instead of using the raw face image, a 3D frontalization based on the aligned downsampling local binary pattern (ADLBP) feature is adopted. This allows us to deal with the uncontrolled environments effectively.

• A compressive sensing–based RLNSRC (CS-RLNSRC) scheme is proposed for FR. An algorithm is derived for designing the optimal projection matrix that is used to compress the ADLBP feature signals. Such a system is intended to reduce the computation complexity and to prevent the overfitting problem.

The article is outlined as follows. The “Related works and problem formulation” section is devoted to providing some existing works on FR systems using the representation-based classification, which are closely related to ours, to be presented in this article. Our main contribution is given in “A novel framework for FR” section, in which a CS-RLNSRC framework is proposed. Experiments are carried out in the “Experiment results” section to examine the performance of the proposed system. To end this article, some concluding remarks are given in the “Conclusion” section.

Related works and problem formulation

In this section, we will review some existing representation-based techniques developed for classification and formulate the problem we investigate in this article.

Suppose we have J subjects and the jth subject $X_j$ has $L_j$ images/samples which are used for training/learning the classifier. Denote $X_j\triangleq \{{\mathit{x}}_{\mathrm{jl}}{\}}_{l=1}^{L_j}$ , where ${\mathit{x}}_{\mathrm{jl}}\in {\Re }^{N\times 1}$ is the lth training sample of the subject $X_j$ and is usually assumed to be normalized, that is, $||{\mathit{x}}_{\mathrm{jl}}|{|}_2=1$ .

In the linear representation-based classification approach, any sample $\mathit{x}\in {\Re }^{N\times 1}$ belonging to the J subjects is assumed to be a linear combination of a set of atoms $\{{\psi }_k\}$

$$\mathit{x}=\sum _{k=1}^L{s}_k{\psi }_k\triangleq \mathbf{\Psi }\mathit{s}$$ (1)

where $L\triangleq \sum _{j=1}^J{L}_j$ and $\mathbf{\Psi }\in {\Re }^{N\times L}$ is called dictionary with its kth column ${\psi }_k$ , called the kth atom. Let $\mathbf{\Psi }$ have the following partition

$$\mathbf{\Psi }\triangleq \left[\begin{array}{cccccc}{\mathbf{\Psi }}_1\hfill & {\mathbf{\Psi }}_2\hfill & \dots \hfill & {\mathbf{\Psi }}_j\hfill & \dots \hfill & {\mathbf{\Psi }}_J\hfill \end{array}\right]$$ (2)

where ${\mathbf{\Psi }}_j\in {\Re }^{N\times L_j}$ is the dictionary for jth class/subject. Traditionally, ${\mathbf{\Psi }}_j$ is formed with the training samples of the same class

$${\mathbf{\Psi }}_j=\left[\begin{array}{ccccc}{\mathit{x}}_{j1}\hfill & \cdots \hfill & {\mathit{x}}_{\mathrm{jl}}\hfill & \cdots \hfill & {\mathit{x}}_{j{L}_j}\hfill \end{array}\right],\forall j$$

For a given query face, represented by the feature vector $\mathit{x}$ , a representation-based classifier is different from another just in the way how the coefficient vector $\mathit{s}$ is obtained. For a given representation-based classification algorithm, the procedure to classify $\mathit{x}$ is outlined below.

• Alg _RC

• Step I: Determine the coefficient vector $\hat{\mathbf{s}}$ ;

• Step II: Compute the corresponding classification deviation ${\sigma }_j^2$ for $\mathit{x}$ to belong to the jth class for all $j=1,2,\dots ,J$

$${\sigma }_j^2=\parallel \mathit{x}-{\mathbf{\Psi }}_j\hat{s}{\parallel }_2^2,\forall j$$ (3)

• Step III: Classify $\mathit{x}$ to the $\hat{j}\mathrm{th}$ class, where $\hat{j}$ is determined with

$$\hat{j}\triangleq \arg \underset{j\in \{1,2,\cdots ,J\}}{min}\left\{{\sigma }_j^2\right\}$$ (4)

In the next subsections, we will specify some of the existing representation-based classifiers.

Linear regression classification

In the linear regression classification (LRC), the coefficient vectors are obtained from the following minimization

$${\hat{\mathit{s}}}_j=\arg \underset{\mathbf{s}}{min}\parallel \mathit{x}-{\mathbf{\Psi }}_j\mathit{s}{\parallel }_2^2,\forall j$$ (5)

As $N>L_j$ usually holds, equation (5) is well conditioned, and hence ${\hat{\mathbf{s}}}_j$ can be obtained with

$${\hat{\mathit{s}}}_j={\left({\mathbf{\Psi }}_j^T{\mathbf{\Psi }}_j\right)}^{-1}{\mathbf{\Psi }}_j^T\mathit{x}$$ (6)

The corresponding classification deviations $\{{\sigma }_j^2\}$ can be computed using equation (3) and the classification can be done using equation (4).

Sparse representation–based classification

The SRC assumes that $\mathit{x}$ can be well represented by a sparse vector $\mathit{s}$ , whose nonzero elements correspond to one class only. In that case

$$\hat{s}\triangleq \arg \underset{\mathbf{s}}{min}||\mathit{s}|{|}_0\mathrm{s}.\mathrm{t}.\mathit{x}=\mathbf{\Psi }\mathit{s}$$ (7)

where $||\mathit{v}|{|}_0$ denotes the number of nonzero elements in vector $\mathit{v}$ . Finding the solution to equation (7) is NP-hard as it is a non-convex optimization problem. Suboptimal solutions to this problem can be found by matching pursuit or orthogonal matching pursuit. Such a problem can be regularized by replacing the $l_0$ norm in equation (7) with the $l_1$ norm as follows

$$\hat{s}\triangleq \arg \underset{\mathit{s}}{min}\parallel \mathit{s}{\parallel }_1\mathrm{s}.\mathrm{t}.\mathit{x}=\mathbf{\Psi }\mathit{s}$$ (8)

It can be shown ¹⁷ that if certain conditions are satisfied, the solution of l₀-based problem is equivalent to equation (8), while the latter can be efficiently solved by convex optimization such as the l₁/l₂-based algorithm. ¹⁸

Similarly, the corresponding classification deviations $\{{\sigma }_j^2\}$ to the obtained $\hat{s}$ can be computed using equation (3) and the classification can be done using equation (4).

Robust sparse coding

In Yang et al., ¹⁶ a robust representation–based classification approach was proposed. Denote the residual of signal representation as

$$\mathit{\xi }\triangleq \mathit{x}-\mathbf{\Psi }\mathit{s}$$

Assume that $\xi (1),\xi (2),\dots ,\xi (N)$ are of an independent identity distribution with the probability density function $f(\mathit{\xi })=\underset{n=1}{\overset{N}{\Pi }}{f}_0({\xi }_n)$ . Maximizing this likelihood function is equivalent to minimizing the objective function

$$F\left(\mathit{\xi }\right)\triangleq -\ln \left[f\left(\mathit{\xi }\right)\right]=-\sum _{n=1}^N\ln \left[f_0\left({\xi }_n\right)\right]\triangleq \sum _{n=1}^N{\rho }_0\left({\xi }_n\right)$$ (9)

The optimized sparse vector can be solved by

$$\hat{\mathit{s}}\triangleq \arg \underset{\mathit{s}}{min}F\left(\mathit{\xi }\right)\mathrm{s}.\mathrm{t}.||\mathit{s}|{|}_0\le \kappa$$ (10)

As it is a non-convex problem, Yang et al. ¹⁶ proposed an iterative method which is described as follows.

Define

$$\tilde{F}\left(\mathit{\xi }\right)\triangleq F\left({\mathit{\xi }}_k\right)+{(\mathit{\xi }-{\mathit{\xi }}_k)}^{T}F\prime \left({\mathit{\xi }}_k\right)+\frac{1}{2}{(\mathit{\xi }-{\mathit{\xi }}_k)}^T{\mathit{R}}_k(\mathit{\xi }-{\mathit{\xi }}_k)$$ (11)

where $F\prime ({\mathit{\xi }}_k)\triangleq \mathrm{d}F(\mathit{\xi })/\mathrm{d}\mathit{\xi }{|}_{\mathit{\xi }={\mathit{\xi }}_k}$ , ${\mathit{R}}_k$ is a diagonal matrix, and equation (11) can be viewed as the first-order Taylor approximation of $F(\mathit{\xi })$ evaluated at $\mathit{\xi }={\mathit{\xi }}_k$ . Both $\tilde{F}(\mathit{\xi })$ and $F(\mathit{\xi })$ reach the minimum at $\mathit{\xi }=0$ . It can be shown that

$${\mathit{R}}_k(n,n)=\frac{F\prime \left({\mathit{\xi }}_k\right)\left(n\right)}{{\mathit{\xi }}_k\left(n\right)},\forall n$$ (12)

and hence

$$\tilde{F}\left(\mathit{\xi }\right)=\frac{1}{2}{\mathit{\xi }}^T{\mathit{R}}_k\mathit{\xi }+F\left({\mathit{\xi }}_k\right)-{\xi }_k^{T}F\prime \left({\mathit{\xi }}_k\right)+\frac{1}{2}{\mathit{\xi }}_k^T{\mathit{R}}_k{\mathit{\xi }}_k$$

Replace $\tilde{F}(\mathit{\xi })$ with $F(\mathit{\xi })$ , then the solution to the corresponding equation (10) is equivalent to

$${\hat{\mathit{s}}}^{(k)}\triangleq \arg \underset{\mathit{s}}{min}||{\mathit{W}}_k(\mathit{x}-\mathbf{\Psi }\mathit{s})|{|}_2^2\mathrm{s}.\mathrm{t}.||\mathit{s}|{|}_0\le \kappa$$ (13)

where ${\mathit{R}}_k={\mathit{W}}_k^2$ .

$\mathit{W}$ can be updated with equation (12) using ${\mathit{\xi }}_{k+1}=\mathit{x}-\mathbf{\Psi }{\hat{\mathit{s}}}^{(\mathbf{k})}$ . Let us define $\hat{\mathit{W}}\triangleq \underset{k\to +\infty }{lim}{\mathit{W}}_k$ , $\hat{\mathit{s}}\triangleq \underset{k\to +\infty }{lim}{\hat{\mathit{s}}}^{(k)}$ . Here, ${\sigma }_j^2$ , instead of using equation (3), is evaluated with

$${\sigma }_j^2\triangleq \parallel \hat{\mathit{W}}(\mathit{x}-{\mathbf{\Psi }}_j{\hat{\mathit{s}}}_j){\parallel }_2^2,\forall j$$ (14)

and $\mathit{x}$ is then classified using equation (4).

To reduce the complexity in updating $\mathit{W}$ with equation (12), the following updating was proposed in Yang et al. ¹⁶

$${\mathit{W}}_k(n,n)\triangleq \frac{{\mathrm{e}}^{\mu \delta -\mu {\xi }_k^2(n)}}{1+{\mathrm{e}}^{\mu \delta -\mu {\xi }_k^2(n)}},\forall n$$ (15)

where $\mu$ and $\delta$ are two positive constants and ${\mathit{\xi }}_0\triangleq \mathit{x}-\overline{\mathit{x}}$ , where $\overline{\mathit{x}}$ is the mean of all training samples.

Nonnegative coefficient sparse representation–based classification

The nonnegative coefficient sparse representation–based classification is a constrained SRC, formulated with

$${\hat{\mathit{s}}}_j\triangleq \arg \underset{\mathit{s}}{min}||\mathit{s}|{|}_0\mathrm{s}.\mathrm{t}.\mathit{x}={\mathbf{\Psi }}_j\mathit{s},\mathit{s}\ge 0,\forall j$$ (16)

As understood, this is a non-convex problem and is hard to be solved. However, when the nonnegative sparse solution of equation (16) exists and is unique, it is equivalent to nonnegative linear regression. ¹⁹ Therefore, equation (16) is alternated to

$${\hat{\mathit{s}}}_j\triangleq \arg \underset{{\mathit{s}}_j}{min}\parallel \mathit{x}-{\mathbf{\Psi }}_j{\mathit{s}}_j{\parallel }_2^2\mathrm{s}.\mathrm{t}.{\mathit{s}}_j\ge 0,\forall j$$ (17)

which can be solved by the nonnegative least squares function lsqnonneg.m in MATLAB. ²⁰

The classification deviations $\{{\sigma }_j^2\}$ and the classification are the same as the LRC presented before.

Problem formulation

It should be pointed out that all the listed representation–based classification approaches work well only for the constrained environment. The environment in real life is usually unconstrained, and hence a pre-process such as frontalization is definitely needed. Also, the RSC algorithm presented above is very slow due to the step of updating the weighting $\mathit{W}$ . Therefore, a simplification is required. As there exists a lot of redundancy in a raw face image, which affects the recognition, it is desired to use the important features of the image for recognition. Therefore, we will propose a novel FR scheme that, besides a frontalization stage, consists of a local binary pattern (LBP) feature extraction, compressive sensing-based dimension reduction, and classification using robust nonnegative coefficient SRC. Such a system is referred as CS-RLNSRC.

A novel framework for FR

Figure 1 depicts the proposed FR scheme. Such a system consists of four main modules that will be described in the following subsections.

Figure 1.

The proposed CS-RLNSRC face recognition system.

Image pre-process

In this stage, we just concentrate on the process of face frontalization. The method used here is similar to that proposed by Hassner et al. ¹³ The basic idea is to extract face features that are used to match a 3D face mode and then frontalize the matched model to get the front face. See Figures 2 and 3, which show the six original faces (of the same person) and those frontalized using the process, respectively. In the traditional representation–based classification, one would put many different unfrontalized face images into the dictionary in order to deal with the unconstrained environment. Thus, the frontalization process can not only handle complicated environments but also avoid increasing the size of the dictionary as it contains the front face images only.

Figure 2.

Six images from LFW data set.

Figure 3.

The six frontalized face images.

Feature extraction

After frontalization, the images are processed using a downsampling LBP algorithm to reduce the illumination and misalignment effect. ²¹ Let ${\mathbf{Y}}_{\mathrm{DLBP}}$ denote the matrix representing an image that is the output of this algorithm. An SLMD is applied to eliminate the face deformation in $Y_{\mathrm{DLBP}}$

$$\begin{array}{c}\mathit{Y}\triangleq \arg \underset{\tilde{\mathit{Y}},\mathit{E}}{min}\{\mathrm{rank}\left[\tilde{\mathit{Y}}\right]+\parallel \mathit{E}{\parallel }_0\}\hfill \\ \mathrm{s}.\mathrm{t}.{\mathit{Y}}_{\mathrm{DLBP}}=\tilde{\mathit{Y}}+\mathit{E}\hfill \end{array}$$

where $\mathit{E}$ is the deformable residual and $\mathit{Y}$ is the ADLBP feature data matrix of the input face image.

The augmented Lagrange multiplier method can be used in the SLMD process. ²²

Dimension reduction

The obtained feature matrix $\mathit{Y}$ is re-arranged into a vector $\mathbf{y}\triangleq \mathrm{vec}[\mathit{Y}]\in {\Re }^{M\times 1}$ by stacking all the column vectors of $\mathit{Y}$ . As M is usually very big, recognition directly using $\mathit{y}$ is sometimes impossible. Therefore, a dimension reduction is needed. This can be achieved with a projection

$$\mathit{x}=\mathbf{\Phi }\mathit{y}$$ (18)

where $\mathbf{\Phi }\in {\Re }^{N\times \mathbf{M}}$ is called projection matrix.

The recognition can then be carried out much efficiently using $\mathbf{x}$ . Classically, a random projection matrix is adopted. Haupt et al. ²³ gave the following result in evaluating the performance of a classifier using a random projection.

Theorem 1

Let ${\mathcal{S}}_{\mathit{y}}\triangleq \{{\mathit{y}}_l{\}}_{l=1}^L$ be a set of L samples with ${\mathit{y}}_l\in {\Re }^{M\times 1}$ and $||{\mathit{y}}_l|{|}_2^2=1,\forall l$ . Assume that the elements of $\mathbf{\Phi }\in {\Re }^{N\times \mathbf{M}}$ are of i.i.d. with $\mathcal{N}(0,1/N)$ and

$${\mathit{x}}_l=\mathbf{\Phi }{\mathit{y}}_l+{\mathit{e}}_l,\forall l$$ (19)

where ${\mathit{e}}_l$ is the noise in the measurement domain with the assumption that all the elements of ${\mathit{e}}_l$ are i.i.d. following $\mathcal{N}(0,1/N)$ . Let ${\hat{\mathit{y}}}_l$ be the estimate of ${\mathit{y}}_l$ , obtained with

$${\hat{\mathit{y}}}_l\triangleq \arg \underset{\mathit{y}\in {\mathcal{S}}_{\mathit{y}}}{min}||{\mathit{x}}_l-\mathbf{\Phi }\mathit{y}|{|}_2^2$$ (20)

Then, the false acceptance rate (FAR) probability bound of this estimation is given by

$$\mathrm{Pr}({\hat{\mathit{y}}}_l\ne {\mathit{y}}_l)\le (L-1){(1+\frac{{\rho }_{\min }}{4{\sigma }^{2}M})}^{-N/2}$$ (21)

where ${\rho }_{\min }\triangleq \underset{k\ne l}{min}\{{\rho }_E({\mathbf{y}}_k,{\mathbf{y}}_l)\}$ .

In the representation-based classification framework, the ADLBP feature signal $\mathbf{y}$ of an image is considered to be of form

$$\mathit{y}=\mathit{As}$$

where $\mathit{s}\in {\Re }^{L\times 1}$ is the coefficient vector and $\mathit{A}\in {\Re }^{M\times L}$ is the dictionary in ADLBP feature domain. Combining it with equation (18), one has

$$\mathit{x}=\mathbf{\Phi }\mathit{As}\triangleq \mathbf{\Psi }\mathit{s}$$ (22)

where $\mathbf{\Psi }=\mathbf{\Phi }\mathit{A}$ is the dictionary of the images in the measurement domain with $\mathit{A}$ formed using the ADLBP feature vectors of the training face images.

In the classical SRC-based FR framework, the signals are the original (front) face images, while in the proposed FR scheme the signals are the compressed version $\mathit{x}$ of the ADLBP feature vectors $\mathit{y}$ of these face images, obeying equation (22).

The compressive sensing theory suggests that an optimized projection/sensing matrix can outperform significantly the random one in terms of keeping the information contained in the high-dimensional signals and enhancing the recovery accuracy.^24,25 The problem of optimizing projection matrix was first proposed in Elad ²⁴ and a large class of existing algorithms is of the following formulation ²⁵

$$\underset{min}{\mathbf{\Phi }}\parallel \mathit{G}-{\mathit{G}}_t{\parallel }_F^2\mathrm{s}.\mathrm{t}.\mathit{G}={\mathbf{\Psi }}^T\mathbf{\Psi }$$ (23)

where $\parallel ·{\parallel }_F$ is the Frobenius norm, $\mathit{G}$ is the Gram matrix of the dictionary defined in (equation (22)) $\mathbf{\Psi }\triangleq \mathbf{\Phi }\mathit{A}$ with $\mathit{A}$ the ADLBP domain dictionary assumed to be given, and ${\mathit{G}}_t$ is a target Gram matrix. This formulation is based on the argument that reducing the mutual coherence of the equivalent dictionary can enhance the signal recovery accuracy in a compression-oriented compressive sensing system. ²⁵ One algorithm for designing the optimal $\mathbf{\Phi }$ differs from another mainly in the choice of ${\mathit{G}}_t$ that belongs to a set of symmetric matrices having its diagonal elements all equal to one and the others absolutely smaller than one.

Denote ${\mathit{G}}_A$ as the Gram of the ADLBP dictionary $\mathit{A}=\left[\begin{array}{ccccc}{\mathit{A}}_1\hfill & \cdots \hfill & {\mathit{A}}_j\hfill & \cdots \hfill & {\mathit{A}}_J\hfill \end{array}\right]$ , that is

$$\begin{array}{c}{\mathit{G}}_{\mathit{A}}\triangleq {\mathit{A}}^T\mathit{A}\hfill \\ =\left[\begin{array}{cccc}{\overline{G}}_{11}\hfill & {\overline{G}}_{12}\hfill & \cdots \hfill & {\overline{G}}_{1J}\hfill \\ \multicolumn{1}{c}{{\overline{G}}_{21}}& {\overline{G}}_{22}\hfill & \cdots \hfill & {\overline{G}}_{2J}\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{{\overline{G}}_{J1}}& {\overline{G}}_{J2}\hfill & \cdots \hfill & {\overline{G}}_{\mathrm{JJ}}\hfill \end{array}\right]\hfill \end{array}$$

where ${\overline{G}}_{\mathrm{ij}}\triangleq {\mathit{A}}_i^T{\mathit{A}}_j$ .

In our proposed system, the target Gram ${\mathit{G}}_t$ is chosen as

$${\mathit{G}}_t=\mathrm{\Delta }·{\mathit{G}}_{\mathit{A}}$$ (24)

where

$$\mathrm{\Delta }\triangleq \left[\begin{array}{cccc}{\mathrm{\Delta }}_{11}\hfill & {\mathrm{\Delta }}_{12}\hfill & \cdots \hfill & {\mathrm{\Delta }}_{1J}\hfill \\ \multicolumn{1}{c}{{\mathrm{\Delta }}_{21}}& {\mathrm{\Delta }}_{22}\hfill & \cdots \hfill & {\mathrm{\Delta }}_{2J}\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{{\mathrm{\Delta }}_{J1}}& {\mathrm{\Delta }}_{J2}\hfill & \cdots \hfill & {\mathrm{\Delta }}_{\mathrm{JJ}}\hfill \end{array}\right]$$

with ${\mathrm{\Delta }}_{\mathrm{ij}}$ defined as

$${\mathrm{\Delta }}_{\mathrm{ij}}(m,n)=\{\begin{array}{cc}1-\eta ,\hfill & i\ne j\hfill \\ \multicolumn{1}{c}{1+\eta ,}& i=j,m\ne n\hfill \\ \multicolumn{1}{c}{1,}& i=j,m=n\hfill \end{array}$$ (25)

where $0\le \eta \le 1$ is adopted to adjust the inter-class and intra-class correlation.

The motivation for such a choice of target Gram is explained below. We note that with the same partition as ${\mathit{G}}_t$ , the sub-matrix ${\mathbf{G}}_{\mathrm{ij}}$ of the Gram of $\mathbf{\Psi }$ with $i\ne j$ represents the cross-correlation between two different classes of subjects, while ${\mathit{G}}_{\mathrm{jj}}$ reflects the auto-correlation between the jth class. From the viewpoint of classification, it is desired to design the projection matrix $\mathbf{\Phi }$ such that the elements in ${\mathit{G}}_{\mathrm{ij}}$ are as big as possible and those in ${\mathit{G}}_{\mathrm{jj}}$ are close to zero. By doing so, the discrimination and hence the classification are expected to enhance.

With ${\mathit{G}}_t$ given above, the optimal sensing matrix design problem (equation (23)) can be solved using the algorithm derived in Yu et al. ²⁶ In fact, let

$$\mathit{A}={\mathit{U}}_A\left[\begin{array}{cc}{\Sigma }_A\hfill & 0\hfill \end{array}\right]{\mathit{V}}_A^T$$

be a singular value decomposition of $\mathit{A}$ with both ${\mathit{U}}_A$ and ${\mathit{V}}_A$ an orthogonal matrix of dimensions M and L, respectively, and

$${\Sigma }_A=\left[\begin{array}{cc}{\Sigma }_{11}\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& {\Sigma }_{22}\hfill \end{array}\right]\in {\Re }^{M\times M}$$

the diagonal matrix with ${\Sigma }_A(n,n)\ge {\Sigma }_A(n+1,n+1)>$ $0,\forall n$ , where the sub-matrix ${\Sigma }_{11}\in {\Re }^{N\times N}$ is assumed to be non-singular. The solution of equation (23) is given by

$$\mathbf{\Phi }=\mathit{U}\left[\begin{array}{cc}{\mathbf{\Sigma }}_{11}\hfill & 0\hfill \end{array}\right]{\left[\begin{array}{cc}{\mathit{V}}_{11}\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& {\mathit{V}}_{22}\hfill \end{array}\right]}^T{\mathbf{\Sigma }}_{\mathbf{A}}^{-1}{\mathit{U}}_{\mathbf{A}}^T$$ (26)

where $\mathit{U}\in {\Re }^{N\times N}$ , ${\mathit{V}}_{11}\in {\Re }^{N\times N}$ , and ${\mathit{V}}_{22}\in {\Re }^{(M-N)\times (M-N)}$ are all arbitrary orthogonal matrices.

When ${\mathit{V}}_{11}$ and ${\mathit{V}}_{22}$ are set to the identity matrix, a particular $\mathbf{\Phi }$ and the corresponding dictionary $\mathbf{\Psi }$ are given by

$$\mathbf{\Phi }=\mathit{U}\left[\begin{array}{cc}{\mathit{I}}_N\hfill & 0\hfill \end{array}\right]{\mathit{U}}_{\mathbf{A}}^T,\mathbf{\Psi }=\mathit{U}\left[\begin{array}{cc}{\mathbf{\Sigma }}_{11}\hfill & 0\hfill \end{array}\right]{\mathit{V}}_{\mathbf{A}}^T$$ (27)

With $\mathbf{\Phi }$ obtained, the ADLBP feature vector $\mathbf{y}$ of a query face image can be compressed using equation (18), yielding a low dimension vector $\mathit{x}$ . The FR can be carried out with the obtained $\mathbf{x}$ and dictionary $\mathbf{\Psi }$ using one of the representation-based classification algorithms discussed in the “Related works and problem formulation” section. In the next subsection, we will derive an improved RSC with the nonnegative coefficient constraint embedded.

An improved RSC

Recall the RSC in the “Robust sparse coding” section. As mentioned before, the procedure used in such an algorithm for updating the weighting factors is time-consuming. This prevents the algorithm from real-time application.

In the proposed CS-RLNSRC FR system, the coefficient vector for the jth class is obtained from the following nonnegative coefficient constrained minimization

$${\hat{\mathit{s}}}_j\triangleq \arg \underset{\mathit{s}}{min}\parallel \mathit{x}-{\mathbf{\Psi }}_j\mathit{s}{\parallel }_2^2\mathrm{s}.\mathrm{t}.\mathit{s}\ge 0,\forall j$$ (28)

which can be solved by the nonnegative least squares function lsqnonneg.m in MATLAB. The representation residual, as defined before, is given by

$${\mathit{\xi }}_j=\mathit{x}-\mathbf{\Psi }{\hat{\mathit{s}}}_j,\forall j$$

The classical hypothesis of noise interference follows the Gauss distribution. The actual distribution is not always the case. In human eyes, we usually focus on the most of face feature points that match, ignoring those mismatched. Based on this observation, we propose a much simplified procedure to update the weighting matrix $\mathit{W}$ . Instead of directly using $\{||{\mathit{\xi }}_j|{|}_2^2\}$ in the classification, a weighted version is utilized.

Denoting ${\eta }_j$ as the $K_0\mathrm{th}$ ascending ordered element of $\{|{\mathit{\xi }}_j(n)|\}$ , we define the weighted (diagonal) matrix for ${\mathit{\xi }}_j$

$${\mathit{W}}_j(n,n)=\{\begin{array}{cc}1,\hfill & |{\mathit{\xi }}_j\left(n\right)|\le {\eta }_j\hfill \\ \multicolumn{1}{c}{0,}& \mathrm{otherwise}\hfill \end{array},\forall j$$ (29)

The query face image is then classified with

$$\hat{j}\triangleq \arg \underset{j}{min}\{||{\mathit{W}}_j{\mathit{\xi }}_j|{|}_2^2\}$$ (30)

We use Alg _RLNSRC to denote the classification algorithm specified in equations (28) to (30).

Remark 3.1

• As seen from equation (29), the weighting matrix can be easily obtained without using an iterative procedure and statistical information of the residuals. Experiments show that such a weighting strategy is very effective for enhancing recognition performance.

• We note that the choice of weighting factors given by equation (29) is intended to keep the $K_0$ smallest entries in the residual vector ${\mathit{\xi }}_j$ for classification. A more general choice for the weighting factors is

$$\mathit{W}(n,n)={\lambda }_n,\mathrm{s}.\mathrm{t}.\sum _{n=1}^N{\lambda }_n=1,{\lambda }_n\ge 0,\forall n$$

Clearly, equation (29) is equivalent to the above for the case where ${\lambda }_n=1/K_0$ for the $K_0$ smallest entries in the residual vector ${\mathit{\xi }}_j$ , and ${\lambda }_n=0$ otherwise.

In many situations, a query face image may not belong to any of the J classes. Therefore, it does not make much sense to classify this image. We use the relative residual index (RRI) to determine if the query image belongs to the J classes or not for the open-universe test

$$\mathrm{RRI}\left(\hat{j}\right)\triangleq \frac{||{\mathit{W}}_{\hat{j}}{\mathit{\xi }}_{\hat{j}}|{|}_2^2}{\sum _{j\ne \hat{j}}||{\mathit{W}}_j{\mathit{\xi }}_j|{|}_2^2}$$ (31)

Since $||{\mathit{W}}_{\hat{j}}{\mathit{\xi }}_{\hat{j}}|{|}_2^2$ is the smallest among the J elements of $\{||{\mathit{W}}_j{\mathit{\xi }}_j|{|}_2^2\}$ (see equation (30)), we have $0\le \mathrm{RRI}(\hat{j})<1$ . Now, let $\tau$ be the RRI threshold. The obtained $\hat{j}$ with equation (30) is accepted (i.e. to accept the query image belonging to the class) if and only if $\mathrm{RRI}(\hat{j})<\tau$ .

Suppose that we have K query images and $K_0$ is the number of images that are accepted for classification. The recall rate is then defined as

$$r_{\tau }\triangleq \frac{K_0}{K}$$

Furthermore, let $k_0$ be the number of images among the $K_0$ accepted ones, which are correctly classified. To evaluate the actual performance of the recognition, we use the following index to represent the actual recognition rate

$${\gamma }_{\tau }\triangleq \frac{k_0}{K_0}$$

Obviously, both $r_{\tau }$ and ${\gamma }_{\tau }$ are function of the RRI threshold $\tau$ . Before turning to the “Experiment results” section, we make the following remarks.

Remark 3.2

Lack of training samples is an important factor affecting the performance of representation-based classification techniques. It has been noted that in face images, the facial features have symmetrical attributes. This means that if we have one face image, we can get another with its mirror face image, achieved using MATLAB command fliplr.m. By doing so, the size of training samples can be doubled, which is applied to the dictionary $\mathbf{\Psi }$ . In our system, this is implemented.

Experiment results

In this section, we will examine the performance of the proposed CS-RLNSRC system for FR and compare it with some of the existing FR systems. It contains two portions. In the first one, several existing FR systems, which are all similar to our proposed one, using the representation-based classification are implemented for comparison, while in the second portion, three FR systems implemented with different strategies are used for the comparison.

Labeled faces in the wild (LFW) are used in our experiments. The LFW data set contains 13,323 web images of 5749 celebrities. A subset of the LFW data set, having $J=158$ persons, is taken for experiments, in which $L_j=10$ images with various poses of each person are used for training.

Portion I—RC-based FR systems

It should be pointed out that in this subsection, all the systems have the same structure, containing four modules among which the first three are the same as those in our proposed system. This means that one system differs from another just in the way how the classification is done. We will consider four representation-based classification algorithms: Alg _RLNSRC , defined in the “An improved RSC” section, denotes the classification algorithm used in our proposed CS-RLNSRC system, while Alg _SRC , Alg _LRC , and Alg _WL denote the classification algorithms reported in Wright et al., ²⁷ Naseem et al., ²⁸ and Mairal et al., ²⁹ respectively.

Six images of a person are displayed in Figure 2. These images are pre-processed for frontalization, and the resultant images $(90\times 90)$ are shown in Figure 3, where the fifth one failed in being frontalized.

Figure 4 displays the evolution of the first frontalized image in Figure 3 from frontal face to ADLBP featured image which is $32\times 32$ , leading to a ADLBP vector $\mathbf{y}\in {\Re }^{M\times 1}$ with $M=1024$ .

Figure 4.

ADLBP extraction process: (a) frontalized image, (b) LBP, (c) DLBP, and (d) ADLBP.

Set $N=200$ . With the training images and the approach to optimal sensing matrix design described in the “Dimension reduction” section, we can obtain a sensing matrix $\mathbf{\Phi }\in {\Re }^{N\times \mathbf{M}}$ that projects the ADLBP vector $\mathbf{y}$ into $\mathit{x}=\mathbf{\Phi }\mathit{y}$ . Such kinds of signals/vectors are used for classification using the four algorithms: Alg _RLNSRC , Alg _SRC , Alg _LRC , and Alg _WL .

The effect of parameter $\eta$

We note that in designing the sensing matrix, one has to choose the parameter $\eta$ . Figure 5 shows the relationship between the recognition rate $r_{\mathrm{ec}}$ and the parameter $\eta$ .

Figure 5.

The parameter $\eta$ versus the recognition rate $r_{\mathrm{ec}}$ .

As it is shown in Figure 5 that the recognition rate can be increased with a proper choice of $\eta$ . It should be pointed out that the optimal $\eta$ is dependent on the training samples and has to be determined experimentally. In the sequel, $\eta$ is fixed to 0.16.

Effect of the number of training samples $L_j$

Table 1 displays the effect of $L_j$ on the recognition rate of the FR systems implemented using different representation-based classification algorithms.

Table 1.

The number $L_j$ versus recognition rate $r_{\mathrm{ec}}$ (%).

$L_j$	5	7	9
$\mathbf{Al}{\mathbf{g}}_{\mathrm{WL}}$	55.80	67.83	72.40
$\mathbf{Al}{\mathbf{g}}_{\mathrm{SRC}}$	74.33	77.34	87.60
$\mathbf{Al}{\mathbf{g}}_{\mathrm{LRC}}$	73.30	78.52	88.60
$\mathbf{Al}{\mathbf{g}}_{\mathrm{RLNSRC}}$	78.61	87.94	92.25

The values in bold are the best one in each column of the table.

As displayed in Table 1, increasing the number of training samples can enhance the recognition rate. This is coincident with equation (21). The results also indicate that our proposed algorithm outperforms the others.

Effect of data compression ratio M/N

We fix $M=1024$ and variate N. Figure 6 depicts the effect of N on the recognition rate $r_{\mathrm{ec}}$ .

Figure 6.

Relationship between N and the recognition rate $r_{\mathrm{ec}}$ .

One observes that for each algorithm, the recognition rate increases with N. This is expected and is coincident with the theoretical result given in equation (21). Once again, our proposed algorithm is the best among all the four algorithms.

Effect of features extraction

Let ${\mathit{Y}}_0$ denote the matrix representing a raw face image. As defined before, $\mathit{Y}$ is the corresponding ADLBP features. Here, we will examine how the feature selection affects the classification. Table 2 demonstrates how the recognition rate changes with the choice of features extraction for the four classification algorithms, where ${\mathit{Y}}_{\mathrm{LBP}}$ and ${\mathit{Y}}_{\mathrm{DLBP}}$ are the LBP and downsampling LBP features of ${\mathit{Y}}_0$ , respectively.

Table 2.

Feature selection versus recognition rate $r_{\mathrm{ec}}$ (%).

Feature	${\mathit{Y}}_0$	${\mathit{Y}}_{\mathrm{LBP}}$	${\mathit{Y}}_{\mathrm{DLBP}}$	$\mathit{Y}$
$\mathbf{Al}{\mathbf{g}}_{\mathrm{WL}}$	50.29	53.7	70.56	87.24
$\mathbf{Al}{\mathbf{g}}_{\mathrm{SRC}}$	58.89	60.87	76.63	90.60
$\mathbf{Al}{\mathbf{g}}_{\mathrm{LRC}}$	59.04	61.01	77.48	91.40
$\mathbf{Al}{\mathbf{g}}_{\mathrm{RLNSRC}}$	59.70	68.33	80.17	94.74

One can see that for each algorithm, the ADLBP feature is always better than any of the others in terms of recognition rate. The price paid for that is an increase of computation complexity.

Open-universe experiment

We add 8909 face images as distractors to the set of testing images, and then run the four FR systems. For a given $\tau$ , we can calculate $r_{\tau }$ and ${\gamma }_{\tau }$ . The relationship between the two is depicted in Figure 7, where $N=200$ is used.

Figure 7.

Relationship between the actual recognition rate ${\gamma }_{\tau }$ and the recall rate $r_{\tau }$ .

As seen, the proposed Alg _RLNSRC outperforms all the others and ${\gamma }_{\tau }$ can reach $95\%$ with a recall rate $r_{\tau }=70\%$ . Experiments show that when N increases, ${\gamma }_{\tau }$ augments for the same $r_{\tau }$ . With $N=400$ and $r_{\tau }=80\%$ , ${\gamma }_{\tau }=95\%$ (see Figure 7).

Portion II—comparison with differently structured systems

In the previous subsection, we compare our proposed FR systems with three systems that share the same structure as ours, and the main difference is that each system uses a different representation-based classification algorithm.

In this subsection, we will compare our proposed system with three other FR systems which use significantly different strategies from ours. Here, we use Alg _RLNSRC to denote our proposed system; Alg _G the system that utilizes an efficient classifier for large-scale images; ³⁰ Alg _Hassner the system that uses a hybrid algorithm which achieved the highest score on the LFW challenge in the Image-Restricted, Label-Free Outside Data category; ¹³ and Alg _Chen the system that adopts the learned high-dimensional features for the classification. ³¹

As it is indicated in Table 3, the 3D-based frontalization algorithm, used in Alg _RLNSRC and Alg _Hassner , can improve the recognition performance greatly, compared with Alg _G . A comparable result is achieved by Alg _Chen . Our proposed algorithm surprisingly outperforms Alg _Chen on the database used in the experiments.

Table 3.

Recognition rate $r_{\mathrm{ec}}$ (%) versus FR systems.

System	$r_{\mathrm{ec}}$
Alg G	84.13
Alg Hassner	91.65
Alg Chen	93.18
Alg RLNSRC	94.74

FR: face recognition.

Conclusion

In this article, a novel framework has been proposed, in which a 3D-based frontalization strategy is adopted as a pre-process and the ADLBP features of the frontalized images are employed for recognition. In addition, an optimized projection matrix is designed to reduce the implementation complexity and an improved RSC algorithm has been derived for classification using the lower dimensional measurements. Experimental results on open and closed universe of LFW data set demonstrate the effectiveness of the proposed approaches and show that our proposed system outperforms those RC-based FR systems as well as the three FR systems that have a significantly different structure from ours.

In order to make the proposed FR system be used in a real-time application, more efficient algorithm should be developed for the ADLBP feature extraction. The key to the success of the proposed FR system is to prevent overfitting problem by applying nonnegative coefficient constraint. Similar phenomenon has also been noted in DL.^32,33 DL has become a powerful tool and been used in many areas. How to embed our approach into the DL context will be another direction for future investigation.