Support vector regression model with variant tolerance

Abstract

Most works on Support Vector Regression (SVR) focus on kernel or loss functions, with the corresponding support vectors obtained using a fixed-radius $\epsilon$ -tube, affording good predictive performance on datasets. However, the fixed radius limitation prevents the adaptive selection of support vectors according to the data distribution characteristics, compromising the performance of the SVR-based methods. Therefore, this study proposes an “Alterable ${\epsilon }_i$ -Support Vector Regression” ( $A{\epsilon }_i$ -SVR) model by applying a novel $\epsilon$ , named “Alterable ${\epsilon }_i$ ,” to the SVR model. Based on the data point sparsity at each location, the model solves the different ${\epsilon }_i$ at the corresponding position, and thus zoom-in or zoom-out the $\epsilon$ -tube by changing its radius. Such a variable $\epsilon$ -tube strategy diminishes noise and outliers in the dataset, enhancing the prediction performance of the $A{\epsilon }_i$ -SVR model. Therefore, we suggest a novel non-deterministic algorithm to iteratively solve the complex problem of optimizing ${\epsilon }_i$ associated with every location. Extensive experimental results demonstrate that our approach can improve the accuracy and stability on simulated and real data compared with the baseline methods.

Introduction

Support Vector Machines (SVM) is a powerful machine learning technique based on Statistical Learning and a promising tool to overcome the function approximation problem based on the Structural Risk Minimization (SRM) principle. As an advanced supervised learning algorithm, SVM conducts data classification by constructing a hyperplane ${\omega }^{T}x+b=0,\omega \in R^n,b\in R.$ After decades of development, it has reached a high level of maturity considering its theoretical support^1–5, and has been widely employed in various regression estimation tasks, such as Cancer prevention and treatment,^6–8 renewable energy exploitation prediction,^9–11 and financial time series data.^12–14 This regression scheme is termed Support Vector Regression (SVR) and performs better than current machine learning algorithms. SVR solves practical problems by introducing prediction equation $\hat{f}\left(x\right)={\omega }^{T}x+b,\omega \in R^n,b\in R$ , where the predictor predicts new observations in the feature space and is defined as $\hat{f}\left(x\right)={\omega }^T\varphi \left(x\right)+b$ , where $\varphi ()$ is a transformation function in the covariate space. By introducing SVR kernel functions, it presents an excellent nonlinear function regression capability and therefore has been successfully applied to nonlinear prediction systems.

Over the past few decades, various studies have focused on the kernel function $\kappa (x_i,x_j)=〈\varphi (x_i),\varphi (x_j)〉$ and loss function $L(y-\hat{f}\left(x\right))$ . Regarding the former, it aims to avoid directly computing high-dimensional inner products and improve the algorithm’s generalization performance.^15–17 The literature presents several kernel functions such as the random radial basis function (RRBF) kernel, ¹⁸ the composite wavelet kernel, ¹⁹ the Hermite orthogonal polynomial kernel, ²⁰ and the robust low-rank multiple kernels. ²¹ Through these kernel functions, the SVR model can transform the original data into the nonlinear finite-dimensional kernel space to avoid the “Dimensional Disaster” and improve classification accuracy and prediction.

Considering the conventional loss functions, these include Huber loss function-based SVR, ³ quadratic loss function-based $\mathrm{LS}$ -SVR, ²² Maximum Likelihood Optimal and Robust Support Vector Regression model, ²³ and $L_1$ -norm SVR. ²⁴

Motivated by $\epsilon$ -SVR, Pritam ²⁵ introduced an $\epsilon -\mathrm{PSVR}$ model utilizing an $\epsilon$ -penalty loss function that offers different penalty rates to the data points depending upon whether they lie inside or outside the $\epsilon$ -tube. Moreover, Gupta and Gupta ²⁶ proposed the asymmetric $v$ -twin support vector regression (Asy- $v$ -TSVR) that finds two non-parallel hyperplanes to construct the corresponding decision functions given two different $\epsilon$ -insensitive loss functions, which are the lower and upper bound functions. Besides, Balasundaram and Meena ²⁷ presented an $\epsilon$ -insensitive asymmetric Huber function-based $\epsilon$ -AHSVR model, where an $\epsilon$ -insensitive loss function is integrated with the Huber loss function to compensate for the high complexity of the latter. Cheng and Lu ²⁸ developed an $\epsilon$ -insensitive square loss function-based Bayesian $\epsilon$ -SVR model that adopts the minimizing structural risk principle through the $\epsilon$ -insensitive square loss function while provides point-by-point probability predictions and allows determining the optimal hyperparameters by maximizing the Bayesian model evidence.

Most innovations on loss functions rely on the $\epsilon$ -insensitive loss functions because, among all points in the training set, the $\epsilon$ -insensitive loss function ignores the points inside the $\epsilon$ -tube. Therefore, the prediction model is essentially determined by those support factors located at the edges or outside the tube, enhancing the generalization ability of $\epsilon$ -SVR. Here, the $\epsilon$ -insensitive loss function ¹ is given by:

$$L_{\epsilon }=\{\begin{array}{cc}0,\hfill & \mathrm{if}|y_i-f\left(x_i\right)|\le \epsilon ,\hfill \\ \multicolumn{1}{c}{|y_i-f\left(x_i\right)|-\epsilon ,}& \mathrm{otherwise}.\hfill \end{array}$$ (1)

where $\epsilon >$ 0. Figure 1 shows some import loss function. Employing $\epsilon$ in the $\epsilon$ -SVR enhances robustness to errors with the error tolerance corresponding to each position being $\epsilon$ . Such an approach allows the solution vector of the $\epsilon$ -SVR model to become sparse.

Figure 1.

Loss function corresponding to several different models.

However, these new loss functions only focus on the $\epsilon$ -tube radius value or optimizing the hyper-parameter to minimize the empirical risk. However, when complex datasets are involved, a single value of $\epsilon$ cannot obtain the support vectors at different locations of high-dimensional data. Indeed, an $\epsilon$ -tube with a fixed radius limits the choice of support vectors for each position, prohibiting it from utilizing the full information of the training set.

To address the above challenges, this work proposes the‘Alterable ${\epsilon }_i$ -Support Vector Regression’ ( $A{\epsilon }_i$ -SVR) based on an alterable $\epsilon$ -insensitive loss function. Similar to the standard $\epsilon$ -SVR, the $A{\epsilon }_i$ -SVR exploits the loss function computed by the $\mathrm{Alterable}$ ${\epsilon }_i$ to measure the empirical risk with a regularization term $\frac{1}{2}{w}^{T}w$ . This loss function is given by:

$$L_{\mathrm{A}\epsilon }=\{\begin{array}{cc}0,\hfill & \mathrm{if}|y_i-f\left(x_i\right)|\le {\epsilon }_i,\hfill \\ \multicolumn{1}{c}{|y_i-f\left(x_i\right)|-{\epsilon }_i,}& \mathrm{otherwise},\hfill \end{array}$$ (2)

where ${\epsilon }_i>$ 0, i = 1,2,…,n. The value of Alterable- ${\epsilon }_i$ depends on the distribution sparsity of the data points’ location, allowing $A{\epsilon }_i$ -SVR to capture support vectors for each position adaptively. Figure 2 illustrates proposed ε _i loss function with different value of ε.

Figure 2.

The proposed alterable ${\epsilon }_i$ loss function with different value of $\epsilon$ .

The $\epsilon$ -tube constraints on standard $\epsilon$ -SVR and $A{\epsilon }_i$ -SVR are $|y_i-({\omega }^T{x}_i+b)|\le \epsilon +{\xi }_1^i$ and $|y_i-({\omega }^T{x}_i+b)|\le {\epsilon }_i+{\xi }_1^i$ , respectively. Hence, this paper introduces the concept of adaptive ${\epsilon }_i$ , which affords $A{\epsilon }_i$ -SVR to exploit the information provided by the training set and thus adapt better to the data distribution characteristics and improve the prediction effect. Although the kernel function can map the data to a high-dimensional Hilbert space based on statistical learning and effectively dealing with high-dimensional data, the hyper-plane is not flexible enough to fit every point in the space when making predictions in the high-dimensional feature space. Therefore, the hyper-surface is obtained after training the $A{\epsilon }_i$ -SVR model on the training set, which affords a better prediction performance than the hyper-plane.

However, the hyper-plane’s overall distribution varies with the value of ${\epsilon }_i$ at each location, posing a large computational burden to choose and solve ${\epsilon }_i$ . Hence, to overcome these challenges, this paper suggests a novel non-deterministic algorithm for determining ${\epsilon }_i$ .

Specifically, the proposed SVR model improves the previously discussed methods, including the loss function (standard $\epsilon$ -SVR expression) and the kernel function (RBF kernel function) by introducing many penalty values ${\epsilon }_i$ through an iterative process, providing a variable $\epsilon$ -tube. Opposing the standard $\epsilon$ -tube, the variable $\epsilon$ -tube based $A{\epsilon }_i$ -SVR model allows a different penalty depending on the data distribution that enhances our method’s flexibility in removing outliers, which in turn mitigates the noise interference on the prediction curve, and finally achieves better prediction results.

The remainder of this paper is organized as follows. Section 2 compares the proposed $A{\epsilon }_i$ -SVR model against the $\epsilon$ -SVR-based model for both linear and nonlinear scenarios. Section 3 derives $A{\epsilon }_i$ -SVR, and Section 4 introduces a novel iterative fluctuation self-selection algorithm. Sections 5 and 6 present and discuss the key comparisons based on real-world benchmarks, and finally, Section 7 concludes this work and provides some future research directions.

Theory of alterable ${\mathbf{\epsilon }}_{\mathit{i}}$ -support vector regression

This section shortly reviews the standard $\epsilon$ -Support vector regression models and compares them with the Alterable $A{\epsilon }_i$ -SVR in both linear and nonlinear scenarios. Given a training set $T=\{(x_i,y_i);x_i\in R^n,y_i\in R,i=1,2,\dots ,n\},$ C is a pre-specified value, b is the offset item, and ${\xi }^{+}={({\xi }_1^{+},{\xi }_2^{+}\cdots {\xi }_m^{+})}^T$ and ${\xi }^{-}={({\xi }_1^{-},{\xi }_2^{-}\cdots {\xi }_m^{-})}^T$ are slack variables that represent the upper and lower constraints on the system outputs.

Linear support vector regression model

To estimate the linear function $f\left(x\right)={\omega }^{T}x+b$ , where $\omega$ is the weight vector and b is the offset term, the standard $\epsilon$ -Support Vector Regression minimizes:

$$\underset{(\omega ,{\xi }^{+},{\xi }^{-})}{min}\frac{1}{2}\parallel \omega {\parallel }^2+C{\sum }_{i=1}^m({\xi }_i^{+}+{\xi }_i^{-}),$$

subject to

$$\begin{array}{c}{y}_i-({\omega }^T{x}_i+b)\le \epsilon +{\xi }_i^{+},(i=1,2,\dots ,m),\hfill \\ ({\omega }^T{x}_i+b)-y_i\le \epsilon +{\xi }_i^{-},(i=1,2,\dots ,m),\hfill \\ {\xi }_i^{+}\ge 0,{\xi }_i^{-}\ge 0,(i=1,2,\dots ,m),\hfill \end{array}$$ (3)

The $A{\epsilon }_i$ -SVR model minimizes

$$\underset{(\omega ,{\xi }^{+},{\xi }^{-})}{min}\frac{1}{2}\parallel \omega {\parallel }^2+C{\sum }_{i=1}^m({\xi }_i^{+}+{\xi }_i^{-}),$$

subject to

$$\begin{array}{c}{y}_i-({\omega }^T{x}_i+b)\le {\epsilon }_i+{\xi }_i^{+},(i=1,2,\dots ,m),\hfill \\ ({\omega }^T{x}_i+b)-y_i\le {\epsilon }_i+{\xi }_i^{-},(i=1,2,\dots ,m),\hfill \\ {\xi }_i^{+}\ge 0,{\xi }_i^{-}\ge 0,(i=1,2,\dots ,m).\hfill \end{array}$$ (4)

Figure 3 illustrates an intuitive geometric interpretation of the linear SVR. Normally, the standard $\epsilon$ -SVR model has a tube with a fixed $\epsilon$ radius, which, although avoids over-fitting, it cannot obtain all support vectors. Indeed, the fixed radius $\epsilon$ -tube provides the same tolerance for all data points, either closer or further away from the tube wall, enhancing the $\epsilon$ -SVR model’s insensitivity to outliers. To overcome this problem, we propose the Alterable- ${\epsilon }_i$ -SVR model, which, based on the training data set, captures the appropriate support vector machine for each position by iteratively obtaining the distinct $\epsilon$ values per position. When the location with a dense data point distribution is involved, the $A{\epsilon }_i$ -SVR model increases the number of support vectors at that location by shortening the $\epsilon$ -tube radius. On the contrary, at a location with a sparse data point distribution, our method increases the $\epsilon$ -tube radius, decreasing the number of support vectors. This way, the hyper-plane orientation and displacement can be changed to fit the data points better and exhibit better predictive performance.

Figure 3.

Comparison of linear support vector regression models: (a) $\epsilon$ -SVR (kernel = Linear) and (b) $A{\epsilon }_i$ -SVR (kernel = Linear).

Non-linear support vector regression model

To estimate the linear function $f\left(x\right)={\omega }^T\varphi \left(x\right)+b,$ where $\varphi \left(\right):R^n\to \mathcal{H}$ is a non-linear mapping from the input space to higher dimensional Hilbert space, the standard $\epsilon$ -Support Vector Regression minimizes

$$\underset{(\omega ,{\xi }^{+},{\xi }^{-})}{min}\frac{1}{2}\parallel \omega {\parallel }^2+C{\sum }_{i=1}^m({\xi }_i^{+}+{\xi }_i^{-}),$$

subject to

$$\begin{array}{c}{y}_i-({\omega }^T\varphi \left(x_i\right)+b)\le \epsilon +{\xi }_i^{+},(i=1,2,\dots ,m),\hfill \\ ({\omega }^T\varphi \left(x_i\right)+b)-y_i\le \epsilon +{\xi }_i^{-},(i=1,2,\dots ,m),\hfill \\ {\xi }_i^{+}\ge 0,{\xi }_i^{-}\ge 0,(i=1,2,\dots ,m),\hfill \end{array}$$ (5)

The $A{\epsilon }_i$ -SVR model minimizes

$$\underset{(\omega ,{\xi }^{+},{\xi }^{-})}{min}\frac{1}{2}\parallel \omega {\parallel }^2+C{\sum }_{i=1}^m({\xi }_i^{+}+{\xi }_i^{-}),$$

subject to

$$\begin{array}{c}{y}_i-({\omega }^T\varphi \left(x_i\right)+b)\le {\epsilon }_i+{\xi }_i^{+},(i=1,2,\dots ,m),\hfill \\ ({\omega }^T\varphi \left(x_i\right)+b)-y_i\le {\epsilon }_i+{\xi }_i^{-},(i=1,2,\dots ,m),\hfill \\ {\xi }_i^{+}\ge 0,{\xi }_i^{-}\ge 0,(i=1,2,\dots ,m).\hfill \end{array}$$ (6)

The geometric interpretation of the nonlinear SVR is presented in Figure 4(a), highlighting that $\epsilon$ -SVR performs well on some data points but poorly on others. This is probably because, in this model, all data points are treated equally and have the same tolerance due to the limitation of the fixed-radius $\epsilon$ -tube. Nevertheless, real data sets tend to be unevenly distributed, with both sparsely and densely distributed data points. Limiting the fixed radius prevents adaptively selecting the support vectors according to the distribution characteristics of the data, compromising the predictive performance of $\epsilon$ -SVR.

Figure 4.

Comparison of non-linear support vector regression models: (a) $\epsilon$ -SVR (kernel = RBF) and (b) $A{\epsilon }_i$ -SVR (kernel = RBF).

Considering $A{\epsilon }_i$ -SVR, in Figure 4(b), our model distinguishes the locations with sparse and dense data point distribution and calculates the distinct tolerances at the different positions so that the $\epsilon$ -tube radius changes depending on the location. Moreover, changing the $\epsilon$ -tube radius allows the prediction curve to be more flexible when dealing with datasets, thus producing better prediction results. It should be noted that this section presents only the schematic diagram of $A{\epsilon }_i$ -SVR, and its performance on fitting the real data set is evaluated in Section 5.

Solution of alterable ${\mathbf{\epsilon }}_{\mathit{i}}$ -support vector regression

Given a training set $T=(x_i,y_i):x_i\in R^n,y_i\in R,i=1,2,\dots ,m,\omega \in R^n$ is the weight vector, b is a real constant, and ${\xi }^{+}={({\xi }_1^{+},{\xi }_2^{+}\cdots {\xi }_m^{+})}^T$ , ${\xi }^{-}={({\xi }_1^{-},{\xi }_2^{-}\cdots {\xi }_m^{-})}^T$ are slack variables that represent the upper and lower bound constraints of the system output, and C > 0 is a pre-specified value. To solve equation (4), we formulate Lagrange as follows:

$$\begin{array}{c}\multicolumn{1}{c}{L_P(\omega ,b,{\xi }^{+},{\xi }^{-},\alpha ,\mu )}\\ =\frac{1}{2}\parallel \omega {\parallel }^2+C{\sum }_{i=1}^m({\xi }_i^{+}+{\xi }_i^{-})-{\sum }_{i=1}^m({\mu }^{+}{\xi }_i^{+}+{\mu }^{-}{\xi }_i^{-})\hfill \\ -{\sum }_{i=1}^m{\alpha }_i^{+}({\epsilon }_i+{\xi }_i^{+}+y_i-\omega x_i-b)\hfill \\ -{\sum }_{i=1}^m{\alpha }_i^{-}({\epsilon }_i+{\xi }_i^{-}-y_i+\omega x_i+b),\hfill \end{array}$$ (7)

where ${\alpha }_i^{+}\ge 0$ and ${\alpha }_i^{-}\ge 0$ (i = 1,2,…,m) are the Lagrange multipliers.

$$\frac{\partial L_P}{\partial \omega }=\omega -\sum _{i=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})x_i=0,$$ (8)

$$\frac{\partial L_P}{\partial b}=\sum _{i=1}^m({\alpha }_i^{+}-{\alpha }_i^{-})=0,$$ (9)

$$\frac{\partial L_P}{\partial {\xi }_i^{+}}=C-{\mu }_i^{+}-{\alpha }_i^{+}=0,$$ (10)

$$\frac{\partial L_P}{\partial {\xi }_i^{-}}=C-{\mu }_i^{-}-{\alpha }_i^{-}=0,$$ (11)

$$\omega =\sum _{i=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})x_i.$$ (12)

According to the above KKT conditions, the Wolfe dual of the primal problem equation (4) can be obtained as follows:

$$\begin{array}{c}\underset{{\alpha }^{-},{\alpha }^{+}}{max}-\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})({\alpha }_j^{-}-{\alpha }_j^{+})x_i{x}_j\hfill \\ -\sum _{i=1}^m{\epsilon }_i({\alpha }_i^{-}+{\alpha }_i^{+})+\sum _{i=1}^m{y}_i({\alpha }_i^{-}-{\alpha }_i^{+}),\hfill \end{array}$$

subject to

$$\sum _{i=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})=0,{\alpha }_i^{-},{\alpha }_i^{+}\in [0,C].$$ (13)

If there is a nonlinear relationship between y and x, then we can linearize it by mapping x to a higher dimensional feature space ²⁹ through the mapping relationship $\varphi :R^n\to$ $\mathcal{H}$ . The estimated regressor is given by:

$$f(x)={\omega }^T\varphi (x)+b.$$ (14)

Equation (13) is rewritten as

$$\begin{array}{cc}\multicolumn{1}{c}{\underset{{\alpha }^{-},{\alpha }^{+}}{max}}& -\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})({\alpha }_j^{-}-{\alpha }_j^{+})\varphi \left(x_i\right)\varphi \left(x_j\right)\hfill \\ \multicolumn{1}{c}{}& -\sum _{i=1}^m{\epsilon }_i({\alpha }_i^{-}+{\alpha }_i^{+})+\sum _{i=1}^m{y}_i({\alpha }_i^{-}-{\alpha }_i^{+}),\hfill \end{array}$$

subject to

$$\sum _{i=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})=0,{\alpha }_i^{-},{\alpha }_i^{+}\in [0,C]$$ (15)

According to the Mercer condition, $\varphi \left(x_i\right)\varphi \left(x_j\right)$ can convert to a positive definite kernel $\kappa (x_i,x_j).$

$$\kappa (x_i,x_j)=\varphi \left(x_i\right)\varphi \left(x_j\right)$$ (16)

Given the excellent fitting properties of the radial basis function (RBF), we employ it to define the kernel function $k(x_i,x_j)$ . An RBF kernel function can be expressed as:

$$\kappa (x_i,x_j)=\exp (-\frac{{\Vert x_i-x_j\Vert }^2}{2{\sigma }^2}).$$ (17)

This transformation reduces the computational complexity of the optimization problems, and therefore the problem can be reformulated by equation (16) as follows:

$$K=\left[\begin{array}{ccc}\kappa (x_1,x_1)\hfill & \cdots \hfill & \kappa (x_1,x_m)\hfill \\ \multicolumn{1}{c}{⋮}& \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{\kappa (x_m,x_1)}& \cdots \hfill & \kappa (x_m,x_m)\hfill \end{array}\right],$$ (18)

$${\mathrm{A}}^{-}=\left[\begin{array}{c}{\alpha }_1^{-}\hfill \\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{{\alpha }_m^{-}}\end{array}\right],{\mathrm{A}}^{+}=\left[\begin{array}{c}{\alpha }_1^{+}\hfill \\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{{\alpha }_m^{+}}\end{array}\right],$$ (19)

$$E=\left[\begin{array}{c}{\epsilon }_1\hfill \\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{{\epsilon }_m}\end{array}\right],y=\left[\begin{array}{c}{y}_1\hfill \\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{y_m}\end{array}\right],$$ (20)

The above formula in the given computer calculation form becomes:

$$\underset{{\alpha }^{-},{\alpha }^{+}}{max}-\frac{1}{2}{\left[\begin{array}{c}{\mathrm{A}}^{-}\hfill \\ \multicolumn{1}{c}{{\mathrm{A}}^{+}}\end{array}\right]}^T\left[\begin{array}{cc}K\hfill & -K\hfill \\ \multicolumn{1}{c}{-K}& K\hfill \end{array}\right]\left[\begin{array}{c}{\mathrm{A}}^{-}\hfill \\ \multicolumn{1}{c}{{\mathrm{A}}^{+}}\end{array}\right]+{\left[\begin{array}{c}\mathrm{E}-y\hfill \\ \multicolumn{1}{c}{\mathrm{E}+y}\end{array}\right]}^T\left[\begin{array}{c}{\mathrm{A}}^{-}\hfill \\ \multicolumn{1}{c}{{\mathrm{A}}^{+}}\end{array}\right],$$

subject to

$${\left[\begin{array}{c}H\hfill \\ \multicolumn{1}{c}{-H}\end{array}\right]}^T\left[\begin{array}{c}{A}^{-}\hfill \\ \multicolumn{1}{c}{A^{+}}\end{array}\right]=0,H={\left[\begin{array}{c}1\hfill \\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{1}\end{array}\right]}_{m\times 1},{\alpha }_i^{-},{\alpha }_i^{+}\in [0,C],$$ (21)

Equation (21) is the Linear Programing Problem (LPP), where ${\alpha }_i^{-}$ and ${\alpha }_i^{+}$ denote the Lagrange multipliers estimated when the condition ${\epsilon }_i$ and the constant C are given. Here, we use the quadratic programing package in Python to solve ${\alpha }_i^{-}$ and ${\alpha }_i^{+}$ . Using equation (14), the $A{\epsilon }_i$ -SVR function can be represented as:

$$f(x)=\sum _{i=1}^m({\alpha }_i^{-}-{\alpha }_i^{+})\kappa (x_i,x_j)+b.$$ (22)

It is evident that the influence of $\epsilon$ on the support vector has been transformed into the influence on the value of $({\alpha }_i^{-}-{\alpha }_i^{+})$ . Concerning the training set, the samples in the sample set $S=\{(x_i,y_i):x_i\in R^n,y_i\in R,i=1,2,\dots ,s\}$ that satisfy the condition $({\alpha }_i^{-}-{\alpha }_i^{+})\ne 0$ are used as support vectors for the SVR and must locate on the edges or outside the tube. Here, we alter ${\epsilon }_i$ at each position in the training set and then change the number of support vectors for the counterpart location. In this case, given the training set, the adaptive ${\epsilon }_i$ is computed through the iterative algorithm (Algorithm 1) to determine the best support vectors at every position. Based on the KKT condition, each input dimension b is described as:

$$b_i=y_i-\sum _{i=1}^s({\alpha }_i^{-}-{\alpha }_i^{+})\kappa (x_i,x_j)+{\epsilon }_i,$$ (23)

Finally, we take the average value of b obtained by all the support vectors and obtain:

$$b=\frac{\sum _{i=1}^s{b}_i}{s}.$$ (24)

Algorithm 1: The iterative fluctuation self-selection algorithm

Input: $Dataset\left(x_i,y_i\right)$ , parameters $E_0$ , $E_{\mathrm{original}}$ , $E_{\mathrm{up}}$ , $E_{\mathrm{md}}$ , $E_{\mathrm{dow}}$ , $E_{\mathrm{best}}$ , in the first iteration $E_{\mathrm{best}}=E_0=(\epsilon ,\epsilon ,\cdots ,\epsilon )$ , $i=1,\cdots ,N$ , repeat times M, constant C, kernel parameter $\sigma$ , fluctuation $F(0<F<1)$ Output: $E_{\mathrm{best}}=({\epsilon }_1^{*},{\epsilon }_2^{*},\cdots ,{\epsilon }_i^{*})$ 1 According to a certain proportion P, randomly divided into a function of training set          $S_{\mathrm{train}}={\{{\mathit{x}}_{\mathbf{i}},{\mathit{y}}_i\}}_{i=1}^P\mathrm{and}\mathrm{test}\mathrm{set}{S}_{\mathrm{test}}={\{{\mathit{x}}_{\mathcal{j}},{\mathbf{y}}_j\}}_{j=1}^{1-P},{\mathit{x}}_{\mathcal{i}},{\mathit{x}}_{\mathcal{j}}\in R^d;{\mathit{y}}_{\mathcal{i}},{\mathit{y}}_{\mathcal{j}}\in R$ 2      for $i=1$ Mdo3      $\mathrm{Calculate}{\mathit{e}}_{\mathit{i}}=y_{\mathit{i}}-f(x{)}_{E=E_{\mathrm{best}}}$ 4     for $i=1$ N do5      if $|{\mathbf{e}}_i/y_i|>Q$ then6           $\mathrm{Calculate}\mathrm{the}{\epsilon }_i^{\mathrm{up}}=(1+F)·e_i,{\epsilon }_i^{\mathrm{dow}}=(1-F)·e_i\mathrm{and}{\epsilon }_i^{\mathrm{med}}=e_i$ 7     else8           ${\epsilon }_i^{\mathrm{up}}={\epsilon }_i^{\mathrm{dow}}={\epsilon }_i^{\mathrm{md}}={\epsilon }_i^{\mathrm{best}}$ 9      end10  end11   Calculate $f(x{)}_{E=E_{\mathrm{up}}},f(x{)}_{E=E_{\mathrm{md}}}$ , and $f(x{)}_{E=E_{\mathrm{dow}}}$ according to Eq12   Calculate the training $\mathrm{MA}{E}_{E=E_{\mathrm{up}}},\mathrm{MA}{E}_{E=E_{\mathrm{md}}}\mathrm{and}\mathrm{MA}{E}_{E=E_{\mathrm{dow}}}$ 13   if $\mathrm{MA}{E}_{E=E_{\mathrm{dow}}}<\mathrm{MA}{E}_{E=E_{\mathrm{up}}}\mathrm{andMA}{E}_{E=E_{\mathrm{dow}}}<\mathrm{MA}{E}_{E=E_{\mathrm{md}}}$ then14        $E_{\mathrm{best}}=E_{\mathrm{dow}}$ 15   else if $\mathrm{MA}{E}_{E=E_{\mathrm{dow}}}\ge \mathrm{MA}{E}_{E=E_{\mathrm{up}}}\mathrm{and}\mathrm{MA}{E}_{E=E_{\mathrm{up}}}<\mathrm{MA}{E}_{E=E_{\mathrm{md}}}$ then16       $E_{\mathrm{best}}=E_{\mathrm{up}}$ 17   else18    $E_{\mathrm{best}}=E_{\mathrm{med}}$ 19   end20 end

Iterative fluctuation self-selection algorithm

The developed framework comprises three steps. First, the data is divided into K subsets, and we randomly select K–1 subsets as training samples and the rest are the test samples, while the kernel parameter $\sigma$ , adaptive ${\epsilon }_i$ , and constant C are given. The second step determines the starting position of the iteration by introducing the standard $\epsilon$ -SVR, after which ${\mathit{e}}_i=y_i-f\left(x_i\right)$ is calculated. In this case, data points outside the tube should satisfy ${\mathit{e}}_i\ge \epsilon +{\xi }^{+}$ . Here F is the fluctuation coefficient, which provides a fluctuation range for the e value and increases the randomness of each e value to prevent overfitting. Moreover, we set a criterion Q value (Q = 0.1), when $|e_i/y_i|>Q$ , and $e_i$ is considered as the corresponding ${\epsilon }_i$ for $A{\epsilon }_i$ -SVR. After screening, the $E_{\mathrm{original}}=({\epsilon }_1,{\epsilon }_2,\cdots ,{\epsilon }_i)$ can be obtained. The last step is to acquire the optimal $E_{\mathrm{best}}=({\mathbf{\epsilon }}_1^{\mathrm{best}},{\mathbf{\epsilon }}_2^{\mathrm{best}},\cdots ,{\mathbf{\epsilon }}_i^{\mathrm{best}})$ through the iterative fluctuation self-selection algorithm, which is summarized in Algorithm 1.

In Algorithm 1, MAE is used as the reliability index, and the fluctuation $F(0<F<1)$ is exploited to improve the model’s generalization ability. By appropriately setting $F$ , $E_{\mathrm{UP}}$ , and $E_{\mathrm{dow}}$ are generated, and then we find the best value $E_{\mathrm{best}}$ among $E_{\mathrm{UP}}$ , $E_{\mathrm{MD}}$ , and $E_{\mathrm{dow}}$ . If the F value is set too large, we may miss a satisfactory $E_{\mathrm{best}}$ , and if it is set too small, the iteration process will last longer.

Figure 5 illustrates the relationship between the reliability index and the number of algorithm calls, revealing that as the number of iterations increases, the prediction accuracy of the $A{\epsilon }_i$ -SVR model improves.

Figure 5.

Summary of the steps performed by the iterative fluctuation self-selection algorithm.

Applications on simulation and real data

This section evaluates the proposed approach on seven simulated and seven real-world benchmark datasets. For all the subsequent experiments, we set $F=0.1$ , and we employ the RBF kernel $\kappa (x_i,x_j)=\exp (-\frac{{\Vert x_i-x_j\Vert }^2}{2{\sigma }^2})$ , where $\sigma$ is the kernel parameter in all regression models, and its value is searched in the set $\{0.1,0.3,0.5,0.7,0.9,1,3,5,7,10,30\}$ . Additionally, the values of parameter $C$ and $\epsilon$ for all SVR models are obtained by searching in the sets $\{0.01,0.1,1,10,100,1000\}$ and $\{0.001,0.01,0.03,0.05,0.07,0.09,0.1,1,2\}$ , respectively. Moreover, the following three common criteria are presented to verify the effectiveness of the $A{\epsilon }_i$ -SVR model.

$$\mathrm{SSE}/\mathrm{SST}=\frac{{\sum }_{i=1}^m{(y_i-{\hat{y}}_i)}^2}{{\sum }_{i=1}^m{(y_i-{\overline{y}}_i)}^2},$$ (25)

$$\mathrm{RMSE}=\sqrt{\frac{1}{m}\sum _{i=1}^m{(y_i-{\hat{y}}_i)}^2},$$ (26)

$$\mathrm{MAE}=\frac{1}{n}\sum _{i=1}^n|y_i-{\hat{y}}_i|.$$ (27)

Simulation data application

We generate seven simulated datasets randomly using Python (http://www.python.org/) to evaluate the proposed method’s performance and behavior. Accordingly, we employ seven datasets, that is, 1–2, 3–5, 6, and 7, from Anand et al., ²⁵ Scorent, ³⁰ van der Laan, ³¹ and Meier et al, ³² respectively. Regarding the regression frameworks, we consider the covariates $X=(X_1\cdots X_n)$ . In dataset 1–2, each predictor $X_i$ is from $U[-10,10]$ while in dataset 3–7, each predictor $X_i$ is from $\mathcal{N}(0,1)$ , and $X_i$ is converted to ${\tilde{X}}_i$ through the function ${\tilde{X}}_i=2(X_i-0.5),i=1,\cdots ,n$ . All datasets are described below:

$$•\mathrm{Dataset}1:Y=\frac{\sin \left(X\right)}{X}+\mathcal{N}(0,0.1)$$

$$\begin{array}{c}\multicolumn{1}{c}{•\mathrm{Dataset}2:Y=\left|\frac{X-1}{4}\right|+\left|\sin \left(\pi (1+\frac{X-1}{4})\right)\right|}\\ +1+\mathcal{N}(0,0.1)\hfill \end{array}$$

$$•\mathrm{Dataset}3:Y={\tilde{X}}_1^2+e^{-{\tilde{X}}_2^2}+\mathcal{N}(0,0.25)$$

$$\begin{array}{c}\multicolumn{1}{c}{•\mathrm{Dataset}4:Y={\tilde{X}}_1{\tilde{X}}_2+{\tilde{X}}_3^2-{\tilde{X}}_4{\tilde{X}}_7+{\tilde{X}}_5{\tilde{X}}_8}\\ -{\tilde{X}}_6^2+\mathcal{N}(0,0.5)\hfill \end{array}$$

$$\begin{array}{c}\multicolumn{1}{c}{•\mathrm{Dataset}5:Y=-\sin \left({\tilde{X}}_1\right)+{\tilde{X}}_2^2+{\tilde{X}}_3-e^{-{\tilde{X}}_4^2}}\\ +\mathcal{N}(0,0.5)\hfill \end{array}$$

$$\begin{array}{c}\multicolumn{1}{c}{•\mathrm{Dataset}6:Y={\tilde{X}}_1^2+{\tilde{X}}_2^2{\tilde{X}}_3{e}^{-\left|{\tilde{X}}_4\right|}+{\tilde{X}}_6}\\ -{\tilde{X}}_5+\mathcal{N}(0,0.5)\hfill \end{array}$$

$$•\mathrm{Dataset}7:Y={\tilde{X}}_1+3{\tilde{X}}_2^2-2e^{-{\tilde{X}}_3}+{\tilde{X}}_4$$

All datasets contain 400 training and 100 test samples. Table 1 reports the performance of the $A{\epsilon }_i$ -SVR, $\epsilon$ -SVR, and $\mathrm{LS}$ -SVR models under several evaluation metrics. SSE/SST, RMSE, and MAE were calculated on seven simulation data sets with different data distribution characteristics, and the optimal parameter settings for $\epsilon$ -SVR per dataset are listed in Table 1. To compare the performance more objectively, the values of C and $\sigma$ of the competitor models are aligned with the $\epsilon$ -SVR. Table 1 highlights that without changing the parameters, the performance of the proposed $A{\epsilon }_i$ -SVR model presents a significant improvement in the SSE/SST, RMSE, and MAE metrics compared with the $\epsilon$ -SVR and $\mathrm{LS}$ -SVR models. Specifically, $A{\epsilon }_i$ -SVR obtains the lowest RMSE value (0.103) in the artificial Dataset 1, inferring that the proposed $A{\epsilon }_i$ -SVR model consistently outperforms the $\epsilon$ -SVR and $\mathrm{LS}$ -SVR models, demonstrating an appealing generalization ability.

Table 1.

Performance comparison of $A{\epsilon }_i$ -SVR, $\epsilon$ -SVR, and $\mathrm{LS}$ -SVR models on the lation datasets.

Datasets	Regressor	SSE/SST	RMSE	MAE	$(\sigma ,C,\epsilon )$
Dataset 1	$\epsilon$ -SVR	0.080	0.107	0.086	(0.1, 1, 0.2)
	$\mathrm{LS}$ -SVR	0.081	0.108	0.085	(0.1, 1, –)
	$A{\epsilon }_i$ -SVR	0.074	0.103	0.081	(0.1, 1, –)
Dataset 2	$\epsilon$ -SVR	0.321	2.210	1.760	(0.1, 10, 1)
	$\mathrm{LS}$ -SVR	0.320	2.207	1.735	(0.1, 10, –)
	$A{\epsilon }_i$ -SVR	0.317	2.194	1.729	(0.1, 10, –)
Dataset 3	$\epsilon$ -SVR	0.597	0.483	0.376	(0.7, 10, 0.5)
	$\mathrm{LS}$ -SVR	0.582	0.477	0.370	(0.7, 10, –)
	$A{\epsilon }_i$ -SVR	0.576	0.474	0.366	(0.7, 10, –)
Dataset 4	$\epsilon$ -SVR	0.377	0.532	0.419	(0.1, 10, 0.6)
	$\mathrm{LS}$ -SVR	0.379	0.534	0.427	(0.1, 10, –)
	$A{\epsilon }_i$ -SVR	0.374	0.529	0.415	(0.1, 10, –)
Dataset 5	$\epsilon$ -SVR	0.289	0.489	0.383	(0.1, 10, 0.3)
	$\mathrm{LS}$ -SVR	0.292	0.492	0.388	(0.1, 10, –)
	$A{\epsilon }_i$ -SVR	0.283	0.482	0.377	(0.1, 10, –)
Dataset 6	$\epsilon$ -SVR	0.281	0.565	0.461	(0.1, 10, 0.01)
	LS-SVR	0.285	0.569	0.464	(0.1, 10, –)
	$A{\epsilon }_i$ -SVR	0.273	0.558	0.453	(0.1, 10, –)
Dataset 7	$\epsilon$ -SVR	0.199	0.514	0.422	(0.1, 100, 0.6)
	$\mathrm{LS}$ -SVR	0.193	0.505	0.412	(0.1, 100, –)
	$A{\epsilon }_i$ -SVR	0.188	0.499	0.408	(0.1, 100, –)

Next, we calculated the reduction in SSE/SST, RMSE, and MAE for $A{\epsilon }_i$ -SVR relative to the $\epsilon$ -SVR and $\mathrm{LS}$ -SVR models. The corresponding results are illustrated in Figure 6, highlighting that the proposed model has an enhanced predictive performance compared to the competitor models, with the most obvious improvement being in the SSE/SST index. In Dataset 1, the SSE/SST value of $A{\epsilon }_i$ -SVR is 0.074, achieving an improvement of 7.5% and 8.6% relative to $\epsilon$ -SVR and $\mathrm{LS}$ -SVR, respectively. Additionally, Figure 6 reveals that the proposed model affords an improved prediction performance over the competitor models, reflecting the importance of “Alterable ${\epsilon }_i$ .” Therefore, the experiments imply that assigning variable $\epsilon$ depending on the data distribution characteristics is superior to treating each data point in the training set equally.

Figure 6.

Percentage reduction of SSE/SST, RMSE, and MAE.

Real data application

We also apply our proposed method to real benchmark datasets, namely Traizines, Servo, NO₂, Chwirut, Auto Mpg, Nelson, and Boston Housing. These datasets were downloaded from the UCI repository ³³ (https://archive.ics.uci.edu/address) and NLSRD (https://www.itl.nist.gov/div898/strd/nls/nls_main.shtml/address), and are commonly used to evaluate regression methods. Further information on the seven datasets is reported in Table 2. In this trial, we challenged the proposed SVR against some existing SVR models, such as the $\epsilon$ -SVR model, the Huber SVR model, and the $\epsilon$ -PSVR. The eigenvectors in the range $[0,1]$ were normalized for all datasets, and a 10-fold cross-validation method was used (Figure 7). The performance criterion for $\epsilon$ -PSVR is the result obtained with the same dataset and processing. ²⁵ In Figure 8, we present the iterative results of three real benchmark datasets and highlight that Algorithm 1 convergences after about 30 iterations.

Table 2.

Description of seven real-world benchmark datasets.

Regressor	No. of instances	Total features	Source
Auto Mpg	398	9	UCI
Boston Housing	506	14	UCI
Traizines	186	61	UCI
NO2	500	8	UCI
Servo	167	5	UCI
Nelson	128	3	NLSRD
Chwirut	214	2	NLSRD

Figure 7.

Convergence curves of algorithm 1 for real world datasets versus iteration times.

Figure 8.

Comparison of models fitting in the AutoMpg dataset.

To comprehensively verify the effectiveness of the proposed $A{\epsilon }_i$ -SVR model, this section evaluates our method based on the SSE/SST, RMSE, and MAE metrics and compares it against the current advanced methods presented in Tables 3–6. The experiments are conducted on seven real data sets, where 90% of the data are used for training and 10% for testing. During the trials, we adopted the original parameter setup as suggested by each paper. Table 3 reports the prediction performance of all competitor models, highlighting that $A{\epsilon }_i$ -SVR yields the lowest RMSE (2.505) followed by $\epsilon$ -PSVR (2.553) in the Auto Mpg dataset while employing the same parameters C and sigma. Additionally, the prediction performance of $A{\epsilon }_i$ -SVR is superior to $\epsilon$ -PSVR (2.553), Huber SVR (2.566), and $\epsilon$ -SVR (2.567). Besides, the SSE/SST metric of $A{\epsilon }_i$ -SVR shows a noticeable improvement in the Boston Housing dataset (0.123), which is much lower than $\epsilon$ -SVR (0.229) and $\epsilon$ -PSVR (0.228). After analyzing all three evaluation metrics, we conclude that $A{\epsilon }_i$ -SVR attains better performance in the seven real datasets than SVR, where the $\epsilon$ -tube has a single fixed value or two different values for inside and outside. Furthermore, although the $A{\epsilon }_i$ -SVR model performs better on all datasets, the performance gain over the competitor methods varies. Therefore, Tables 4–6 calculate our model’s improved prediction performance for each evaluation index and compare it against the other models.

Table 3.

Performance comparison of $A{\epsilon }_i$ -SVR, $\epsilon$ -SVR, Huber SVR, $\epsilon$ -PSVR, and $A{\epsilon }_i$ -SVR models on real world benchmark datasets.

Datasets	Regressor	(t2, t1)	SSE/SST	RMSE	MAE	$(\sigma ,\mathrm{C},\epsilon )$
Auto Mpg	$\epsilon$ -SVR	(1, 0.1)	0.115	2.567	1.844	(0.5, 64, 1)
	Huber SVR		0.115	2.566	1.832	(0.5, 64, 1)
	$\epsilon$ -PSVR		0.112	2.553	1.849	(0.5, 64, 1)
	$A{\epsilon }_i$ -SVR		0.101	2.505	1.818	(0.5, 64, –)
Boston Housing	$\epsilon$ -SVR	(1, 0.1)	0.229	3.363	2.402	(2, 128, 2)
	Huber SVR
	$\epsilon$ -PSVR		0.235	3.395	2.396	(2, 128, 2)
	$A{\epsilon }_i$ -SVR		0.228	3.351	2.398	(2, 128, 2)
			0.123	3.106	2.031	(2, 128, –)
Chwirut	$\epsilon$ -SVR	(1.5, 0.7)	0.023	3.231	2.249	(0.15, 64, 0.3)
	Huber SVR		0.022	3.219	2.242	(0.15, 64, 0.3)
	$\epsilon$ -PSVR		0.022	3.225	2.287	(0.15, 64, 0.3)
	$A{\epsilon }_i$ -SVR		0.019	3.110	2.234	(0.15, 64, –)
Traizines	$\epsilon$ -SVR	(1, 0.1)	0.893	0.132	0.099	(1, 4, 0.1)
	Huber SVR		0.914	0.136	0.101	(1, 4, 0.1)
	$\epsilon$ -PSVR		0.874	0.132	0.099	(1, 4, 0.1)
	$A{\epsilon }_i$ -SVR		0.807	0.131	0.095	(1, 4, –)
NO2	$\epsilon$ -SVR	(1, 0.4)	0.484	0.502	0.398	(0.5, 8, 0.3)
	Huber SVR
	$\epsilon$ -PSVR		0.495	0.507	0.401	(0.5, 8, 0.3)
	$A{\epsilon }_i$ -SVR		0.482	0.502	0.400	(0.5, 8, 0.3)
			0.479	0.501	0.392	(0.5, 8, –)
Nelson	$\epsilon$ -SVR	(1, 0.2)	0.153	1.311	0.978	(10, 102, 0.2)
	Huber SVR		0.152	1.300	0.965	(10, 102, 0.2)
	$\epsilon$ -PSVR		0.149	1.293	0.963	(10, 102, 0.2)
	$A{\epsilon }_i$ -SVR		0.147	1.271	0.950	(10, 102, –)
Servo	$\epsilon$ -SVR	(1.2, 0.1)	0.165	0.538	0.305	(2.8, 4, 0.1)
	Huber SVR		0.173	0.529	0.292	(2.8, 4, 0.1)
	$\epsilon$ -PSVR		0.162	0.530	0.304	(2.8, 4, 0.1)
	$A{\epsilon }_i$ -SVR		0.136	0.527	0.261	(2.8, 4, –)

Table 4.

Percentage reduction in $A{\epsilon }_i$ -SVR compared to other models on SSE/SST.

Regressor	Datasets
	Auto Mpg (%)	Boston Housing (%)	Chwirut (%)	Traizines (%)	NO2 (%)	Nelson (%)	Servo (%)
$\epsilon$ -SVR	12.17	46.29	13.04	9.63	1.03	3.92	17.58
Huber SVR	12.17	47.66	9.09	11.71	3.23	3.29	21.39
$\epsilon$ -PSVR	9.82	46.05	9.09	7.67	1.34	1.34	16.05

Table 5.

Percentage reduction in $A{\epsilon }_i$ -SVR compared to other models on RMSE.

Regressor	Datasets
	Auto Mpg (%)	Boston Housing (%)	Chwirut (%)	Traizines (%)	NO2 (%)	Nelson (%)	Servo (%)
$\epsilon$ -SVR	2.42	7.64	3.74	0.76	0.20	3.05	2.04
Huber SVR	2.38	8.51	3.39	3.68	1.18	2.23	0.38
$\epsilon$ -PSVR	1.88	7.31	3.57	0.76	1.70	1.70	0.57

Table 6.

Percentage reduction in $A{\epsilon }_i$ -SVR compared to other models on MAE.

Regressor	Datasets
	Auto Mpg (%)	Boston Housing (%)	Chwirut (%)	Traizines (%)	NO2 (%)	Nelson (%)	Servo (%)
$\epsilon$ -SVR	1.41	15.45	0.67	4.04	1.51	2.86	14.43
Huber SVR	0.76	15.23	0.36	5.94	2.24	1.55	10.62
$\epsilon$ -PSVR	1.68	15.30	2.32	4.04	1.35	1.35	14.14

Tables 4–6 highlight that from the three evaluation metrics, the developed model attains the highest performance on the Boston Housing dataset, affording an improvement of 46.29%, 7.64%, and 15.45% on the SSE/SST, RMSE, and MAE metrics, respectively. This is because our model focuses on solving the problem of inconsistent data distribution in real data, which is the case for the Boston Housing dataset. It should be noted that $A{\epsilon }_i$ -SVR performs well on the other datasets, with the improvement being around 1%, but in some cases exceeds 10% compared to existing methods.

Next, we randomly select one independent variable from the multiple independent variables as the x-axis and the dependent variable as the y-axis to demonstrate the model’s fit to the data sets on a two-dimensional plane.

Figures 9–11 reveal that the $A{\epsilon }_i$ -SVR model outperforms $\epsilon$ -SVR on all data sets, with the Boston Housing dataset presenting the most pronounced prediction gap. Particularly, many data points are located in the lower half, and the distribution is relatively dense when the Indus value is 0.2. When the Indus value is 0.5, there are more data points in the upper half, and the distribution is sparse, with $\epsilon$ -SVR being unable to adapt to the data distribution characteristics because the locations where the data are sparse and dense are treated equally. Consequently, the model’s fitting effect on Boston Housing at positions 0.2 and 0.5 is poor. However, since $A{\epsilon }_i$ -SVR can vary the $\epsilon$ -tube radius depending on the sparsity levels, it exhibits a more flexible predictive performance, allowing it to fit data points better at both sparse and dense locations.

Figure 9.

Comparison of models fitting in the Boston Housing dataset.

Figure 10.

Comparison of models fitting in the NO₂ dataset.

Figure 11.

Comparison of models fitting in the Chwirut dataset.

Furthermore, the Chwirut dataset is a two-dimensional dataset with low dimensionality and a relatively uniform data point distribution, evenly distributed at each location. In this dataset, $A{\epsilon }_i$ -SVR and $\epsilon$ -SVR have a relatively satisfactory fit to this dataset, with Figure 12 highlighting the minor fitting difference between the two lines due to the thinness of $\epsilon$ -tube.

Figure 12.

The ten-fold cross validation.

The experimental results demonstrate that $A{\epsilon }_i$ -SVR outperforms the other SVR models because it has different penalty values for each data point in the dataset, which can be changed during the iteration process based on the sample information, aiming to achieve better prediction.

Parameter sensitivity study

This experiment uses a 10-fold cross-validation sampling strategy to evaluate the prediction performance of $A{\epsilon }_i$ -SVR under different parameter setups. Six independent experiments were conducted to investigate the sensitivity of the $A{\epsilon }_i$ -SVR and $\epsilon$ -SVR models while altering the c and gamma values on the Auto Mpg, Boston Housing, and NO₂ data sets. The range of the c and gamma values is chosen adjacent to the optimal value, and only one parameter is changed at a time, with the other taking the optimal value. The corresponding results are illustrated in Figure 13.

Figure 13.

Performance evaluation of $A{\epsilon }_i$ -SVR and $\epsilon$ -SVR on three datasets. First row (a–c): the testing accuracy of two models with different $\mathrm{sigma}$ . Second row (d–f): the training time of the two models with different $C$ .

For this experiment, we adopt a 10-fold cross-validation strategy, and in each fold, the average value is considered the final evaluation index after being evaluated on the test set. As observed, the prediction performance of the $A{\epsilon }_i$ -SVR model is better than the $\epsilon$ -SVR model under different C and sigma values. Compared with the standard $\epsilon$ -SVR model, $A{\epsilon }_i$ -SVR is less sensitive to the c-value as it maintains a good prediction effect even for a large c-value range. The sensitivity of $A{\epsilon }_i$ -SVR to sigma is the same as for the SVR model. If for the standard SVR model the time complexity is $O(·)$ , while the time complexity of the $A{\epsilon }_i$ -SVR model proposed in this paper is $\mathrm{MNO}(·)$ , where $M$ is the number of iterations set in Algorithm 1 and $N$ is the number of samples for training. When the sample size is large, it will increase the computation time significantly. Therefore, when selecting the $A{\epsilon }_i$ -SVR model, we can set a large search step for the C value and a standard search step for sigma. Through this setting, we offset some of the time spent to iteratively solve for ${\epsilon }_i$ , thus balancing training time and prediction performance.

Conclusion and future works

This paper proposes a novel “Alterable- ${\epsilon }_i$ ” model, which is adaptive to the data set distribution characteristics by determining ${\epsilon }_i$ at each position. Experimental results on several simulated and real data sets demonstrate that Alterable- ${\epsilon }_i$ has an obvious performance improvement over the state-of-the-art models due to the non-fixed $\epsilon$ -tube radius. Specifically, the proposed model has a significant advantage over current models for certain datasets.

Overall, the proposed $A{\epsilon }_i$ -SVR model’s performance outperforms the standard SVR model for the following reasons. First, the developed iterative fluctuation self-selection algorithm can find the appropriate $\epsilon$ penalty parameter for each location. Second, $A{\epsilon }_i$ contributes to the removal of noise and outliers in the dataset through a variable $\epsilon$ -tube formed by the free parameters. Third, $A{\epsilon }_i$ successfully extracts important data information through the free parameters.

Moreover, flexible bounds can lead to better prediction results in classification learning tasks. This property enables the Alterable- ${\epsilon }_i$ proposed for regression tasks to distinguish data points with inconsistent distribution sparsity, which is quite coherent with the requirements of classifying data. Therefore, in the future, we aim to extend Alterable- ${\epsilon }_i$ to solve classification problems, such as image classification.