A modified higher-order feed forward neural network with smoothing regularization

Khidir Shaib Mohamed, Liu Yan, Wu Wei

Abstract


This paper proposes an offline gradient method with smoothing L1/2 regularization for learning and pruning of the pi-sigma neural networks (PSNNs). The original L1/2 regularization term is not smooth at the origin, since it involves the absolute value function. This causes oscillation in the computation and difficulty in the convergence analysis. In this paper, we propose to use a smooth function to replace and approximate the absolute value function, ending up with a smoothing L1/2 regularization method for PSNN. Numerical simulations show that the smoothing L1/2 regularization method eliminates the oscillation in computation and achieves better learning accuracy. We are also able to prove a convergence theorem for the proposed learning method.


Keywords


Convergence, pi-sigma neural network, oine gradient method, smooth ing L1/2 regularization

Full Text:

PDF

References


SHIN Y., GHOSH J. The pi-sigma network: an efficient higher-order neural network for pattern classification

and function approximation, International joint conference on neural networks, 1991, 1, 13– 18.

JIANG L.J., XU F., PIAO S.R. Application of pi-sigma neural network to real classification of seafloor

sediments. Applied Acoustics. 2005, 24(6), 346 – 350.

SHIN Y., GHOSH J., SAMANI D. Computationally efficient invariant pattern classification with higherorder

pi-sigma networks, Intelligent Engineering Systems Through Artificial Neural Networks. 1992, 2,

– 384.

HUSSAINA A.J., LIATSISB P. Recurrent pi-sigma networks for DPCM image coding, Neurocomputing

, 55(1–2), 363 – 382.

WU W., XU Y. S., Deterministic convergence of an online gradient method for neural networks, Journal of

Computational and Applied Mathematics, 2002, 144(1-2), 335-347.

WU W., FENG G. R., LI X. Z., Training multiple perceptrons via minimization of sum of ridge functions,

Advances in Computational Mathematics ,2002 17, 331-347.

P FRIEDMAN J. H., An overview of predictive learning and function approximation, in Cherkassky, V.,

Friedman, J.H., and Wechsler, H. (Eds.), From Statistics to Neural Networks, Proc. NATO/ASI Workshop,

pp.1-61, 1994. Springer Verlag.

LE CUN Y., DENKER J.S., SOLLA S.A., Optimal brain damage,” Advances in Neural Information Processing

Systems-2, 1990, 598-605.

HERTZ J., KROGH A., PALMER R., Introduction to the Theory of Neural Computation. Redwood City,

CA: Addison Wesley, 1991.

HINTON G., Connectionist learning procedures. Artif Intell, 1989, 40, 185-235.

SRIVASTAVA N., HINTON G., KRIZHEVSKY A., SUTSKEVER I., SALAKHUTDINOW R., Dropout:

A simple way to prevent neural networks from overfitting. J Mach Learn Res, 2014, 15(1), 1929-1958

MACKAY D.J.C., Probable networks and plausible predictions- a review of practical Bayesian methods

for supervised neural networks, Network: Computation in Neural Systems, 995, 6(3), 469-505.

WANG C., VENKATESH S. S., JUDD J. S., Optimal stopping and effective machine complexity in learning.

Adv in Neural Inf Process Syst, 1994, 6, 303-310.

LOONE S.M., IRWIN G. Improving neural network training solutions using regularization, Neurocomputing.

, 37, 71–90.

XU Z. B., ZHANG H., WANG Y., CHANG X. Y., L1/2 Regularizer. Sci China Ser F: Inf Sci, 2010, 52,

-9.

LIU Y., LI Z. X., YAN D. k., MOHAMED KH. SH., WANG J., Wu W., Convergence of batch gradient

learning algorithm with smoothing L1/2 regularization for Sigma–Pi–Sigma neural networks, Neurocomputing,

, 151, 333–341.

FAN, Q.W., ZURADA J.M., WU W. Convergence of online gradient method for feed forward neural networks

with smoothing L1=2 regularization penalty, Neurocomputing. 2014, 131, 208–216.

YUAN Y.X., SUN W.Y. Optimization Theory and methods, Science Press, Beijing. 2001.




DOI: http://dx.doi.org/10.14311/NNW.2017.032

Refbacks

  • There are currently no refbacks.


Should you encounter an error (non-functional link, missing or misleading information, application crash), please let us know at nnw.ojs@fd.cvut.cz.
Please, do not use the above address for non-OJS-related queries (manuscript status, etc.).
For your convenience we maintain a list of frequently asked questions here. General queries to items not covered by this FAQ shall be directed to the journal editoral office at nnw@fd.cvut.cz.