On-line Mixture-model-based Logistic Regression

Ivan Nagy, Evgenia Suzdaleva


The paper deals with a problem of modeling discrete variables depending on continuous  variables. This problem is known as the logistic regression estimated by  numerical methods. The paper approaches the problem via the recursive Bayesian estimation of mixture models with the purpose of exploring a possibility of constructing the continuous data dependent switching model that should be estimated on-line.  Here the model of the discrete variable dependent on continuous data is represented as the model of the mixture pointer dependent on data from mixture components via their parameters, which switch according to the activity of the components. On-line estimation of the data dependent pointer model has a great potential for tasks of clustering and classification. The specific subproblems include (i) the model parameter estimation both of the pointer and of the components obtained  during the learning phase, and (ii) prediction of the pointer value during the testing phase. These two phases can be joined together in the case of necessity. A real-data experimental comparison with theoretical counterparts shows a competitiveness of the approach in the discussed field.


on-line modeling; on-line logistic regression; recursive mixture estimation; data dependent pointer


D. W. Hosmer Jr., S. Lemeshow and R. X. Sturdivant, Applied Logistic Regression, 3rd ed., Wiley, 2013.

D. A. Freedman, Statistical Models: Theory and Practice, Cambridge University Press, 2009.

M. Kologlu, D. Elker, H. Altun, I. Sayek, Validation of MPI and OIA II in two different groups of patients with secondary peritonitis, Hepato-Gastroenterology, vol. 48, 37 (2001), p. 147–151.

S. Biondo, E. Ramos, M. Deiros et al, Prognostic factors for mortality in left colonic peri- tonitis: a new scoring system, J. Am. Coll. Surg, vol. 191, 6 (2000), p. 635-642.

W. DuMouchel, Multivariate Bayesian logistic regression for analysis of clinical study safety issues, Statistical Science, vol. 27, 3 (2012), p. 319–339.

L. C. Thomas, K. Mun Jung, S. D. Thomas, Y. Wu, Modeling consumer acceptance proba- bilities, Expert Systems with Applications, vol. 30 (2006), p. 499–506.

T. C. Mattozo, G. Soares da Silva, A. P. Fernandes Neto, J. A. F. Costa, Logistic regression applied to airport customer satisfaction using hierarchical quality model, in: Intelligent data engineering and automated learning - IDEAL 2012, Lecture notes in computer science, vol. 7435, Springer Berlin Heidelberg, 2012, p. 558–567.

J. Nicolau, An analysis of the 2002 presidential elections using logistic regression, Brazilian political science review, vol. 1, 1 (2007), p. 125–135.

S. Menard, Applied logistic regression analysis (Quantitative applications in the social sci- ences), 2nd ed., SAGE Publications Inc., 2001.

C. J. Lin, R. C. Weng, S. S. Keerthi, Trust region Newton method for logistic regression, J. Mach. Learn. Res., vol. 9 (2008), p.627–650.

T. P. Minka, Algorithms for maximum-likelihood logistic regression, Technical Report, Carnegie Mellon University, 2003.

K. Watanabe, T. Kobayashi, N. Otsu, Efficient optimization of logistic regression by direct use of conjugate gradient, in: Proceedings of the 10th International Conference on Machine Learning and Applications and Workshops (ICMLA), vol.1, Honolulu, HI, 18-21 Dec. 2011, IEEE, p.496–500.

P. D. Allison, Convergence failures in logistic regression, in: SAS global forum 2008, Statistics and data analysis, vol. 360, 2008.

B. J. Park, Y. Zhang, D. Lord, Bayesian mixture modeling approach to account for hetero- geneity in speed data, Transportation Research Part B: Methodological, vol. 44, 5 (2010), p. 662–673.

Y. Zou, Y. Zhang, D. Lord, Analyzing different functional forms of the varying weight parameter for finite mixture of negative binomial regression models, Analytic Methods in Accident Research, vol. 1 (2014), p. 39–52.

Y. Xiong, F. L. Mannering, The heterogeneous effects of guardian supervision on adoles- cent driver-injury severities: A finite-mixture random-parameters approach, Transportation Research Part B: Methodological, vol. 49 (2013), p. 39–54.

Jie Yu, A nonlinear kernel Gaussian mixture model based inferential monitoring approach for fault detection and diagnosis of chemical processes, Chemical Engineering Science, vol. 68, 1 (2012), p. 506–519.

Jie Yu, A particle filter driven dynamic Gaussian mixture model approach for complex process monitoring and fault diagnosis, Journal of Process Control, vol. 22, 4 (2012), p. 778–788.

Jianbo Yu, Fault detection using principal components-based Gaussian mixture model for semiconductor manufacturing processes, IEEE Transactions on Semiconductor Manufactur- ing, vol. 24, 3 (2011), p. 432–444.

W. Bolstad, Understanding computational Bayesian statistics, Wiley, 2010.

A. Gelman, A. Jakulin, M. G. Pittau, Y.-S. Su, A weakly informative default prior distribu- tion for logistic and other regression models, Ann. Appl. Stat., vol. 2, 4 (2008), p. 1360–1383.

C. Liu, Robit regression: a simple robust alternative to logistic and probit regression, in: Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives: An Essential Journey with Donald Rubin’s Statistical Family (eds A. Gelman and X.-L. Meng), John Wiley & Sons, Ltd, Chichester, UK, 2004, doi: 10.1002/0470090456.ch21.

T. M. McCormick, A. E. Raftery, D. Madigan, R. S. Burd, Dynamic logistic regression and dynamic model averaging for binary classification, Biometrics, vol. 68, 1 (2012), p. 23–30.

Y. Maruyama, W. E. Strawderman, A new Monte Carlo sampling in Bayesian probit regres- sion, arXiv preprint, arXiv:1202.4339 (2012).

M. A.Pourhoseingholi, Y. Mehrabi, H. Alavi-Majd, P. Yavari, Using latent variables in logis- tic regression to reduce multicollinearity, A case-control example: breast cancer risk factors, Italian journal of public health, vol. 5, 1 (2008), p. 65–71.

S. Chib, I. Jeliazkov, Accept–reject Metropolis–Hastings sampling and marginal likelihood estimation, Statistica Neerlandica, vol. 59, 1 (2005), p. 30–44.

M. R. Gupta, Y. Chen, Theory and use of the EM method, in: Foundations and Trends in Signal Processing, vol. 4, 3 (2011), p. 223–296.

O.Boldea,J.R.Magnus,Maximumlikelihoodestimationofthemultivariatenormalmixture model, Journal Of The American Statistical Association, vol. 104, 488 (2009), p. 1539–1549.

J. A. Cuesta-Albertos, C. Matra ́n, A. Mayo-Iscar, Robust estimation in the normal mix- ture model based on robust clustering, Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 70, 4 (2008), p. 779–802.

H. xian Wang, B. Luo, Q. bin Zhang, S. Wei, Estimation for the number of components in a mixture model using stepwise split-and-merge EM algorithm, Pattern Recognition Letters, vol. 25, 16 (2004), p. 1799–1809.

H. Zeng, Y.-M. Cheung, Learning a mixture model for clustering with the completed likeli- hood minimum message length criterion, Pattern Recognition, vol. 47, 5 (2014), p. 2011–2030.

S. K. Ng, G.J. McLachlan, Mixture models for clustering multilevel growth trajectories, Computational Statistics & Data Analysis, vol. 71 (2014), p. 43–51.

C.A.McGrory,D.M.Titterington,VariationalBayesiananalysisforhiddenMarkovmodels, Australian & New Zealand Journal of Statistics, vol. 51(2009), p. 227–244.

V. ŠmÍdl, A. Quinn, The Variational Bayes Method in Signal Processing, Springer-Verlag Berlin Heidelberg, 2006.

Z. Ghahramani, G. E. Hinton, Variational learning for switching state-space models, Neural Computation, vol. 12, 4 (2000), p. 831–864.

S. Frühwirth-Schnatter, Finite Mixture and Markov Switching Models, Springer-Verlag New York, 2006.

S. Frühwirth-Schnatter, Fully Bayesian analysis of switching Gaussian state space models, Annals of the Institute of Statistical Mathematics, vol. 53, 1 (2001), p. 31–49.

R. Chen, J. S. Liu, Mixture Kalman filters, Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 62 (2000), p. 493–508.

V. Peterka, Bayesian system identification, in: Trends and Progress in System Identification (ed. P. Eykhoff), Oxford, Pergamon Press, 1981, p. 239–304.

M. Kárný, J. Böhm, T. V. Guy, L. Jirsa, I. Nagy, P. Nedoma, L. Tesař, Optimized Bayesian Dynamic Advising: Theory and Algorithms, Springer-Verlag London, 2006.

M. Kárný, J. Kadlec, E.L. Sutanto, Quasi-Bayes estimation applied to normal mixture, in: Preprints of the 3rd European IEEE Workshop on Computer-Intensive Methods in Control and Data Processing (eds. J. Rojíček, M. Valečková, M. Kárný, K. Warwick), CMP’98 /3./, Prague, CZ, 07.09.1998–09.09.1998, p. 77–82.

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall/CRC Texts in Statistical Science), 3rd ed., Chapman and Hall/CRC, 2013.

P. M. Lee, Bayesian Statistics: An Introduction, 4th ed., Wiley, 2012.

D. M. Titterington, A. F. M. Smith, U. E. Makov, Statistical Analysis of Finite Mixture Distributions (Wiley Series in Probability and Statistics – Applied Probability and Statistics Section), 1st ed., Wiley, 1986.

P. Deuflhard, Newton Methods for Nonlinear Problems, Affine Invariance and Adaptive Algorithms, Springer-Verlag Berlin Heidelberg, 2004.

S. Chapra, R. Canale, Numerical Methods for Engineers, 7th ed., McGraw-Hill Education, 2014.

I. Nagy, E. Suzdaleva, M. Kárný, T. Mlynářová, Bayesian estimation of dynamic finite mixtures, Int. Journal of Adaptive Control and Signal Processing, vol. 25 9 (2011), p. 765– 787.

I. Nagy, E. Suzdaleva, Mixture estimation with state-space components and Markov model of switching, Applied Mathematical Modelling, vol. 37 24 (2013), p. 9970–9984.

DOI: http://dx.doi.org/10.14311/NNW.1901.%25x


  • There are currently no refbacks.

Should you encounter an error (non-functional link, missing or misleading information, application crash), please let us know at nnw.ojs@fd.cvut.cz.
Please, do not use the above address for non-OJS-related queries (manuscript status, etc.).
For your convenience we maintain a list of frequently asked questions here. General queries to items not covered by this FAQ shall be directed to the journal editoral office at nnw@fd.cvut.cz.