Skip to content
KPI Science News
ISSN 2617-5509 · e‑ISSN 2663-7472 Open Access · CC BY 4.0
Vol. · Issue 2 · 2021 Aug 31, 2021 article

CHOOSING A QUALITY CRITERION FOR EVALUATING THE FORECAST OF NONLINEAR NON-STATIONARY PROCESSES

AB
Andrii O. Belas Corresponding Igor Sikorsky Kyiv Polytechnic Institute, Ukraine Ukraine
PB
Petro I. Bidyuk Igor Sikorsky Kyiv Polytechnic Institute, Ukraine Ukraine
PublishedAug 31, 2021
PDF (Ukrainian)
KPISN 2 VOL · 2
VOL · ISS 2 · 2021 View issue

Abstract

Background. The problem of forecasting nonlinear nonstationary processes presented in the form of time series is very relevant, since such series can describe dynamics of the processes in both technical and economic systems. To establish the best model, various metrics are used to assess the quality of forecasts, such as R^2, RMSE, MAE, MAPE. However, in many tasks, when optimizing the model according to the selected criterion, the model becomes worse in relation to another criterion. Therefore it is important to understand which metric must be used to optimize and assess the quality of the forecast in the given task. Objective. The aim of the paper is to develop a criteria base for assessing forecasts of nonlinear nonstationary processes, as well as an approach to choosing a metric in accordance to the specificity of the set forecasting problem. Methods. The paper presents a comparative analysis of the basic metrics for the regression problem, their theoretical and practical meaning, advantages and disadvantages in various cases. New approaches are proposed based on the results of the analysis. Results. Based on the analysis of the selected data, it is shown that by optimizing the model according to the selected criterion, the model becomes worse in relation to another criterion. A criterion basis for assessing forecasts of nonlinear nonstationary processes has been formed, as well as an approach to the selection of a quality criterion in accordance with the specifics of the set forecasting problem. To minimize an absolute error, the RMSE (MSE, R^2) and MAE metrics are analysed and recommended, depending on the need to work with outliers. The RMSLE metric is proposed for solving the problems of minimizing the relative metric, for solving the shown problems of the MAPE metric for this class of problems. Conclusions. The paper shows the importance of choosing a metric that must be used to optimize and assess the quality of the forecasts in the given task. The obtained criterion base and approach can be used in further research to solve practical prob- lems in modelling and forecasting nonlinear nonstationary processes and to develop new methods or general method for solving such problems.

Keywords

#mathematical modeling #forecasting #regression #forecast quality criteria #metrics #time series #nonstationary processes

References

  1. A.K. Palit and D. Popovic, Computational Intelligence in Time Series Forecasting: Theory and Engineering Applications. London, England: Springer-Verlag London, 2005, 372 p. doi: 10.1007/1-84628-184-9
  2. O.M. Belas et al., “General approach to forecasting nonlinear nonstationary processes using mathematical models on statistical data,” Special Telecommun. Syst. Inform. Security, vol. 2, no. 4. pp. 26–31, 2018.
  3. Walmart Recruiting – Store Sales Forecasting [Online]. Avaliable: https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting/data .
  4. O.M. Belas et al., “Comparative analysis of autoregressive approaches and recurrent neural networks for modeling and forecasting of nonlinear nonstationary processes,” Inform. Technol. Security, vol. 7, no. 1, pp. 91–99, 2019.
  5. P.I. Bidyuk et al., Time Series Analysis. Kyiv, Ukraine: Polytechnica, 2010, 317 p.
  6. R.J. Hyndman and G. Athanasopoulos, Forecasting: Principles and Practice. Melbourne, Australia: OTexts, 2013.
  7. I. Goodfellow et al., Deep Learning. New York: MIT Press, 2016, 800 p.
  8. C. Bishop, Pattern Recognition and Machine Learning. New York: Springer-Verlag New York, 2006, 738 p.
  9. S. Kolassa, “Evaluating predictive count data distributions in retail sales forecasting,” Int. J. Forecasting, vol. 32, no. 3, pp. 788–803, 2016. doi: 10.1016/j.ijforecast.2015.12.004
  10. E.F. Krause, Taxicab Geometry: An Adventure in Non-Euclidean Geometry. New York: Dover Publications, 1987, 96 p.
  11. J. S. Armstrong, Long-range Forecasting: From Crystal Ball to Computer. New York: John Wiley & Sons, 1978.
  12. S. Makridakis, “Accuracy measures: theoretical and practical concerns,” Int. J. Forecasting, vol. 9, no. 4, pp. 527–529, 1993. doi: 10.1016/0169-2070(93)90079-3
  13. P. Goodwin and R. Lawton, “On the asymmetry of the symmetric MAPE,” Int. J. Forecasting, vol. 15, no. 4, pp. 405–408, 1999. doi: 10.1016/S0169-2070(99)00007-2
  14. R.J. Hyndman and A.B. Koehler, “Another look at measures of forecast accuracy,” Int. J. Forecasting, vol. 22, no. 4, pp. 679–688, 2006. doi: 10.1016/j.ijforecast.2006.03.001
  15. F. Petropoulos and N. Kourentzes, “Forecast combinations for intermittent demand,” J. Operat. Res. Soc., vol. 66, no. 6, pp. 914–924, 2015, doi: 10.1057/jors.2014.62
  16. A. Davydenko and R. Fildes, “Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts,” Int. J. Forecasting, vol. 29, no. 3, pp. 510–522, 2013. doi: 10.1016/j.ijforecast.2012.09.002
§ 06 — Related

Similar articles in this journal

Related peer-reviewed studies published in this journal.
View all issues

AN APPROACH TO CONSTRUCTING AN INFORMATIVE FEATURE SPACE OF TECHNICAL INDICATORS FOR FINANCIAL PROCESSES BASED ON NEURAL NETWORK MODELS

Roman Tertychnyi, Denys Chuprin, Petro Bidyuk

Background. Financial time series are characterized by nonlinearity, non-stationarity, and a high level of noise, which significantly complicates their modeling and…

#informative feature space construction approach #technical indicators #Parkinson volatility #neural networks
10.20535/kpisn.2026.3.359610

Most read articles by the same author(s)

Similar Articles

1-10 of 201

You may also start an advanced similarity search for this article.