Financial and Real Sector Leading Indicators of Recessions in Brazil Using Probabilistic Models

We examine the usefulness of various financial and real sector variables to forecast recessions in Brazil between one and eight quarters ahead. We estimate probabilistic models of recession and select models based on their outof-sample forecasts, using the Receiver Operating Characteristic (ROC) function. We find that the predictive out-of-sample ability of several models vary depending on the numbers of quarters ahead to forecast and on the number of regressors used in the model specification. The models selected seem to be relevant to give early warnings of recessions in Brazil.


INTRODUCTION
The most recent financial crises showed, once again, the relevance of forecasting the downturns of business cycles.Economists in general did not anticipate the recessions that took place worldwide.
Economies evolve over time and are subject, sometimes, to large unanticipated structural breaks.Such breaks can be precipitated by sudden changes in economic policy, major scientific and technological discoveries and innovations, political turmoil or permanent macroeconomic shocks.
Economists often use complex mathematical models to forecast the path of the GDP and the likelihood of a recession (Bank of England, 2000;Hatch, 2001).The models used to understand and forecast processes as complicated as GDP are far from perfect representations of their behavior. 1 Simpler indicators such as interest rates, spread of interest rates, stock price indexes, monetary aggregates, and some readily available real sector indicators contain very relevant information about future economic activity. 2 These indicators can be used to verify both econometric and judgmental predictions by signaling a problem that might otherwise have gone unidentified.If forecasts from an econometric model and forecasts from these indicators agree, confidence in the model's results can be enhanced.In contrast, if these indicators forecasts give a different signal, it may be worthwhile to review the assumptions and relationships that led to the prediction of the more complex econometric models.
The indicators mentioned above are, in general, associated with expectations regarding the occurrence of future events, as shown by Estrella & Mishkin (1997) and Stock & Watson (2001), and therefore are natural candidates for leading indicators of economic activity.They also present some of the necessary properties of leading indicators.They conform to the business cycles; have economic significance, statistical accuracy and little need for revisions.Therefore, the development, as well as the monitoring, of such indicators can be very relevant for the formulation and implementation of macroeconomic policies, given that they give additional evidence about the state of the economy.
In this paper, we examine the usefulness of various financial and real sector variables in out-ofsample predictions of whether or not the Brazilian economy will be in a recession between one and eight quarters in the future.Variables with potential predictive content are selected from a broad array of candidates and are examined by themselves and in some plausible and parsimonious combinations.
We focus simply on predicting recessions rather than on quantitative measures of future economic activity.We believe that this is a useful exercise because it addresses a question frequently posed by policy makers and market participants. 3 We also are not concerned with misspecified models.As Hendry & Clements (2002) posit, it is by now well documented in the literature the fact that well specified models based on historical data may forecast out-of-sample worse than misspecified ones.The fundamental reason for this is the existence of unanticipated shifts or structural breaks in the economy in the future. 4After such a shift, a previously well-specified model may forecast less accurately than one that is misspecified.The best causal description of the economy may not be robust to such sudden shifts. 5 To assess how well each indicator variable predicts recessions, we use the so-called extreme value model-a particular case of a probabilistic model-, which, in our applications, directly relates the probability of being in a recession to specific groups of explanatory variables. 6 We also assess the capacity of variables to forecast recessions, and by this contribute to the literature, by selecting models based on out-of-sample forecasts, using a metric related to the Receiver Operating Characteristic (ROC) curve. 7The ROC curve plots the fraction of true positives (crisis = 1) that a given model signals (out of all positives in the sample) vs. the fraction of false positive signals (out of all negatives in the sample) along contiguous threshold settings.The best model according to this criterion is the one that delivers the highest trade-off frontier between true and false alarms. 8 , 9 We find that the predictive out-of-sample ability of several models vary depending on the numbers of quarters ahead to forecast and on the number of regressors used in the model specification.The best models selected can be thought as early warning signals of recessions in Brazil.Our selected models do good job in anticipating this recession.
There is a vast literature by now that search for good leading indicators of recessions such as we do in this paper.Just to mention some, Estrella & Mishkin (1997) use a probit model to evaluate the usefulness of financial variables to predict U.S. recessions, both in-and out-of-sample.Their full sample covers a number of recessions.Their main findings are that stock prices are the best leading indicators of recessions at the 1-and 2-quarter horizons.Bernard & Gerlach (1996) examine the ability of the term structure to predict recessions in eight countries (Belgium, Canada, France, Germany, Japan, the Netherlands, the United Kingdom and the United States) between the period 1972:1 and 1993:4.For all the countries, their study also shows that the yield curve provides information about the likelihood of future recessions up to eight quarters ahead.Lamy (1997) studies several macroeconomic indicators, to verify if they predict recessions in Canada.He finds the Department of Finance index of leading indicators of economic activity and the Bank of Canada nominal monetary conditions index to be strongest at predicting recessions for a forecast horizon of one quarter.At the horizon of two to four quarters, he finds the yield curve to be the best variable to predict recession.
In the case of forecasting recessions in Brazil, we can select two very interesting papers in the literature.Morais & Chauvet (2011) build leading indicators to predict the capital goods business cycle in Brazil.They propose a probit model with autoregressive dynamics.Their results indicate that the dynamic probit model has a better forecasting performance than the simple probit model in several aspects, both in and out of sample.
Another empirical paper for Brazil is Duarte, Issler, & Spacov (2004).The authors build several coincident and leading indicators of economic activity in Brazil to forecast recessions.Their results show that the best indicators for this purpose are the ones that follow the methodology of the Conference Board.
Two forecasting results emerge from our study.First and most important, the criteria to select models for forecast should be always out-of-sample performance.As Hendry & Clements (2002) point out the distributions of future outcomes are not the same as those in sample.That means that well specified in sample models will not necessarily forecast out-of-sample better than badly specified insample models.It may also be the case that variables that seem irrelevant will forecast better than relevant ones.Second, it is important to determine the optimal out-of-sample horizon for each forecasting model.Further ahead interval forecasts generally lead to worst forecasts than near-horizon ones.
Our results also confirm that despite its non-specific assumptions, a theory of forecasting which allows for structural breaks may provide a useful basis for interpreting, and circumventing, systematic forecast failure in economics.
The rest of the paper is organized as following.Section 2 describes the data.Section 3 presents the empirical analysis.Section 4 presents some case studies analyses.Section 5 concludes.

DATA
The macroeconomic indicators have a good performance record in predicting real activity.The financial series we look at may be less prone to the over fitting problem than the traditional macroeconomic indicators.
Another important consideration is the possible lag in the availability of the data for the explanatory variables.Some variables, such as interest rates and stock prices, are available on a continuous basis with no informational lag.In contrast, many monthly macroeconomic series are only available one or two months after the period covered by the data, and GDP has a lag of almost one full quarter.
Our sample has quarterly data and goes from the first quarter of 1991 to the fourth quarter of 2015.Table A-1 (see section A.1 in Appendix A) shows all the names of all the real sector variables we use in our empirical exercise.We have 83 real sector variables.We use the level and the first difference in logarithm of the series variables.When possible, we also use the seasonally adjusted series in level or in the first difference in logarithm.10Table A-2 shows the financial sector variables.We have 24 variables.Once more, we use the level as well as the first difference in logarithm of these series.
The recession variable is built using the standard two consecutive quarters of negative variation of GDP with seasonal adjustment.We have 8 quarters of recession, which are: 1999Q1, 2001Q3, 2003Q2  and 2009Q1, 2014Q2, 2015Q2, 2015Q3, 2015Q4. 11

Forecasting Methodology
We now turn to the question of how to choose the models that best forecast out-of-sample recessions.Model misspecification by itself cannot account for forecast failure: in the absence of changed economic conditions, a model's out-of-sample forecast performance would, on average, be the same as its in sample fit to the data.
If forecast failure is primarily due to forecast-period location shifts as Hendry & Clements (2002) stress, then there are no possible within sample tests of the models.Structural breaks happen all the time in the economy.Therefore, choosing models to forecast based on in sample forecast performance seems to be a great mistake.
To address these issues we utilize measures out-of-sample performance to discriminate between the best forecast models.We decided not to use Mean Squared Errors (MSE) or any of its variants as our main criteria to select models.The growing consensus among researchers who have been making comparisons among forecasting methods is that the MSE should not be used.Newbold (1993), for example, explores the deficiencies of mean squared errors as a performance measure.Thompson (1990) also concluded that MSE is not appropriate.He proposed a variation on the MSE, the log mean squared error ratio (LRM), that would be appropriate for making comparisons across series.The LMR takes the log of the ratio calculated by dividing the proposed model's MSE by the MSE of a benchmark model.
The out sample performance of our models is gauged with the so-called ROC curve as a model selection tool.The ROC curve plots the fraction of true positives (crisis = 1) that a given model signals (out of all positives in the sample) vs. the fraction of false positive signals (out of all negatives in the sample) along contiguous threshold settings.The best model according to this criterion is the one that delivers the highest trade-off frontier between true and false alarms.Such a choice will be guided by the relative cost of failing to predict a crisis vs. that of a false alarm, credibility cost.
A clear advantage of this approach over traditional model selection criteria previously used in the forecast literature is that the analyst does not have to take a stand a priori on which region of the tradeoff to pick.Distinct models deliver a distinct ROC curve and the overall "best" is the one that delivers the highest area under the curve, i.e., the higher outward frontier above the 45-degree line, where the latter traces out the good vs. false positive trade-off under random guesses.
There are other several advantages of ROC in comparison to other possible metrics of forecasting comparisons.For example, Estrella & Mishkin (1997) use out-of-sample Pseudo R 2 as a metric to compare the performance of models.As the authors acknowledge, in some cases out-of-sample Pseudo R 2 furnish negative results.This makes it a much worse metric than ROC in our view to compare the out-of-sample performance of models. 12 The ROC methodology focuses on a fundamental characteristic of forecasting, that is, its ability to capture the occurrence of an event with an underlying high hit rate, while maintaining the false alarm rate to some acceptable level.
Recent applications of the ROC curve methodology to historical data on domestic bank credit in 14 advanced countries are provided in Jordà, Schularick, & Taylor (2011), whereas Satchell & Xia (2006) present an earlier application to credit rating models.Catão & Milesi-Ferretti (2013) use it in an in-sample framework to forecast financial crisis.Yet, we are not aware of any other paper that uses it in the same way and context that we do in this paper. 13 ROC, as any other empirical methodology, has also some drawbacks.As it is only based on a forecast of binary values, it ignores the magnitude of the forecast errors.It may have low power in small samples, because it does not consider the magnitudes of these forecast errors.ROC can be understood as a criteria of unconditional evaluation, because it does not make a distinction between the existence (or not) of temporal clusters of the binary variable (Kupiec, 1995;Christoffersen, 1998).Finally, although very useful to establish a forecast ranking among different models, one cannot verify if two models produce forecasts that are statistically significant and different.
To address some of the issues mentioned above, we will compare our results with some more traditional forecast models of rare events, such as the directional tests of Pesaran & Timmermann (1992).
We are interested in selecting one to four regressors models that best forecast recessions in Brazil from 1 to 8 quarters ahead. 14Being more specific, our methodology is the following.We estimate an equation such as (1) below using a probabilistic extreme value model with only one regressor. Pr where f (extreme value function), and K = 1, . . ., 8. Our first estimation period goes from 1991Q1 to 2002Q1.Then we forecast K periods ahead (K from 1 to 8), considering levels of cutoff probabilities that range from 0.005 to 1 and that vary in each step by 0.005.If the forecast value of recession is less that the cutoff probability that we are considering 12 Lahiri & Wang (2013) stress that often-conventional goodness-of-fit statistics in probabilistic models, such as Pseudo R 2 , among others fail to identify the type of I and type of II errors in predicting the event of interest.Lahiri and Wang examine the quality of probability forecasts in terms of calibration, resolution and alternative variance decompositions.They discuss several measures of goodness of fit, like, for instance, the Brier's Quadratic Probability Score, the Prequential Test for Calibration, the Skill Score and the Murph and Yates Decompositions. 13See Catão & Milesi-Ferretti (2013) for a utilization of ROC to forecast financial crisis.
we take the forecast to be zero.Otherwise, the forecast is one.We compare these values with the values of the recessions that occurred after the estimation period. 15e then increase the estimation period by one quarter and repeat the process above for every forecast period until we reach our final estimation period that goes from 1991Q1 to 2015Q3.We then calculate the number of success (correct forecasts) divided by the total number of successes (recessions); we also calculate the number of failures (false positive signals) and divide that by the total number of failures (all periods in which there were no recessions).By doing this, we are able to build a ROC function for each model with one regressor for every K quarters ahead forecast.We then integrate this function from 0 to 1 and name this value the ROC of the model.The best models are the one with the highest ROCs for each K forecast period.
We use the regressors of the models selected with one regressor in the specifications of the models with two regressors.We repeat the methodology above for every one of these models and select the best models as the ones with the highest ROCs for every K forecast period.After selecting the two-regressor models, we repeat the process with three regressors, where two of them are the ones that proved best in forecasting.Finally, we choose the four-regressor models using the same process and considering the three regressor models selected as the basis for the four-regressor models.

Results
Table A-3 and Table A-4 (see section A.2 in Appendix A) present the best models in terms of one-year ahead forecasts.Table A-5 presents the variables that make the models and Table A-6 present the ROC values of these models.For one-quarter ahead forecasts, with one regressor the best model is the one that has energia as the only regressor.The out-of-sample ROC of this model is 0.8710.In the case of the forecasts of two quarters, with one regressor the best model is the one that has sa_sond as the only regressor.The out-of-sample ROC of this model is 0.5701.When we consider 3 quarters ahead forecasts, with three regressors the best model is the one that has icc_fecom, ibrx100fim and dloginaduse1.The ROC of this model is 0.9777.Finally, for 4 quarters ahead the best model is the four regressor model with impbk, sond_pres, ibovespa and dlogempformpub with ROC of 0.9990.
For forecasts of more than one year, the results are presented in Table A-5 and Table A-6 (see section A.2 in Appendix A).Table A-5 presents the variables that make the models and Table A-6 present the ROC values of these models.For five periods ahead, the best model is the one that has four regressors, ibovespa, sond_press, empformtot and dlogprodauto, with ROC of 0.9732.For six quarters ahead, the best model is the one with three regressors icc_fecom, balcom, empformpub with ROC of 0.9800.For seven quarters ahead, the best model is the one with 3 regressors, M2, dlogibovespa, dlogimpbk, and ROC of 0.6672.Finally, for eight quarters ahead the best model is the one with 3 regressors, icc_fecom, impbk, energia, with ROC of 0.9405.
We use the maximum variance of Birnbaum & Klose (1957) and verify that all ROC areas presented in tables A-3 to A-6 are statistically significant.As one can observe from the results, financial variables are relevant for forecasting.Particularly, those related to the stock market, like ibovespa and ibrx100fim.These variables take part as regressors in many specifications.They are observed individually over their respective primary horizons, or they may be combined to produce very accurate models in terms of out of sample forecasts.
In general, prices of financial assets are supposed to contain expectations about the future path of the economy.The most convincing theoretical foundation of this assumption is the expectations theory of the term structure.The expectations hypothesis postulates that, for any choice of holding period, investors do not expect to realize different returns from holding bonds or bills of different maturities.
Not consistent with the findings of Estrella & Mishkin (1997), Estrella & Hardouvelis (1990), Bernard & Gerlach (1996), and Plosser & Rouwenhorst (1994), we have not shown that the term spread has significant information content for forecasting recessions in Brazil.The term structure of interest rate is an important leading indicator for recessions in USA.
Some real sector indicators seem also relevant to forecast.The series energia, empformtot and empformpub are the ones that are more important.Their appearance in the best models is expected, because they reflect earlier than other real sector variables the possibility of a recession in the near future.
Confidence indicators and some monetary aggregates also play special roles in forecasting.In the case, of confidence indicators, we ponder that this may occur because households are getting better in understanding the dynamics of business cycles.In the case of monetary aggregates, we think the reason may be related to the high demand for public bonds in Brazil.
In sample results are based on equations estimated over the entire sample period.Their predictions or fitted values are then compared with the actual recession dates.Three types of results are presented: an in-sample ROC, a Pseudo R 2 , and a MSE.We present the statistics of the same models we selected from the out sample forecasts analysis above in Tables A-7 and A-8 (see section A.3 in Appendix A).The in sample forecasts measures give a different indication of the forecast capacity of the models selected.Some models that have better out-of-sample performance, do worse if we consider in-sample measures.Again, using the Birnbaum & Klose (1957) statistic we verify that all ROC values are statistically significant.
In Table A-9 and Table A-10 (see section A.4 in Appendix A), we present the Pesaran & Timmermann (1992) statistic of the directional tests, which gives an idea of how well our models selected are good in forecasting change in direction of the variable of interest and the MSE statistic associated with each one of the models.Only a few of the statistics are not statistically significant (those in boldface).The results show clearly that the great majority of the models selected with the ROC criteria reject the null Hypothesis of not being able to forecast the changes in directions.

CASE STUDIES: FORECAST INDEXES WITH OUT OF SAMPLE ROC AND RECESSIONS IN BRAZIL FROM 2000Q1 TO 2012Q4
Predicting the future is a tricky business.A good example of what may happen is provided by the experience with the Stock & Watson (1990) leading indicators.Stock & Watson (1993) describe and analyze the disappointing performance of their indicator predicting the 1990-1991 recession.
Here we examine the performance of our chosen forecast models to predict recessions in Brazil in the period from 2000Q1 to 2015Q4.We consider the eight models that gave us the best out-of-sample ROC for each forecast period.We construct 4 indexes.The first one (Index1) is an equal weighted average of the forecasts of our best models in terms of ROC for each forecast horizon, from one to eight quarters.The second one (Index2) is a weighted average of our best forecast models (the one quarter ahead forecast with weight equal to eight and the others with weights decreasing until one).The third one (Index3) is an equal weighted average of the best ROC models selected (one to four regressors) for all horizons.The fourth indicator is a weighted weighted average of our best forecast models (the one quarter ahead forecast with weight equal to four and the others with weights decreasing until one).
Our comparison analyses are graphical and are of three types.We look at how these series behaved individually to forecast the recessions.We compare our forecasts with those made by a leading financial indicator of GDP that we built.Finally, we look at how our forecasts compare with those made by the market and for this we use the GERIN database of GDP forecasts in Brazil the market.
In Figures B-1 to B-3 (see Appendix B), one can see that our indicators do a good job in forecasting recessions.The indexes, their average as well as the maximum and minimum increase in relevant ways in all recession quarters.This seems to be evidence that they are anticipating recessions.They seem relevant as warnings of recessions.
We also build a leading financial indicator index based on Index of Economic Activity, Brazil (IBC-Br), that incorporates the pathway of the variables considered as proxies to the development of three most important economy sectors (agriculture and livestock, industry, and services).
To build the index, we considered the same 24 financial series we used in this paper.Initially, we calculated the current and lagged correlations between these series and the first difference of IBC-BR seasonally adjusted.Then, these series were submitted to Granger causality tests to find the final selection: end of period monthly return of ibovespa ibrx100fim lagged 2 periods.Figure B-4 presents the dynamics of this index together with the average of the indexes we built with ROC.The figure shows that our recession indicators do a better job in forecasting recession in all cases.
Finally, we compare our forecasts with the market forecasts.The Central Bank of Brazil collects every week forecasts of market participants with respect to five or four quarters ahead in time of GDP growth in Brazil.We create a market signal variable of recession if the market forecasts two consecutive quarters of negative growth.Otherwise, this variable is zero.Then we take the quarterly average of this variable.We create two market signals, one is an equally weighted average of the quarterly forecasts and the other is a weighted forecast (the one quarter ahead forecast with weight equal to four and the others with weights decreasing until one).

CONCLUSION
Economic forecasting that allows for structural breaks and misspecified models has radically different implications from one that considers stationary and well-specified ones.It is well known by now in the literature that models that are well specified in-sample may perform very poorly out sample.There are many reasons for this, but maybe the most important is the occurrence of structural breaks in out-ofsample.
In this paper, we examine the usefulness of various financial and real sector variables in out-ofsample predictions of whether or not the Brazilian economy will be in a recession between one and eight quarters in the future.Variables with potential predictive content are selected from a broad array of candidates and are examined by themselves and in some plausible combinations.
The models selected, in our view, adapt quickly after any shift is discovered, therefore avoiding systematic failure of forecasting.We think they capture some of the robustness characteristics of the models that win forecasting competitions.
The predictive out-of-sample capacity of several models vary depending on the numbers of quarters ahead to forecast and on the number of regressors used in the model specification.
We think that our results are relevant for the literature of forecasting rare events, such as recessions.The best models selected can be thought as early warning signals of recessions in Brazil.
Of course, we do not propose that these indicators substitute macroeconomic models and judgmental forecasts.Rather, we conclude that our selected models can usefully supplement the former models and other forecasts, and can serve as a quick, reliable check of more elaborate predictions.

A.1. Descriptive Analysis of the Database
Our sample has quarterly data and goes from the first quarter of 1991 to the fourth quarter of 2015.We have 84 real sector variables.Table A-1 shows the real sector variables.We use the level and the first difference in logarithm of the series variables.When possible, we also use the seasonally adjusted series in level or in the first difference in logarithm.Table A-2 shows the financial sector variables.We have 26 variables.Once more, we use the level as well as the first difference in logarithm of these series.

A.2. Out-of-sample ROCs
Our sample has quarterly data and goes from the first quarter of 1991 to the fourth quarter of 2015.Our leading indicators of recessions are composed of 83 real sector variables and 24 financial variables.We use both levels and first difference of these variables seasonaly adjusted and non seasonaly adjusted.We use a probabilistic extreme value model with only one regressor.
Our first estimation period goes from 1991Q1 to 2002Q1.Then we forecast K periods ahead (K from 1 to 8), considering levels of cutoff probabilities that range from 0.005 to 1 and that vary in each step by 0.005.If the forecast value of recession is less that the cutoff probability that we are considering we take the forecast to be zero.Otherwise, the forecast is one.We compare these values with the values of the recessions that occurred after the estimation period.We then increase the estimation period by one quarter and repeat the process above for every forecast period until we reach our final estimation period that goes from 1991Q1 to 2015Q3.We then calculate the number of success (correct forecasts) divided by the total number of successes (recessions); we also calculate the number of failures (false positive signals) and divide that by the total number of failures (all periods in which there were no recessions).By doing this we are able to build a ROC function for each model with one regressor for every K quarters ahead forecast.We then integrate this function from 0 to 1 and name this value the ROC of the model.
The best models are the one with the highest ROCs for each K forecast period.We use the regressors of the models selected with one regressor in the specifications of the models with two regressors.We repeat the methodology above for every one of these models and select the best models as the ones with the highest ROCs for every K forecast period.After selecting the two regressor models, we repeat the process with three regressors, where two of them are the ones that proved best in forecasting.
Finally, we choose the four regressor models using the same process and considering the three regressor models selected as the basis for the four regressor models.Table A-3 shows the one year ahead models with the best forecasts and Table A-4 shows their ROCs.Table A-5 shows the two-year best models and Table A-6 shows their ROCs.
Table A-3.Models of one year ahead forecasts.Pr for each model with one regressor for every K quarters ahead forecast.We then integrate this function from 0 to 1 and name this value the ROC of the model.The best models are the one with the highest ROCs for each K forecast period.We use the regressors of the models selected with one regressor in the specifications of the models with two regressors.We repeat the methodology above for every one of these models and select the best models as the ones with the highest ROCs for every K forecast period.After selecting the two regressor models, we repeat the process with three regressors, where two of them are the ones that proved best in forecasting.Finally, we choose the four regressor models using the same process and considering the three regressor models selected as the basis for the four regressor models.

Pr
The t-statistics of the Pesaran & Timmermann (1992, 2009) directional tests are presented in tables A-9 and A-10.
Figure B-5 shows the market signal are slightly better in predicting the 2003Q2 and 2009Q1 recessions but that our indicators are much better in predicting the more recent recessions, 2014Q2 and 2015Q2, 2015Q3.

Figure B- 2 .
Figure B-2.Recession probabilities with indexes of equal weighted average of forecast of best ROC models from 1 to 4 quarters (Index3, Index4).

Figure B- 5 .
Figure B-5.Market forecasts of recessions and models selected with out-of-sample ROCs.

Table A -
1. Real sector variables.

Table A -
2. Financial sector variables.

Table A -
4. ROCs of one year ahead models.

Table A -6. ROCs
of one year ahead models.

Table A -
9. One year ahead.

Table A -
10. Two year ahead.Recession probabilities with indexes of equal weighted average of forecast of best ROC models from 1 to 8 quarters (Index1, Index2).