Economic and financial models have come in for a lot of criticism in the context of the global financial crisis, much of it deserved. One of the primary targets is models that financial institutions widely used to (mis)estimate risk, such as Value-at-Risk (VaR) models for measuring risk exposures (which we’ve discussed elsewhere) or the Gaussian copula function for quantifying the risk of a pool of assets.
In September, the Subcommittee on Oversight and Investigations of the House Science and Technology Committee held a hearing on the role of risk models in the financial crisis and how they should be used by financial regulators, if at all. The hearing focused largely on VaR models, which attempt to quantify the amount that a trader (or an entire bank) stands to lose on a given day, with a certain confidence level. (For example, a one-day 1% VaR of $10 million means that on 99% of days you will lose less than $10 million.)
Although the witnesses ranged from Nassim Taleb, who has been arguing for years that VaR models are toxic, to Gregg Berman, who heads a company that develops VaR models for customers, there was surprising agreement on the problems of VaR. As Richard Bookstaber put it, VaR depends on three assumptions, all of which are generally false: not all assets, particularly illiquid ones, are included in the VaR calculation; estimates are based on past data that is unrepresentative of the future; and because financial returns exhibit “fat tails” (extreme outcomes are more likely than you would expect), VaR estimates tell you very little about how bad things can get that last 1% of the time.
The question, then, is what to do about it. One thing that seems clear is that risk models that are designed to function in normal market conditions should not be relied upon to predict outcomes in times of crisis. On this account, VaR doesn’t kill banks; executives who don’t recognize the limits of VaR kill banks. As Bookstaber put it, “one has to look beyond VaR, to culprits such as sheer stupidity or collective management failure: The risk managers missed the growing inventory [of risky assets], or did not have the courage of their conviction to insist on its reduction, or the senior management was not willing to heed their demands. Whichever the reason, VaR was not central to this crisis.”
Given that everyone is agreeing sophisticated risk models are worthless in crises, it seems particularly remarkable that regulators allowed some banks to use their in-house models in determining their own capital requirements – since one of the purposes of capital requirements is precisely to provide a cushion that protects banks (and their creditors, and taxpayers) in the event of a crisis. The obvious solution is that regulators should rely on cruder constraints, such as an absolute limit on leverage that banks cannot arbitrage around (one of the recommendations of Treasury’s recent white paper on capital requirements, which we discussed here), or periodic stress tests that estimate how bank asset portfolios will perform in a real crisis.
But there is a more interesting question to ask as well: why did VaR become so popular? It’s important to remember that competition among models is shaped by the human beings who create and use them, and those human beings have their own incentives.
David Colander made this point about economic models: the sociology of the economics profession gave preference to elegant mathematical models that could describe the world using the smallest number of parameters. “Common sense does not advance one very far within the economics profession,” he says.
A similar point can be made about VaR models. Sure, maybe all the financial professionals who design and work with VaR know about its shortcomings, both mathematical and practical. But nevertheless, using VaR brought concrete benefits to specific actors in the banking world. If common sense would lead a risk manager to crack down on a trader taking large, risky bets, then the trader is better off if the risk manager uses VaR instead.
Not only that, but imagine the situation of the chief risk manager of a bank in, say, 2004. As Andrew Lo has argued, if he attempted to reduce his bank’s exposure to structured securities such as CDOs, he would be out of a job; VaR gave him a handy tool to rationalize a situation that defied common sense but that made his bosses only too happy. And at the top levels, CEOs and directors who probably did not understand the shortcomings of VaR were biased in its favor because it told them a story they wanted to hear.
In other words, models succeed because they meet the needs of real human beings, and VaR was just what they needed during the boom. And we should assume that a profit-seeking financial sector will continue to invent models that further the objectives of the individuals and institutions that use them. The implication is that regulators need to resist the group think of large financial institutions. If everyone involved is using the same roadmap of risks, we will all drive off the cliff again together.
By Simon Johnson and James Kwak
This is a slightly edited version of a post that first appeared on the NYT.com’s Economix, and it is used here with permission. Anyone wishing to reproduce the entire post should contact the New York Times.
The Baseline Scenario 
This kind of risk model which seeks to limit the risks of the casino presupposes the casino.
So it’s of course the political gambit and self-hypnosis of those who profit from the casino, and of corrupted/captured (those two terms mean the same thing, BTW) “regulators” who take it for granted that the casino should exist at all.
Real reform would not try to make the casino less risky, it would shut it down completely, forever.
VAR proved toxic in the valuation of mortgage backed securities. There was no history of residential mortgage lending to homeowners lacking a capacity to repay. Thus, the modeling was built upon sand. Whether or not VAR is useful in modeling securitizations of corporate loans depends upon whether any lending standards exist. The same problem arose with Milken’s junk bond research. It was used to justify the creation of junk, even though the research was limited to fallen angels. The real issue is what activities ought to be conducted using insured deposits. Separate commercial and investment banking and then let the investment banks play whatever games they like.
Simon,
I’d commend to you a report by the Senior Supervisors Group titled Observations on Risk Management Practices During the Recent Market Turmoil that was published in March 2008. Specifically, section V.b&c have a discussion of the supervisors observations on VaR, stress tests, scenario analysis, and other tools. The report can be found here: http://www.newyorkfed.org/newsevents/news/banking/2008/rp080306.html
The report stresses that identifying risks cannot be done just through models, but requires a firm-wide culture of communication and dialogue. The report also has an extensive section on risk measurement.
I’ll conclude with a few sentences from the report:
“Firms that avoided such problems demonstrated a comprehensive approach to viewing firm-wide exposures and risk, sharing quantitative and qualitative information more effectively across the firm and engaging in more effective dialogue across the management team.”
“They had more adaptive (rather than static) risk measurement processes and systems that could rapidly alter underlying assumptions to reflect current circumstances; management also relied on a wide range of risk measures to gather more information and different perspectives on the same risk exposures and employed more effective stress testing with more use of scenario analysis.”
“Firms that tended to avoid significant challenges through year-end 2007 typically had management information systems that assess risk positions using a number of tools that draw on differing underlying assumptions.”
“Most importantly, managers at better performing firms
relied on a wide range of measures of risk, sometimes including notional amounts of gross and net positions as well as profit and loss reporting, to gather more information and different perspectives on the same exposures.”
“Firms that encountered more substantial challenges seemed more dependent on specific risk measures incorporating outdated (or inflexible) assumptions that management did not probe or challenge and that proved to be wrong.”
I have worked for a number of institutions (not named here) which were involved in much of the recent crisis. My bias out in the open, I must say that the risk culture between the different firms is strikingly different. At one of my previous employers, risk was pretty much entirely a back-office operation. The right didn’t really know what the left hand was doing, i.e. the people charged with managing the risk tended to know very little about the underlying products they were managing and were removed by uncomfortably many steps from those in the front office making the trading/investing decisions.
It seemed bad, but I didn’t realize how bad (or rather, it didn’t quite sink in), until I saw how it was done elsewhere– those managing risk are intimately involved and very proximate to the decisions being made. This is refreshing to see, but it is rather unfortunate to know that TBTF institutions are essentially doing the equivalent of using nuclear power in the stone age; it’s essentially magic until it catastrophically breaks down.
To cite one example from my previous employer, when criticizing the way risk was being computed, the response I got boiled down to “we have a stable risk profile and we don’t want to change that. The sensitivities have the right sign!” That, frankly, was shocking to me. Anyone could tell you the right sign, but the magnitude is what matters!
The problem is with people, not models. People decide what they want, and then justify their actions.The best models are simplifications of reality. Model
assumptions based on a wide range of plausible scenarios are possible. Decision-makers use models to justify their actions.
The Federal Reserve is a perfect example. For the past 20 twenty plus years the Fed has looked in the mirror every morning and congratulated themselves on the splendid job they had done in producing the great economic “moderation”. During the same period the Fed produced one bubble after another leading to the current financial crisis. That crisis was exacerbated by the Fed’s failure to regulate.
The Fed’s failures are not a result of bad Fed models.
The Fed failed because of Fed malfeasance and failure to take responsibility:
1. The Fed failed to regulate because they did not
believe in regulation.
2. The Fed believes in not taking responsibility –
whether it be derivatives,bubbles, or regulation.
Read these comments by Fed Vice-Chair Kohn on the housing bubble and models (speech November 19, 2008):
“…the Federal Reserve examined whether house prices were overvalued and arrived at a wide range of answers. For example, one set of models that linked rental rates and house prices indicated as early as 2004 that the market was significantly overvalued, while another set of models, suggested as late as December 2005, that house prices could be justified by fundamentals.”
The Fed picked the model that works best for them-the one where they don’t have to take responsibility, the one that pleases the crowd, politicians, “give them what they want” philosophy.
VaR models and the Gaussian cupula function strike me as a kind of artificial intelligience that “mere” mortals placed too much confidence in. The financial meltdown being an indication of the limitations of AI.
Well, the first thing to do is to throw out all models based on gaussian distributions, since, empirically, we know that financial markets do not exhibit them. Seems like a good, logical first step.
Cheers,
Carson
I think it is true that 12-15 years ago, management and traders tended to view these models more literally than made sense. But that practice died in 1998. Models have always been a convenient mechanism to approximate risk, almost like an adjunct to a balance sheet or income statement–also tools not to be taken literally.
To think this crisis had anything whatsoever to do with models or that “better” models will prevent the next crisis is of course misguided. Uncertainty and complexity abound. Common sense also goes missing. I am a broken record on this point……but the optimal way to protect a system in the long run is to force consequences on all participants. Instead, we have the Fed buying mortgages, the government subsidizing home purchases, the government propping up prices as well as institutions, etc., etc.
The next crisis (aren’t we still in the ‘last” crisis?) will look completely different. We cannot prevent them. What we can do is eliminate artificial incentives which help propagate them. I just doubt that models have much to do with it.
Nice post, and the Wired article was a fun read. There are, perhaps, a couple points it misses:
– The examples of unstable correlations it discusses (e.g. two companies, one launches a media campaign and the correlation changes) are really not the kind of changes one worries about. Sure, if you try to pair securities that can happen – and viewing the gamma between two pairs of items as perfectly stable is silly. But this was not what killed CDOs. The gamma for a batch of 10,000 mortgages is much less vulnerable to pairwise changes. And the article talks about sensitivity to tiny changes, but arguably this wasn’t really the problem either (unless you were massively leveraged on a concentrated position).
– Here are three deeper problems:
1) The Uber-Gamma: Call this the gamma of gammas. Essentially, it’s the state of the world. Financial models, and most modelling in general, likes – or, to be more accurate, needs – to assume somewhat continuous distributions that usually have nice properties. The Wired article talks about “fat tails”, but that’s not exactly what goes on.
Consider this: If you were to take the history of quarter-on-quarter housing price changes and plot them, you get a bell-curve-like distribution with “fat tails”. But that is far too kind to the model… it presumes that each of those quarterly price points really is its own data point. In practice, they are part of a couple big trends. A boom trend, and a bust trend.
The real world is discontinuous, or multi-modal. Game theory sort of gets this with its notion of multiple Nash equillibria (but doesn’t spit out “tractable” models, and often requires hyper-rational actors, and has other problems).
2) The past predicts the future. Or, more accurately, the past will look like the future. Or, even more accurately, the future will look like the IMMEDIATE past (past decade or two, during which we have good data).
I can’t remember all the times I’ve heard people referring to long-term US growth trends, and expressing faith in things like automagic recovery from recession (the “self-healing” properties of markets, like, you know, the credit markets). Using US, or even OECD, data creates a sample selection problem. We’re looking at the _survivors_, and making conclusions about survival rates!
That’s like looking at climate data and arguing there is no evidence the world is going to warm 7 degrees because it hasn’t yet warmed 7 degrees.
3) Excessive trust in markets’ predictive capabilities. What’s funny (as in, laugh out loud funny) about the Wired article – and I hadn’t known until I read it – was that they were estimating correlation parameters from CDS market price data on the debt… and the market pricing was largely determined (in the short run, during which correlation parameters are estimated) by the same models they were using to fit their data. The word “circular” just doesn’t do this justice.
—-
Anyway, as SJ and JK and the other articles note – people KNEW these problems. The truth was out there. The truly fundamental problem wasn’t that the truth wasn’t known, but that there were multiple truths – and this truth was easy, convenient, and advantageous to believe. The other truths were inconvenient. Moreover, the critics – regulators and academics – had every incentive (career, prestige, reputation) to act as cheerleaders rather than skeptics.
Carson Gross, IIRC you are a computer scientist in the Bay Area. Do you believe artificial intelligence is within the realm of possibility?
“The implication is that regulators need to resist the group think of large financial institutions”
Yes and why not elect Dr Pangloss as a lifetime regulator for financial risk?
FWM made some good points
People respond to incentives.
New rules and regulations will just provide a legal framework for the next generation of tinkerers to game the system
a larger problem is that a country that enjoys healthy organic growth can subsidize the losses from an imperfect financial system. The US used to have healthy organic growth. No longer. We as a nation are over indebted and have a close to 0% real growth rate prospect. The answer is to delever. Do you see any of that going on? The government has turned into a more agressive version of Countrywide…if you have a heartbeat here’s a loan. You can waste your time with pathetically cathartic financial reforms but they will do no good as long as we as a society are floating on a sea of debt
People respond to incentives. How do you think debt changes the incentives for people?
Isn’t another ‘feature’ of VaR the uncorrelated nature? If you have 1% chance of losing $10M today, and you hit that 1/100, tomorrow you’re still 1/100. The $10M loss won’t kill you and the probability of 3 straight $10M loss days is 1/1,000,000, when the day after your $10M loss, you’re more likely to have another – general downward trending market.
I would suggest it is a more basic issue that as a rules-based society, we crave anything that yields a precise number, whether accurate or not.
Take driving, an outrageously high cause of death in the US. There are many inputs that go into driving safely, from a driver’s inherent coordination and reaction time to environmental conditions and traffic density. Yet virtually all driving enforcement is focused on one of two metrics: exceeding a given speed over the ground, as measured by radar or laser; exceeding a given blood-alcohol level, as measured by a breathalyzer.
VaR stuck for the same reason the batting average (or, perhaps these days, OPS) was trotted out for fifty years as the key metric of baseball greatness: it gives the false confidence of a specific number. “Joe Smith batted .267 last month, he is clearly better than Mike Stevens, who batted .254, release Mike and get me Joe.”
Before we had VaR, we had managers who were sure Nick wouldn’t lose money because he looked like an upstanding guy. After VaR, we had managers who were sure Nick wouldn’t lose money because a number told them he couldn’t. In both cases, the flaw was simply that they didn’t have the external information to decide and were making a prediction based on whatever scraps they had. You cannot write a regulation to “make better decisions”, so the government writes them to require use of a specific method. That the specific method does little good is hardly the point.
I think this is misguided. The point of building models is to be able to quantify measures you believe are important. Consider the following new model– take the result of an old model, X, and output “VaR is at least X; In fact, I think the VaR is considerably greater due to tail risk.” Is this model any “smarter” than the first one? Of course not, but this is essentially what all the criticisms of models boil down to and everyone Oohs and Aahs at how smart they are. Anyone can say that the models are understating the risk, but no-one is willing to put a number on it, because, frankly, no one CAN really put any confidence behind a single number. This has nothing to do with the limitations of AI.
Here’s a stray thought, btw:
It turns out JK’s post from yesterday (https://baselinescenario.com/2009/09/30/how-to-waste-45-minutes-like-i-just-did/) and this post are connected.
If you read Nate Silver’s description of why the school survey data is too clean (aka, the tails aren’t fat enough, because in truth performance across questions should be correlated with quality of studet, and this means data points are non-independent and “move in groups”), this is (essentially) the same problem with VaR…
‘You cannot write a regulation to “make better decisions”’
I don’t even believe that you can write a regulation that will prevent people from getting into trouble. You can change the risk/reward of their behavior by saying that if you do this, you will face death by a firing squad. But given enough money at stake, some people still would do it. As your mother said, if all your friends were jumping off a cliff, would you follow? It depends on how popular I want to be.. or maybe how close the bear that was chasing us was. Peoples are different enough that somebody is going to try their best to game the system. And someone is going to optimize it except for the terminal case.
What was the argument again for a systemic risk regulator? People seem to think that a regulator can see and effectively respond to bubbles. A regulator is not going to prevent the mispricing of risk. The only way to limit the damage is to concentrate on what happens when everything goes wrong (failure mode and effects analysis). If the answer is that the system disintegrates, then the rules need to be changed so that it isn’t a terminal case.
Good luck getting that line of thinking through a political process.
Humans have evolved. Why couldn’t anything else?
Well this guy (hardly an expert) says: Hang on just a few years
http://www.computerworld.com/s/article/9138726/Nanotech_could_make_humans_immortal_by_2040_futurist_says
Uploading your personality… sounds familiar. Hmmmm.
Why? All it would take is some well-compensated lobbyists. Pony up?
Completely agree. There should be an everyday name for cherry-picking statistical models.
I disagree slightly – the drive to parsimonious models was not simply driven by a cultural phenomena that is inherent in a rules based society (craving hard numbers).
It was driven by a desire to reduce complexity of trading instruments to allow mechanistic pricing of risk, because in doing so the financial engineers were able to create (and rapidly grow) a new and profitable trading market. There were other models, but this one was simple enough to facilitate trading. As such, it was USEFUL, (not in the social sense, but in the narrow sense that it was profitable for those using it).
They picked the model that was most advantageous to them.
And, certainly, one can argue these models have value. But the problem comes in this – people began to conflate the measure of the thing (risk) with the thing itself. Management does not like complexity when it disagrees with their bonus package.
Consider this from another angle:
Industrial Organization has a subfield in optimal incentives in labor contracting. One of the key debates is whether to use relational contracting (salaries), or incentive contracting (piece rates). One key to answering this question is whether you can develop an incentive mechanism that correlates well with marginal effort.
If you fail, then employees have a tendency to deploy effort to maximize the MEASURE of their performance, rather than generate real value. Welcome to Corporate America.
In essence, it’s like the “Teaching to the Test” debate in education.
Our entire credit market was “Teaching to the Test”, but the Test was not a good measure of the thing it was trying to measure.
Carson Gross: “Well, the first thing to do is to throw out all models based on gaussian distributions”
Back when I used to play options, I used a Chebyshev approach, because it makes no assumption about underlying distributions. It also gave me fairly high error estimates, but I figured that that was a good thing. ;)
The problem is that you can play stupid games with VaR even if the financial market is a perfect Gaussian distribution.
The martingale betting strategy (if you lose, double your bet, repeat until you win, then start over on your bet) has been invented many times and is foolish because while infinitely often you’ll win, infinitely often you’ll have a streak of bad luck of any given length. So you’re guaranteed to eventually run out of liquidity, assuming finite liquidity.
1% VaR in a day? I’ll give you a 0.1% VaR of $0 for a day.
I’ll just bet that a fair coin flip will come up “Heads” up to ten times in a day. I start with a starting bet, if I lose I double it, and when I finally win I stop for the day.
Chance of losing any money at all requires all 10 “Tails,” which is 1/2^10 = 1/1024 < 0.1%
My one day 0.1% VaR is $0. Look, Mr. Executive, no risk at all!
But those are irrelevant to gaming VaR. VaR can be games without any of those factors, if you follow something akin to the martingale betting strategy.
Any executive (or trader) that relied solely on VaR should not have their job at all. VaR is perfectly vulnerable to the martingale betting strategy, and shows no risk by it.
Heck, with betting on a perfectly fair coin flip, I can give you a strategy that shows zero risk with VaR but if practiced, would inevitably exhaust all your liquidity and cause you to go bankrupt.
1) Bet $1B that a coin comes up Heads.
2) If it comes up tails, bet $2B.
3) Keep doing this until you either win, or you’ve bet ten times in the day.
4) The next day, start over at $1B if you stopped with a win yesterday, or bet double your previous bet if you ended with a loss (of $1.024 T).
Notice that the only time you lose any money is when you have ten Tails flips in a row. That happens 1/2^10 = 1/1024 < 0.1% of the time. So my one-day 0.1% VaR is $0! In fact, for any time period, you can pick a percentage such that VaR is $0! No risk at all, if you only use VaR! Never mind that if you do this strategy, you should expect to need $1.024 T in liquidity after about three years, maybe more, maybe less.
The awesome thing about this strategy is that the one day VaR looks EVEN BETTER the more bets you let yourself make in a day, even though that actually has no effect on how long it takes to need $1.024 T in liquidity. (Which happens if the first ten flips are all tails in a day; that you would make further flips is irrelevant if you can't actually come up with the liquidity.) Allow yourself to make 20 flips in a day, and you can truthfully say that the one-day 0.0001% VaR is $0, even though you'll still expect to need $1.024 T in liquidity some time in three years, and it will happen eventually. (And you could need even more liquidity.)
Congratulations! You've invented a zero-risk investment that, by altering the initial bet and number of flips, can have arbitrarily high return and over any arbitrary time period can have any arbitrary percentage VaR of $0.
The distribution of the largest amount of liquidity you'll need in a day is basically Zipf's law (a discrete counterpart to the Pareto distribution). As long as you limit yourself to a finite number of bets in a day, it's not heavy-tailed. The limit as time goes to infinity, it is heavy-tailed.
Note that the “three problems” don’t apply here:
1. not all assets, particularly illiquid ones, are included in the VaR calculation – not a problem, all assets are included.
2. estimates are based on past data that is unrepresentative of the future – not a problem, this is a perfectly fair coin, past data is perfectly representative of the future.
3. financial returns exhibit “fat tails” (extreme outcomes are more likely than you would expect), VaR estimates tell you very little about how bad things can get that last 1% of the time– the distribution of the amount of *liquidity* needed in a day is not fat-tailed, but the amount of liquidity needed if you adopt this as a general strategy forever definitely is. However, the distribution of *returns* for any finite limit of liquidity– and you will have a finite limit– is NOT heavy-tailed by the strict definition.
You have to consider the maximum leverage you can get to, the maximum amount of liquidity you might need to execute your strategy.
I know this is a digression, but I just can’t resist!
Back in the 60’s, a good friend of mine entirely dismissed any skepticism about AI by saying “the question isn’t whether computers can be made to think, the question is will they be happy?”
But to my mind, this is in fact a strong suggestion that AI, in the sense that most people think of the term, is _not_ possible. Of course, nobody has ever given a real definition of AI. The Turing test perhaps comes closest–but I think it, too, does not capture what most people have in mind. If you believe in cybernetic theory, then the brain is, as Minsky says, simply a “meat machine” and it can, in principle, be emulated in silicon or any other medium. But I don’t think this is what most people mean by AI. Because an emulation of the meat machine would, like the meat machine itself, exhibit not only creativity and intuition and cleverness, but also moodiness, fatigue, arrogance, depression, anger, stubbornness, and the whole range of human mental problems. It would have its own desires and needs and try to pursue them. Nobody wants a machine that does what we already do, only faster. (And if we had such a machine and used it to our ends, would it not be a new form of slavery?)
What people fantasize about AI is a machine that has analytic skills combined with creativity, understanding of human values, ability to learn, and other positive human mental attributes, but without our mental downsides. But it is by no means clear that such a thing is possible. My hunch is that it isn’t, that no algorithm can exhibit intelligence and creativity without also exhibiting a broad spectrum of emotion–just as no algorithm can solve the Halting Problem.
Models have just been used as a way to defend bad decisions. The greater the number that used it, the easier it became to use a particular model.
A lot of people may have known that the models they used didn’t make any sense, but it served their short term or medium term purpose, and so it was all fine.
Re: “In other words, models succeed because they meet the needs of real human beings, and VaR was just what they needed during the boom. And we should assume that a profit-seeking financial sector will continue to invent models that further the objectives of the individuals and institutions that use them. The implication is that regulators need to resist the group think of large financial institutions. If everyone involved is using the same roadmap of risks, we will all drive off the cliff again together.”
I beg to differ with just one word… the models meet the needs of SOME people. However this is not what economics is about. It is exasperating to see how business thought drives and infiltrates our culture and education at all levels.
What happened to the struggle for objective, disinterested analysis? It is as if there is a sex appeal in the brute power of business models that helps keep everyone trapped in a compromised state of thought and dependent on antiquated theory incapable of explaining how the complexities of finance and modern markets affect humanity and our environment. Somebody needs hold onto hope that economic theory may evolve to become smarter than the models. If it does not evolve, than it is for certain that ‘we really will all drive off the cliff together’! Wouldn’t that be just plain stupid?
Without the critical tools of scientific method, consensus and testing, model-building cannot produce accurate models. Isn’t this a version of the same problems of the discipline that are now being fought out at great length? Economists stopped checking the facts and stopped checking each others work. This could not lead to good results.
Put another way, academic economics did not have the sociological order of a science, and financial economics did not have the scientific basis of an engineering discipline. It is no wonder that both failed.
I agree with you that VaR has utility as a measure; it tells you something. But, to take the Teach to the Test metaphor, it’s a bit like deciding that since math tests are easier to score than language tests, you will award all grades on the basis of math tests.
Suppose a call center evaluated employees based on calls per hour. It’s a better method than simply saying “Sally looks like she’s on the phone a lot.” But it is insufficient, because it does not take into account the fact that Jane might be closing a sale with every call and Sally might just be screaming obscenities and hanging up.
The simplicity of VaR made it easy for CEOs to say “we’re in no danger; we have x VaR and 30x book equity, we’ll be fine”, and in their laziness they stopped looking at all of the difficult-to-quantify things that could blow up a firm.
And there is the unaccounted for problem of asymmetrical payouts exhibited by trades but not by coin tosses.
And what about the Cash for Clunkers program. It is a bit like paying people to buy new cars. It gives the auto industry a nice boost. But what if the economy does not pick up after the program ends? Where is the next innovation (or reliable standard) for real economic growth?
I thought of the artificial intelligence analogy because both AI and the VaR models might work if the computer programs could build in the complexity of the real world.
In reading the post and the links it was quite shocking that the Basel Committee seems to have “embedded” VaR models into its regulation. One could argue there was a kind of “fundamentalism” at work. A faith these models are somehow superior to the vagaries of human experience, judgement and intuition.
Oops, reply should be posted here.
I thought of the artificial intelligence analogy because both AI and the VaR models might work if the computer programs could build in the complexity of the real world.
In reading the post and the links it was quite shocking that the Basel Committee seems to have “embedded” VaR models into its regulation. One could argue there was a kind of “fundamentalism” at work. A faith these models are somehow superior to the vagaries of human experience, judgement and intuition.
I’m in the middle of reading J Fox’s “The Myth of Rational Markets” (just taking a break here) and his account of the sweep of business models in 20th century economics… Part of the excitement associated with these models was that they appeared to use ‘scientific method and consensus and testing’. Fox mentions the “ketchup economists” and Warren Buffet’s success was held up as an example of the power of “testing models”…
But it is precisely because economics does not have a scientific basis that it was so vulnerable to the “ketchup economists” and others who seemed to have ‘one up’ on ‘plain vanilla economists’ by virtue of putting models into practice.
However, does that mean that we should all give up hope that the non-scientific basis of economics might be re-visited and successfully challenged or built upon? Why not? But if so that is a terribly depressing idea … If we don’t have hope in the world of ideas…then ….well – gulp – might as well line up with everyone else to drive off the cliff!!
Well said, Stats. As to your Uber-Gamma observation, you’re in good company. Check out John Authers interview with Benoit Mandelbrot in the FT
The Future of Investing (video interview)
http://www.ft.com/cms/93ece7c0-07af-11dd-a922-0000779fd2ac.html?_i_referralObject=10195731&fromSearch=n
I think that you are correct. Without meaningful re=regulation bankers are just going to continue to take risks that can bring about large short term profits but even larger long term losses. It is the problem with having a social class that gambles with other people’s money. Regardless of whether the money is retirement savings or taxpayer funds letting bankers keep the profits of good trades while others absorb the losses from the bad ones is a recipe for disaster.
This is so right on & well written – I very much hope it’s widely read, esp in DC. I’m astounded how you two can write such insightful, informative posts on a daily basis.
P.S. I just wanted to underscore that very few, if any sophisticated financial professionals / execs truly believed in VAR by late ’07 – as the last two paragraphs make clear. VAR provided a convenient, plausible rationale for disastrously misaligned compensation incentives – both at an individual & bank/corporate level.
So put basically: bankers that want the government to tell them they can keep making obscene amounts of money (regardless of reality), and that want reality itself to tell them they can keep making money (regardless of circumstances) were using models guaranteed to continue to give them the go ahead to pull in obscene bonuses regardless of the actual ramifications.
Am I getting this right?
If so then my next question is one of motivation: personal gain with reckless disregard for others, or rose-colored glasses that simply make them unable to comprehend the consequences of their overly optimistic hopes?
Not quite sure I follow – do you mean that a TBTF institution could wqin using a martingale betting strategy to game the system because of the downside “insurance”, which it buys (or gets from government) at a too-cheap price?
OK, I see what you are getting at…
Stats Guy,
There already is. It’s commonly referred to as a “damned lie”. Not sure if that’s listed in a W. Edwards Deming book or not.
So, the moral of the story is, find a mathematical concept/formula(e) which results in an outcome that is agreeable to us, and even if it is unreliable in certain cirmstances we should continue to use it to justify doing the wrong thing. Philosophically this is sort of inverse Machiavellian, that the end, even if it isn’t justified by the means doesn’t matter, since most of the results which interest us come before the end, and then the final coup de grace is that the government (i.e. Congress, regulators, Fed, Treasury, etal) will rescue us even if our models are completely whacked out. This is the way that mathematics can pave the road to ethical hell.
Help us, oh great and good god of finance to continue to find ways, ethical or otherwise, to serve our desire for money, regardless of outcomes for those who are not us.
It is a world that I don’t want to live in. When did this happen, and who will stand up against it? I guess that part of the answer is the pledge signing failure at Harvard Business School, by those who don’t have any believe in the common good!!! Eliot wrote about this outcome.
The collection of emotions together with memories and patterns of thought is what we describe as personality. Seems like this individuality is something we are afraid of giving up. Let’s say we are able to lose the bodies completely soon and we are able to maintain our personalities as free-floating blobs of energy, for example. We won’t have many “needs” at that point. So most of the drivers of our current behaviour will be gone. We’ll still need to worry about upkeep, one would think — safeguarding our existence, whatever form it will take. But other than that, no food, water, land, yacht, sex issues. Need to feel like Alexander the Great for an hour? Not a problem. So, most of our activity, is going to be directed at possibly 1) Continued learning about the universe(s) 2) Games and other pastimes. We’ll probably have to learn how to escape ticking planets like the earth, but other than that life shouldn’t be too uneventful. We’ll probably won’t even need a way to operate machinery at some point. Is it conceivable that we would be able to perform all experimentation in our “heads”?
Last musing: We’ve already become a sort of large interconnected brain with the help of the communication services and devices. Would the ultimate evolution be one single brain, integrated with synthetic new thinking modules that we create ourselves? Would we let this happen, or is our individuality and survival instinct too strong?
Tippy said:
“I thought of the artificial intelligence analogy because both AI and the VaR models might work if the computer programs could build in the complexity of the real world.”
The only way that you will build a model that reflects the complexity of the real work is collecting data using a double-entry book-keeping framework of rules that supports a deep cost accounting. “Capital,” after all, is a potential return on the “cash-value in trade” where cash value is decided using data from actual marketplace trades.
If one is counting gambling bets as having “cash-value” then such a model will always be inaccurate and volatile simply because its data is incorrect in terms of what “capital” is. Where cash value in trade is a real factor in a model, capital is a potential value in exchange for the real value.
The only time you will know the real value of a bet on future conditions is when it succeeds or fails.
What is happening with banks gambling with dollars in place of chips is that the bank has no way of measuring the probability of real-gains [losses] from betting without the deep-cost accounting. If the bank had the deep-cost-accounting capability, transactions that are a gambling bet will not post because a wager has no “present cash-value in trade,” hence it will have no balancing “capital-potential in exchange.”
There will be no solution to this melt-down until we recognize the need for a fair and impartial control-language, which double-entry book-keeping had been until about 1980.
Financial modeling of risk that skips the proper book-keeping framework of rules is now and always will be a bad joke played on the greater culture.
“Models have always been a convenient mechanism to approximate risk, almost like an adjunct to a balance sheet or income statement–also tools not to be taken literally.”
An Income Statement and its Balance Sheet ought to reflect a real history. There is no excuse for it not to be doing so in a computer age. Isn’t the real problem here that large central banks, an experiment that is less than 30 years old, is a proven failure?
I certainly agree with you on the fundamentalism point. However, I re-iterate that the job of AI, however you define it, is not to necessarily model the complexity of the world in a quantitative fashion, whereas that certainly *is* the job of risk models. Frankly, I don’t think people are much better at making financial decisions than most models out there, aside from saying “there is too much uncertainty, therefor I won’t invest.” I feel like financial models are unfairly criticized sometimes because they are expected to “walk the line” of the uncertainty and decided the break even point to a rather unrealistic precision. No one wants a model that says “all this stuff is too risky,” but people are considered wise for saying exactly the same thing.
All links to Taleb appreciated. He’ll never climb the pinnacles of power as he is guilty of truth telling. We should all be thankful for the few like him and you.
While I use economics almost daily, I am also trained in archaeology, ancient and early medieval history, and old, languages such as Latin, Old English, etc. I took the Applied Math class for business, economics and social sciences and refer to my book of Applied Models in Urban and Regional Analysis only when I want to prove a point because math seems to make everyone very happy in my job of town planning, like everyone here I know math lies (hate the show Numbers because of this).
Please put up with the way I am going to get to my point:
Human math even a basic concept of 1+1=2 is wrong — it usually equals 3 (until modern society where it can still be two) or in the case of Twins or Trips 4 or 5, etc. Stats then gets to fill in where math fails here, but it often fails to look at the outliers and question why they are there — norms are looked at and we are taught to drop everything else.
While I cannot comment on VaR, I can comment on both formulas and why the US loves them based on the math stats statement above. It is based on our love of teams both in the work place and in sports, and not the individual. While I cannot prove this at this time for this post, my gut says this is why America likes teams because you get the norm or group consensus raising everyone up and not the individual. Those who say something different or give another view are pushed to the side and often are not listen to, as they are no longer a team player. These groups then find the math to prove their point and the larger the formula the more they get promoted because it shows what everyone is looking for in the group and for those they have to present to. Those who do not use formulas as much are not considered team players and tend to like to look at the outliers that need to be questioned and investigated. As they are not the group norm, these people do not get promoted and eventually find a new job or field. IF you promote the group people enough and not those who do not question, when you get to the top everyone is doing fancy formulas that say nothing but look good and most are very good at being team players so they no longer see the outliers as those people who questioned them are now gone. Not that everyone knows their Real Colors or what Real Colors is as registered program, but I would suggest that most of the people who do not do well in groups today or do not get promoted are the Greens (6% of pop. of which only 40% are female) who rely on facts and not numbers because they know numbers lie yet these were your thinkers prior to the computers who run today’s great math formulas and print the pretty charts. This team idea then gets reinforced with the idea it is not politically correct or polite to question the norm.
As your article says and a couple of other posts, basically, those who are not team players (math formula gurus) get left behind. I will add, these Greens for lack of a better description, will eventually find a new field or often drift from field to field until they start their own business or give up. For those who did not know, this starts in college as the student drifts looking for a field they can finally say yes to without upsetting the Prof by constantly questioning and looking for other ways to do things without formulas. Usually they land in history or they accumulate enough credits in college to have many degrees, but never graduate. In the 1600 – 1800s, they would have been encouraged more and considered a great thinking mind by joining societies just to debate ideas and write papers on them without being a Prof — if they came from the right social class or had a patron. Today’s Universities have started this problem of conformity and formulas for grades and eventually pay in the work place. Teams and formulas unfortunately, do not lend well to honest Socratic teaching/questioning, discussion, philosophy or the running of businesses in general.
In the sentance starting with IF in paragraph three, it should read:
IF you promote the group people enough and not those who question, when you get to the top everyone is doing fancy formulas that say nothing but look good and most are very good at being team players and no longer see the outliers as those people who questioned them are now gone.
It was an illusion. The consensus was among people who already agreed and whose heads had been turned by too much money. The freshwater school didn’t discard falsified hypotheses, didn’t accept critiques from outside, published much work with inadequate peer review or even entirely without peer review. I’ve actually had a professor of that school cite an unreviewed article and expect me to take it as a valid scholarly sources.
Scientific method is hard in any discipline and it has an ethical component. There’s an “easy way out” aspect to the failures. Ultimately, I believe, the failures of the discipline were failures, quite literally, of discipline.
I did… thank you.
There is a lot of data coming out of the field of Cognitive Psychology on exactly what sort of biases humans have in perceiving risks. Maybe practical economic models, or at least regulatory models need to include the cognitive biases of the actors.
Take the risk from a VAR model, and divide the outcome by some factor S(g) where g is the distribution of expected outcomes. The psychological data would allow an exact parameterization of S(g) where human cognitive biases cause a decrease in estimated risk that is roughly proportional to expected gain.
I know this sort of approach isn’t ideal from an objective, scientific perspective…we want our model to only include mechanistic elements of a system. But if I’m a regulator and I don’t have the resources to dissect a company’s risk model in detail, I might do better to evaluate their model using know statistical properties of how humans bias their perceptions.
These human properties wouldn’t change from model to model, even if the model varied wildly, and regulators would deal with enough cases that they could set up their modeler model so that it would be accurate on average with a particular variance.
Brenda, thank you. You have justified my existence.
Brenda, thank you. You have justified my existence.