Big Ideas
Aine Seitz McCarthy
  • Blog
  • About

The elusive quest for exogeneity

4/28/2012

8 Comments

 
More conference comments from the Midwest International Economic Development Conference.

At the migration session, Marieke Kleemans presented a thoughtful paper on labor market changes in response to immigration, using weather shocks as an instrument for internal migration in Indonesia. Her discussant, Michael Clemens from the CGD gave some insightful comments in return, including one that posited that weather stations (the source of her climate data) might actually be located somewhere between the source town and the destination city that drives migration. This called into question the exogeneity of her instrumental variable, and potentially the entire identification. It also led to the appropriately sarcastic exclamation from a frustrated economist: "Great. Even rainfall isn't exogenous anymore."

And this leads me to frustration number three: the elusive quest for exogeneity. In order to be able to discuss the causal effect of X on Y, it is imperative that you ensure both a) that Y doesn't cause X and b) that unobservable factors don't affect both X and Y systematically. Exogeneity is crucial to identifying an unbiased causal effect. However, my concern is that the exogeneity bar (likely due to the gold standard of randomized controlled trials) has been raised so high that our focus on exogeneity as the primary objective can consequently side-line development research objectives.

It certainly isn't the case that research with a great exogenous variable necessarily overlooks development altogether, but my fear is that switching the objective of the research to exogeneity limits the set of development questions we might even ask.  Additionally, my concern does not go so far as to call into question the legitimacy and effectiveness of using randomized controlled trials to study development, although Angus Deaton has written about this (a fantastic video summarizing his views through a review of Poor Economics is available here). However, I do want to re-emphasize that the objective of good economic development research should be informing and shedding light on development.

It seems to me that an obvious source of good development questions is from practitioners working in developing countries and communities of stakeholders, but this is actually fairly uncommon in economics research.  In a nice World Bank working paper, "Evaluation in the Practice of Development," Martin Ravallion point out that "Academic researchers draw the bulk of their ideas from the work of other academic researchers. To be useful, evaluative research [in development] needs to draw more heavily on inputs from non-researchers actively involved in making and thinking about policy in developing countries." Also interestingly, Ravallion will be the keynote speaker next year at the Midwest International Economic Development Conference in Madison, WI.

Last week, Dean Yang, gave a talk at at our APEC Development Seminar on experimental results in Malawi of updating prior savings choices. Since he does a lot of migration research (where the exogeneity challenge is ever present), I asked him about this. And although he said that he would officially place himself in a camp of high value on identification, that the concern of limiting the set of research to causally identifiable effects is an important and potentially controversial meta-level question. And he added that, in terms of publication, research on correlations (that bypass problem of exogeneity) are still relevant and important, especially if you bring a policy-relevant and unique dependent variable to the table. Perhaps 'unique dependent variable' is an economists way of cautiously saying: tell me why I should care.

Hat tips: ALD, DY, CBT
8 Comments
ironmonkey285
4/28/2012 12:11:51 pm

I look forward to reading those papers.. It was interesting to hear Dean's perspective on this. I certainly agree..

On one hand, learning about some of this stuff is a process. If nobody steps out and proposes a theory, shows a correlation, etc because they can't think of some means of identifying the causal effect.. then indeed, research will be very much limited.. and indeed, we'll probably keep talking about the same sorts of questions.

One of the benefits (maybe) is that publication changes this.. basically, people want to hear more about new things and interesting things. Perhaps too much (hence why, as he said, people don't replicate RCT designs really.. even if they should)

In so much as a) there is a high demand for identification b) there is a high demand for interesting discoveries, the challenge comes back to.. the ECONOMIST.

On one hand, there are some questions that might be unanswerable in economics, at least for the next "t" years or so. But if you look back 10 years, and then back 10 years before that, its pretty crazy to think about a) the questions people were asking b) how people were asking these questions. RCT's are an obvious development.. They are an interesting case, in that they're both limiting (generalizability, cost, time, logistics, ethics) but also empowering, as Dean said, because you can ask your own question and design your own way to test it.

In which case the burden is not so much on the discipline, but on you/me (who have to help design good ways of getting at the interesting/important answers).

That said, maybe there is a point where there is diminishing returns to the benefits of identification.. If somebody spends 3 years going crazy, looking for a good instrument so they can show something .. especially if that something is intuitively obvious, well...

Reply
Justin Andrew Johnson link
4/30/2012 03:33:53 am

There certainly must exist a diminishing marginal value of exogeneity. If you measure the quality of some research, Qr, as function of the difference between the predicted coefficient and the actual relationship, Qr = f(cp - ca), I would assume that reducing cp - ca would be easy at first as you eliminate the obviously endogenous modeling choices, but as you get closer to zero, further reductions are harder to find, require more or more expensive data, and require more time as an econometrician dealing with them. If cp - ca is decreasing in in effort E, and d^2(cp-ca)/dE^2 < 0, as I'm assuming, then the optimal E which maximizes Qr - Qc is not the one which minimizes cp-ca if you assume only that cost of research (Qc) is increasing in E.

Or, there exists such a thing as too much hand-wringing in the world of DOING development.

Reply
ironmonkey285
4/30/2012 10:22:43 am

Indeed.

Although when it is a scholarly community that judges whether cp is close enough to ca, sometimes E has to be pretty high to pass the bar.

Clearly the point that you make is important.. if you assume that only the cost of research is increasing in E.
If you can come up with some ironclad identification that is the foundation for some disproportionately large [positive] policy response.. and there is some point where PC [value from "policy change"] is increasing in E.. then the problem changes a bit.

Maybe this signals not only a bigger problem in economics, but a bigger problem in policy (a world where we're still spending a billion bucks on one laptop per child).

Reply
Matt B
4/30/2012 04:45:30 pm

There are a few common paths in econometrics that I've noticed:

1. use non-causal models (usually fancy ones) and pretend they're causal
2. pretend that every question has a valid IV
3. limit your research scope to those questions that have a valid IV
4. do RCTs
5. really spend a lot of time understanding the theory behind the estimation methods you are using, ask good questions and understand what the data is telling you, whether it is causal or not

5 is obviously the hardest, but I'm a big proponent. I'm not sure how popular it is.

I'm going to continue with Justin's math stuff...
Without having any evidence to back this up (what would that evidence even look like?), my main concern is not that f(.) is concave but rather that g(.)>f(.) for most values of (cp-ca). Here g is some other research question that translates data into quality research. But there are constraints that limit how low you can go with cp-ca using g(.), so if your objective is to min cp-ca rather than max Qr then you will end up choosing the inferior research question (f rather than g).

So what I'm saying is that I fear #3 above because it is tantalizing and distracting and leads to irrelevance.

Reply
MB
4/30/2012 04:49:02 pm

in other words, I agree with Aine

Reply
ironmonkey285
5/1/2012 10:11:28 am

Is this a question about personal research or research for the discipline?

It seems to me that it is at the researcher's discretion to choose the research approach. While they might face institutional constraints, funding constraints... and they might have more or less opportunities based on their approach.. they nonetheless are free to research what they want, right? If they have a good idea, chances are they will find some way to support some research into said idea?

As Dean said, new and interesting ideas are often welcome, even if they don't depend on a great identification strategy. What is the barrier here? I'm not necessarily saying such a barrier doesn't exist.. anyhow.

Is the problem about what the discipline is doing (rather than what a given economist is free to do.. note, these two are of course very related sometimes)? In this regard, I think it is compelling to think about where the discipline is now. Behavioral.. experimental.. even thinking of better ways of getting empirical results. a) i'm inclined to think that a researcher is much more able to answer interesting questions, even "within the mainstream," today than they were a decade ago. b) the fact that the discipline has come a long ways in the last couple decades, and is increasingly borrowing from ideas in other disciplines, is a good sign of continued progress (toward answering increasingly interesting questions.. and even toward answering those questions more "effectively" [in terms of identification or otherwise]).

Reply
Aine McCarthy
5/2/2012 11:00:12 pm

I think of this much more as question about the discipline as a whole. While of course institutional norms will affect individual research questions and the motivation to actually ask good questions, the real concern is about the discipline as a whole getting sidelined by methodological development minutiae. However, it is true, Ryan, that economics as a whole has come a seriously long way from where we used to stand. In the long run of intellectual history, it is certainly true that we are asking more creative questions now.

Reply
Aine McCarthy
5/2/2012 11:01:45 pm

P.S. Thanks for all the good comments, this is an interesting debate.

Reply



Leave a Reply.

    Aine Seitz McCarthy

    International development, economics and some pretty ambitious ideas from a stubborn graduate student clinging to her sense of adventure.


    Categories

    All
    Agriculture
    Amusements
    Books
    Camp
    Demography
    Economics
    Education
    Family Planning
    Fieldwork
    Futball
    Gender
    Grad School
    International Development
    News
    Public Policy
    Research
    Travel

    RSS Feed

    Enter your email address:

    Delivered by FeedBurner


Powered by Create your own unique website with customizable templates.