Mechanical Research

If the research community at large is not already well aware of it, I recommend taking a moment to check out Amazon’s Mechanical Turk for its data collection potential. Like the historical chess playing “machine” Turk of its namesake, the site works by harnessing people behind the scenes to do bite sized “Human Intelligence Tasks” or HITs. HITs are small tasks still best performed by human rather than machine intelligence and include tagging photos with the proper labels, eliminating duplicate entries from catalogs, writing reviews, and, most importantly for academic researchers, filling out surveys.

As an example, I just executed a trial run of Mechanical Turk for a study I’m working on regarding curiosity for inherently positive or negative information and the circumstances in which subjects are better able to control the satisfaction of their curiosity. I created a HIT out of the control version of the two different curiosity questionnaires and asked that the survey be completed by up to 100 unique subjects for a reward of 25 cents. In less than four hours from submitting the HIT I have 100 responses at a cost of $27.50 (the site charges a small fee for use). Additionally, Mechanical Turk allows you to reject and not compensate any responders who did not complete their HIT satisfactorily. So, as a quality check I mixed in a question that helped ensure the responders were paying attention. A vast majority of the participants correctly answered a question very similar to the following: “If one hundred thousand and nine is greater than nine thousand enter ‘Q’ otherwise enter ‘T’.”

Though a quick, powerful, and cheap way to collect human subject data, Mechanical Turk does appear to have some major limitations. Most importantly I have yet to figure out a way to bar past respondents from answering subsequent altered versions of surveys used in between subject study designs, though as each respondent has a unique ID it is possible for repeat participants to be eliminated after the fact. Additionally, the baseline demographics of the typical Mechanical Turk worker in the subject pool and the self-selected participant factor may require special statistical treatment. Finally, the interface for creating surveys is rather limited so HTML skills are required.

Even with its limitations, at the very least Mechanical Turk seems like a great vehicle to do pilot studies. I plan to use it next on an outcome bias study to see if there is any merit in pursuing research on an alternative theory to Moral Luck written about previously in this blog.

Towelie Logic

The Society for Judgment and Decision Making held its annual conference this past weekend in beautiful (but cold) downtown Chicago. Roughly 500 researchers from around the world gathered to discuss the latest findings in the JDM realm. This was my second conference and it did not disappoint.

One talk that caught my attention was entitled “Will a Rose Smell as Sweet by Another Name? Specification-Seeking in Decision-Making.” The talk was presented by Christopher Hsee (University of Chicago Graduate School of Business) and Yang Yang (Shanghai Jiao Tong University) based on their forthcoming paper in Journal of Consumer Research. As described in their abstract:

“We offer a framework about when and how specifications (e.g., megapixels of a camera, number of airbags in a massage chair) influence consumer preferences and report five studies that test the framework. Studies 1-3 show that even when consumers can directly experience the relevant products and the specifications carry little or no new information, their preference is still influenced by specifications, including specifications that are self-generated and by definition spurious, and specifications that the respondents themselves deem uninformative. Studies 4 and 5 show that relative to choice, hedonic preference (liking) is more stable and less influenced by specifications.”

I provide an overview of the towel study from the paper as well as my take on the findings in last night’s Hightechfever.tv broadcast.

Je Regrette

ProspectTheory.net is not a political or self-help forum; however, I need to confess the regret I feel over a recent vote. Fortunately my regret is true to theme and related to judgment and decision-making.

On November 4th I voted on a relatively minor ballot issue here in Massachusetts to ban the practice of greyhound dog racing. I knew the issue was on the ballot along with two others of greater importance. This is unusual for me but I entirely forgot about the measure and entered the balloting booth without already having made a decision on the greyhound issue. I was in a hurry so made a snap decision to vote in favor of banning the practice.

This was a simple yes or no question but as a JDM geek I want to understand why I voted this way. I made a quick decision in the heat of moment. Was I using my hot/emotional system and my rational side would have arrived at a different answer, hence the feeling of regret? Quite possibly. When I voted yes I was thinking of the practice of dog racing itself. I visualized caged dogs, elderly men gambling, smoking, and ticket stub litter and it all seemed primitive and sad. However, I believe the greatest contributor to the decision was my internal framing of the question and my related starting point anchor. In the voting booth the question I asked myself was “do I disapprove of this practice?” I could have formed a different question which was actually much closer to the true one, “should the freedom of others be restricted in this domain?” My baseline/default answers to these two questions are quite different, even in the abstract. Do I disapprove of a questionable practice? Yes. Do I want to restrict freedoms? No. Even if I managed to ask myself both questions, the answer to the question I started with could serve as a powerful anchor and determine the ultimate outcome of the decision.

I am sure that politicos and partisans are well aware of this effect and use it to their advantage. Here we have another argument for a Nudge like Libertarian Paternalism when it comes to designing our ballots. Voting only works if we are asking the right questions.

A Curious Education

I am currently reading a broad survey paper by George Loewenstein as background for a possible curiosity experiment (“The Psychology of Curiosity: A Review and Reinterpretation,” Psychological Bulletin 1994. Vol. 116, No. 1.75-98). In the paper, there is a comment that relates to my recent “Buying Behavior” posting and provides credence to Harvard Economics Professor Roland Fryer’s idea to motivate scholastic achievement by paying students for good behavior. My previous post urged caution.

In discussing the societal implications of his information-gap theory of curiosity, which surmises that curiosity arises in the distance between “what one knows and what one wants to know,” Loewenstein states:

“The information-gap perspective has significant implications for education. Educators know much more about educating motivated students than they do about motivating them in the first place. As Engelhard and Monsaas (1988, p. 22) stated, ‘historically, education research has focused primarily on the cognitive outcomes of schooling’ rather than on motivational factors. The theoretical framework proposed here has several implications for curiosity stimulation in educational settings. First, it implies that curiosity requires a preexisting knowledge base. Simply encouraging students to ask questions—a technique often prescribed in the pedagogical literature—will not, in this view, go very far toward stimulating curiosity. To induce curiosity about a particular topic, it may be necessary to ‘prime the pump’ to stimulate information acquisition in the initial absence of curiosity. The new research showing that extrinsic rewards do not quell intrinsic motivation suggests that such rewards may be able to serve this function without drastically negative side effects.”

So there is hope that the extrinsically incented Capital Gains approach advocated by Fryer will “prime the pump” on learning and curiosity will take care of the rest without triggering the adverse effects of entering into economic verses social exchange.

BITH: Buying Behavior

(Carol Leone Childcare)

Behavior in the Headlines: Harvard Economics Professor Roland Fryer has an idea to boost performance in our failing schools – pay the students. A new program called Capital Gains is being piloted in the Washington DC area and it is paying students up to $1,500 for good performance in a variety of areas including testing and attendance. The thinking behind the program is that better incentives will encourage students to exhibit better behavior, leading them eventually to academic success. There are no hard data yet on the effectiveness of the approach so the program is currently being run as an experiment. A short movie on the DC pilot can be viewed here.

Fryer’s motive is laudable and his approach has merit; however, the DC experiment is not without risk. By providing financial incentives, Capital Gains is moving the expectation for good scholastic performance from the domain of social exchange to the domain of economic exchange. Such a transition may produce results in the opposite direction intended and the change caused by the program may be difficult to reverse.

Take for example Uri Gneezy and Aldo Rustichini’s study on an undesirable practice of day care center parents. In their resulting paper, “A Fine is a Price,” Gneezy and Rustichini describe the over-time hours and other difficulties for day care center staff caused by late child pick-ups. Day care center management imposed a financial penalty on tardy patents to discourage the practice; however, management was in for a surprise. Late pick-ups actually increased substantially after the fine policy was imposed. Further, when the day care centers later tried to remove the fine, the occurrence of late pick-ups remained at its new, higher level. One explanation for these results, also championed by Dan Ariely in his new book Predictably Irrational, is that parents no longer felt obligated by social contract to pick-up their children on time. The guilt of imposing a social difficulty was replaced by a specific economic value. If the fine price was lower than the value parents placed on the extra effort needed to get to the day care on time, parents simply showed up late and paid the fine. Once in this economic realm it was difficult to reverse the new framing. Dropping the fine merely gave parents a new fine “price” of zero dollars.

The study discussed was focused on a penalty verses an incentive payment so the results may not be directly applicable. However, in the case of Capital Gains, it is possible that students will not value the incentive money as much as they had previously valued the hopeful expectations of their parents or even their own sense of self respect. After students are in an economic exchange mindset they will understand the benefit of good behavior in explicit financial terms. If a student decides that the extra effort is not worth $1,500, she may decide to put in even less effort than she did prior to the program. If students remain in an economic exchange mindset, a later need to remove the financial incentive could leave students even less motivated than they were before the incentive system.

Though there are risks, I am happy to see programs like Capital Gains attempting to improve our nation’s educational system. The current system is failing our children, especially those from disadvantaged backgrounds. It is time to try something new. Thank you Professor Fryer.

Competitions as Commitment Devices

Today I had the privilege of judging the first phase of MIT’s famous $100K Business Plan competition. My fellow judges and I watched approximately 30 participants give their best 60-second elevator pitch to sell their business idea. Our contestants were all part of the “development track” which consists of businesses that address global issues such as poverty or the environment. Participants covered an impressive variety of quality ideas and technologies including low cost/non-electrical lighting, unique bio-fuels, medical diagnostics that cost 10 cents and can be mailed to the lab on a post card, solar cooking, and mesh wireless networks that overcome last mile issues in rural areas.

I have a strong personal interest in developmental entrepreneurship. In my day job I was a founding member of the IBM World Development Initiative and the leader behind its successful Global ThinkPlace Challenge to brainstorm sustainable solutions to African poverty. I was also a development track $100K participant myself (although back then it was only the $50K). My “Wider Reach” team, consisting of fellow Sloan MBAs Brian Roughan, Armina Karapetyan, Kamal Quadir, and Joe Zeff along with MIT Media Lab PhD student Jose Espinosa, was the winner of the $2,000 IDEAS Award, the $1K development track, and a semi-finalist in the overall $50K competition. We designed a mobile phone accessible marketplace for Bangladesh. Kamal Quadir stuck with the plan and made it a reality. The company is now called CellBazaar and its success has been widely covered by the media including The Wall Street Journal and The Economist.

Kamal’s story illustrates the power of business plan competitions. Kamal went to MIT with the intent of landing a job in finance. Instead he is an entrepreneur helping thousand of impoverished farmers sell their crops more efficiently. How did this happen? While the organizers of business plan competitions may think they are creating entrepreneurs by providing a little financing and exposure, I believe that these competitions are actually serving as Cialdini commitment devices.

Business plan contestants, many of whom had no original intention of becoming entrepreneurs, make small but increasingly significant commitments to the competition. It is easy to put together a 60 second elevator pitch but next they are writing executive summaries and then complete business plans. At each stage contestants declare publicly both orally and in writing their intent to follow through with the business idea. Eventually, through the absolutely amazing power of cognitive dissonance, contestants feel pressure to make sense of their efforts and declarations and rationalize that they must truly want to start these businesses after all. In fact, the large prize money offered in these contests actually takes away from the cognitive dissonance effect as it provides an alternative justification to contestants. So, counter intuitively, if the $100K organizers really want to create more entrepreneurs they might consider dropping the prize money back down to the original $1K!

BITH: The Financial Crisis and Action Ambiguity

Behavior in the Headlines: The stock market is down, credit flow is paralyzed, and Americans are expressing fervent outrage at “Wall Street fat cats and greed.” Constituents are demanding that any “bailout” package include stiff punishment for the financial insiders who “got rich on the path to getting us into this mess.” In the framework of the Moral Luck/Scapegoat discussion, we have a bad outcome along with a strong associated judgment that the actions of financial insiders were unethical and a high willingness to punish. Which of the two theories, Moral Luck or Scapegoat, does a better job of explaining the current situation? Our financial crisis has a unique attribute that may provide insight into these two effects – action ambiguity. In this case at least, it seems that Scapegoat prevails.

While the market was going up (good outcome) Americans had very little interest in the activities of the masters of the universe in high finance. We are all rapidly learning more about our dire economic straights but I believe most still have very little knowledge of the specific actions undertaken by the financial insiders at which Americans are now so angered. This is a case of ethical acts in a black box. With a bad outcome people are willing to judge activity as unethical even before they understand who made what actions. Moral Luck suggests an action will look less ethical in light of a bad outcome. Here we have bad outcomes generating a desire to judge Wall Street actors as unethical before we have even identified what specific actions we are judging.

The financial crisis is a messy real world example and not truly an action black box. Obviously some people, including key thought leaders, know more about the specific questionable actions of Wall Street insiders. However, perhaps a similar black box experimental design could be constructed. Introduce a bad outcome. Next ask subjects if someone should be responsible for the outcome. Then introduce an actor who can be logically associated with the outcome. Minimize the description of the action so that is has very little detail and phrase it in a statement that control subjects would find ethically neutral in a vacuum. “The chef mixed the cake.” I predict that given the right kind of bad outcome (one without culturally predetermined judgments of blame or innocence) subjects will assume that someone must be responsible and further be willing to assign some ethical responsibility to whatever logically connectable subject actor is introduced.