Showing posts with label Elastic Justification. Show all posts
Showing posts with label Elastic Justification. Show all posts

Wednesday, February 18, 2009

ID the Elasticity


Imagine that you were asked to choose between two movies recommended by Netflix. For the purposes of this exercise, also imagine that you are a cheapskate like me and have only subscribed to the “one at a time” movie rental option so you can indeed only choose one movie to watch next. In this scenario, you will not learn the title of the movie before choosing but you will learn bit by bit about three key features for each movie, one feature at a time. First you learn the names of the star actors and actresses for each film. Then, after a few seconds, you learn who directed each film. Finally you are told each movie’s genre. You do not have to make a choice until you have heard all of the information.


Now, does the order in which the feature information is presented make any difference? For instance, if instead you were told first about the genre, then the actors, and then the director would you make a different choice of film than in the original scenario? Could you find yourself spending the weekend with The Wedding Crashers instead of Apocalypse Now simply because the order changed? Normative decision theory holds that order doesn’t matter. However, research into Information Distortion by J. Edward Russo, Victoria Medvec, and others suggests that, in practice, order matters a great deal to decision makers. It has long been known that people irrationally seek out and interpret information in ways to favor a decision that has already been made. This happens for a variety of reasons including cognitive dissonance reduction. However, Information Distortion (ID) takes place much earlier, before the decision has actually been made which makes it all the more fascinating. ID theory suggests that someone just has to form an initial preference for one option over another and thereafter any new information he or she receives gets distorted to favor that initial preference. Like when standing in front of a funhouse mirror, with ID the new facts themselves look materially different. So in our movie example, if the first thing learned was the directors, David Dobkin and Francis Ford Coppola respectively, you would be more likely to interpret further information with a bias toward your initial preference based only on director (hopefully Coppola). If instead you first considered “drama or comedy” and your initial preference tilted toward comedy, then when you later learned about the directors you might suddenly have a new found respect for David Dokin and have a greater chance of watching Vince Vaughn get crushed in backyard football and tied to the bed posts.


Why does ID happen? Russo along with collaborators Kurt Carlson, Margaret Meloy, and Kevyn Yong put forth an answer in their 2008 paper published in the Journal of Experimental Psychology. Their research examines three possible goals as causes of ID: conserving effort, creating separation (making the choices more distinct), and maintaining consistency. Over the course of three experiments they conclude that consistency is the most likely explanation. However, it should be noted that these experiments tested a limited universe of three theories. There could be other significant factors in ID.


Before considering additional factors it is worth making a few observations about the theories chosen for testing by Russo and colleagues. Of the theories, only consistency has a strong social element. I would guess if subjects were asked to explicitly rank their goals by importance that consistency would be ranked highest. Additionally, the goal of conserving effort would be associated with making a quick decision. Quick decisions are only made once, early in the process, and then it is over. So in this case subjects have limited opportunity to express their goal to experimenters as a "conserving effort" subject would presumably make their decision and then try to just ignore further information. On the other hand, a "consistency driven" subject must express their goal each time he or she faces new information. It may be difficult to compare this two goals using the same experimental design.


Now, a few alternative explanations do come to mind. Perhaps subjects experience a form of “trial choice” and start “rooting” for their choice to be right. With this theory, although subject are not yet locked into a decision, they are trying it on for size and simulating the experience of having made a final choice. If this alternative theory is true then there may be very little new at all going on with ID. Instead the phenomenon would be a mere extension of the classic distortion theories to trial as well as final choices. Another, less powerful, explanation lies in subject interpretation of “authoritative intent.” Chefs creating menus, professors designing word problems, and even Netflix recommenders usually present information in an intentional order and that order often follows the rule of most important information first. Subjects may be relying too much on this typical pattern.


A final alternative challenges not the goal but the mechanism of ID, elements of which are suggested in Christopher Hsee’s theory of Elastic Justification. Information Distortion in its very name implies a change in the decision maker’s interpretation of the facts themselves. Movie directors are somehow judged more tallented, when considering a prospective date five foot two is somehow a little taller, etc. However, it is possible that something else is going on. Sure subject interpretation of a new fact may change a little yet the weight given in the final decision to the importance of that fact could be altered more dramatically. You might still think Owen Wilson is brilliant but the importance of actor quality in movie selection could be reduced when you learn the names of the directors first. ID cannot be entirely ruled out because this alternative does not explain all of the results in the paper; however, Russo’s experiments do not test the weighting element possibility. (Note that ID may provide an alternative explanation for Elastic Justification instead of the other way around).

Wednesday, September 17, 2008

Justifying the Unjustifiable


Several weeks ago I met with Columbia University Marketing Professor Leonard Lee. Over a fantastic pistachio milkshake at Tom’s CafĂ© (of Seinfeld fame), we got on the topic of moral decision-making research. Leonard suggested that I read an older paper by Chicago Professor Christopher Hsee. The paper, “Elastic justification: How unjustifiable factors influence judgments,” describes how unjustifiable information can subconsciously creep into the decision-making process. To summarize, Hsee breaks information into two primary categories:

  • Justifiable: Information that is directly relevant to the judgment being made by the accepted criteria of the decision-maker (e.g. a candidate’s experience)
  • Unjustifiable: Information the decision-maker may wish to take into consideration but that he knows, by the criteria of what is acceptable, should not come to bear on the decision. (e.g. a candidate’s race or sex)

Hsee’s studies show that decision-makers do not include unjustifiable information in their decisions except under certain conditions relating to the JUSTIFIABLE information. The degree of "elasticity" in how to weigh the justifiable factors determines how much influence the unjustifiable information has on the judgment. If the importance weighting assigned to various justifiable attributes is clearly defined and fixed (inelastic), unjustifiable information does not influence the decision. However, if the weighting is vague and open to interpretation (elastic), unjustifiable information will influence the decision by reweighting the importance of the justifiable factors. Since past literature shows that people try to make justified decisions and line up consistent facts/evidence for those decisions, Hsee concludes that people do not knowingly incorporate unjustifiable information. The process is subconscious.

The paper’s findings stand out because they are contrary to normative decision theory which would not recognize a split between justifiable and unjustifiable information. Instead, the normative approach takes in all information, assigns each factor a relevant weight according to its probable utility impact, and concludes with the utility maximizing decision (see for example Jon Baron’s work on Multi-Attribute Utility Theory). In contrast, Hsee’s work shows that decision-makers exclude unjustifiable information in certain cases. Reweighting utility impact for a given factor is also non-normative.

In the paper Hsee sites three historical studies as evidence as well as two of his own. It is the second of Hsee’s studies with which I take some issue. In this second study subjects were given a “language intuition test” by which they were asked to select from a multiple choice list the meaning of unfamiliar Chinese characters. 20 questions were asked of each subject, only 10 of which counted towards a total score. Subject later self reported their score with a financial incentive for higher scores. Hsee suggests that two factors would influence the self reported score: 1) actual performance and 2) desire to report a high score for the financial incentive. He then asserts that the former is a justifiable factor and the later is unjustifiable and also predicts that the degree of elasticity in the grading process will determine the amount the unjustifiable factor will influence scores upward. In the inelastic condition subjects were asked to score only odd numbered questions. In the elastic condition, scores were based on the subject’s evaluation of the “ying/yang nature of the symbols” – a highly vague and subjective approach. Subjects were to count the 10 questions whose symbols they found to be the most “yang” in nature. As anticipated, subjects in the elastic condition reported higher overall scores.

Unfortunately the study design does not seem well suited as a test of Hsee’s theory. I do not believe it is possible to determine whether or not subjects are indirectly incorporating unjustifiable information by reweighting the justifiable or merely directly incorporating their “unjustifiable” desire for a high score. In fact, in the elastic condition there is not a justifiable method of selection provided to subjects at all. How does one justifiably assess the “yangness” of a character? Faced with a process such as this one it is no surprise that subjects did not feel unjustified in selecting questions that they happened to get right as yang characters. The the yangness approach may break a social contract with the subjects, allowing them to have free reign in “cheating” on their scoring. One might ask what the selection process of students who did not wish to cheat would have looked like. The study would be better formed if it provided a subjective yet seemingly justifiable procedure for selection.

I do not wish to discard the entire paper based on Study #2. Hsee has a very interesting theory with significant merit conferred by the other studies. There are also some very interesting additional questions the follow from the study. Further research seem warranted to answer the following:

  • Can we evaluate the specific weighting assigned by subjects to justifiable factors? For example in Hsee’s study #1 involving real estate appraisal, what relative weights do subjects place on living room size, appliances, carpets, etc?

  • Can we demonstrate that the weight assigned to a specific factor changed? For instance can we demonstrate that subjects specifically reweighted the value of good appliances relative to carpets as a means to indirectly incorporate unjustifiable factors? Perhaps this can be solicited by asking subjects to assign numeric weightings or at least by rank ordering the importance of the factors.

  • How might we prove/disprove that this process takes place subconsciously? Perhaps subjects instead make conscious decisions and then use a purposeful forgetting/self-deception mechanism to remain justified in their own eyes – (more forthcoming on a "breadcrumbs of self-deception" idea). There would seem to be significant moral implications if this is a conscious process.