How should abstracts of papers be written to maximize impact? In this post, I evaluate the abstracts of academic papers written by my friends at graduate school using recent advancements in language models.

Using Language Models as Benchmarks

One of the many uses of machine learning toolkits and language models is their use as “neutral” benchmarks to which human expectations can be compared and back out potential biases.

Some notable examples include:

• Bianchi, Ludvigson, and Ma (AER, 2022) employs big data algorithms to construct a measure of belief distortion, as measured against the ML-based benchmark.
• Aubry, Kräussl, Manso, and Spaenjers (2020) uses a neural network-based model to generate benchmark prices for art.

The recently released ChatGPT, a variant of the GPT language model that produces human-like text responses in a conversational context, can be used to “benchmark” people’s response upon reading the title of the paper. To the extent that people usually read the title and the abstract of the paper, we can then quantify the extent to which the true abstract of the paper is different from what ChatGPT expects the abstract to be. The greater the difference, the larger the surprise.

Example: Chaudhry (2022)

Aditya Chaudhry, my colleague from Booth and who is on the job market this cycle (if you are reading this, you should hire him!), has an intriguing paper titled “Do Subjective Growth Expectations Matter for Asset Prices?.”

Here’s what ChatGPT thinks is the paper is about, based on the title of the paper:

For comparison, here’s the actual abstract:

To the extent that ChatGPT correctly captures the average response of the reader based on reading the abstract, I’d say this abstract scores high on the surprise factor.

A More Systematic Analysis

The above example presents an interesting and fun exercise, but how does this compare across papers? To ask this question, I implement a more systematic analysis.

Data

To increase the number of observations in this study, I take the abstracts of job market candidates’ JMPs from Chicago Booth this year. Here’s the list of titles from job market candidates in Accounting, Finance, and Joint Program in Financial Economics:

• “The Politics of CSR Activity: Evidence from Press Releases”
• “Processing Industry Classification”
• “Minority Whistleblowers: Evidence from the LGBTQ+ Community”
• “Elections Have Consequences: The Impact of Political Agency on Climate Policy and Asset Prices”
• “Do Subjective Growth Expectations Matter for Asset Prices?”
• “The Effect of Minority Bank Ownership on Minority Credit”
• “Flexible Rent Setting and Rental Income”
• “Local Government Financial Constraint and Spending Multiplier in China”
• “Safety Nets, Credit, and Investment: Evidence from a Guaranteed Income Program”
• “Non-Fungible Cash in the Stock Market”
• “Retail Trading and Asset Prices: The Role of Changing Social Dynamics”

Methodology

Having collected the titles for testing, I then proceed to generate hypothetical abstracts. Specifically, for each title, I generate a hypothetical abstract through the following query on ChatGPT:

  “Write an abstract for the economics paper, “_____________________________________.”

At this step, one paper yields the following error, which I thus exclude from my sample:

Now, for each hypothetical abstract and true abstract pair, I use two language models to compute the cosine similarity after projecting each to a vector space: (1) Spacy and (2) Sentence Transformers.

• Shortly put, both language models allow you to compare the similarity across two texts.
• Typically, transformer-based models are thought to be more sophisticated in their architecture and thus yield more accurate comparisons.

Results

The figure below displays the cosine similarities between the hypothetical abstract and the true abstract for each of the language models:

Focusing on the red bars, we find that the Spacy model does quite poorly, as expected, as there is little variation in the similarity across abstracts — even for the Chaudhry (2022) paper mentioned above. It does a poor job of capturing the nuances of “not” in the text, while the transformer-based model seems to be able to capture it.

Focusing on the gray bars, we find that the similarity is the lowest for Chaudhry (2022). Ideally, it would be negative but this is probably due to poor pre-processing. Nonetheless, it does rank the lowest in terms of cosine simliarity, which is in line with the manual inspection of the abstracts.

The highest cosine similarity is obtained for Park (2022), “Flexible Rent Setting and Rental Income,” whose abstract and its hypothetical equivalent are shown below:

• True Abstract Landlords with greater expertise earn higher rental income by adjusting rent more flexibly. New property-level rent data reveals flexible properties—that adjust rent more frequently and substantially—generated higher rental income during the rental-housing-market collapse as they aggressively discounted rent and consequently experienced lower vacancies. They also produced higher rental income during the market boom because they did not suffer from higher vacancies despite their rent increases. Neither Calvo nor Taylor’s models fully explain these rent and vacancy patterns because properties adjusted rents in response to seasonal rental-housing demand, inflation, and competition, showing they choose the timing and degree of rent changes. On the other hand, flexible rent settings of properties owned by professional, large, or experienced landlords suggest expertise is an important determinant of flexible rent settings.
• Hypothetical Abstract The paper ““Flexible Rent Setting and Rental Income”” investigates the relationship between flexible rent setting and rental income in the housing market. Flexible rent setting refers to the ability of landlords to adjust the rent they charge to tenants in response to changes in market conditions. Using data on rental markets, the study finds that flexible rent setting is associated with higher rental income for landlords. Specifically, the study finds that landlords who are able to adjust the rent they charge in response to changes in demand or other market conditions are able to generate higher rental income compared to landlords who are unable to do so. The paper also explores the potential implications of flexible rent setting for tenants, such as increased financial burden or the risk of eviction. Overall, the study provides insights into the ways in which flexible rent setting affects rental income and the potential consequences for both landlords and tenants.”

The second lowest cosine simliarity is obtained for Leonelli (2022), “Minority Whistleblowers: Evidence from the LGBTQ+ Community.” What’s surprising about this one is that ChatGPT accurately guesses that the paper uses survey evidence, even though the title does not suggest it (see highlighted portions).

• True Abstract I examine how public attention affects whistleblowing activity by minorities, specifically the LGBTQ+ community. I find that, compared with counties that have high protection for LGBTQ+ employees, whistleblowing increases during Pride Month (June) in counties that have low protection for LGBTQ+ employees. In addition, those whistle-blowers are more likely to disclose their identity. To provide more direct evidence, I conduct a complementary survey experiment and find that LGBTQ+ respondents’ willingness to report misconduct increases during Pride Month. The survey responses suggest that the increase in the willingness to report misconduct arises through reduced concerns about retaliation, reputational effects, and adverse responses from the general public. Overall, my analyses provide evidence that public attention on minorities can increase whistleblowing by reducing the expected cost. My findings are important because systematic under-reporting of misconduct in the workplace can have detrimental consequences for minority employees and exacerbate inequality in the labor market.
• Hypothetical Abstract “The paper ““Minority Whistleblowers: Evidence from the LGBTQ+ Community”” investigates the experiences and challenges faced by LGBTQ+ individuals who speak out about wrongdoing or misconduct in their workplace. Using data from interviews and surveys with LGBTQ+ whistleblowers, the study finds that these individuals face unique barriers and discrimination when raising concerns about misconduct. They are more likely to face negative consequences such as job loss, social isolation, and mental health issues as a result of their whistleblowing. The paper also explores the role of supportive workplace cultures and policies in enabling LGBTQ+ whistleblowers to speak out, and suggests that efforts to create inclusive and supportive environments may encourage more individuals from minority groups to report misconduct. Overall, the study sheds light on the experiences of LGBTQ+ whistleblowers and the ways in which discrimination and supportive environments impact their decisions to speak out.”

## Coda

Language models have the potential to revolutionize the way we conduct research, making it more efficient and comprehensive. This post provides an entertaining example of how language models can be employed to generate benchmarks for evaluating certain facets of our academic performance.

Most importantly, I wish best of luck to my friends who are on the market this year!