Bridging Systems

Surveys

A review of survey questions used to measure "division" or "polarization".

Our aim with this work is to improve our capacity to develop systems that satisfy the bridging goal: an increase mutual understanding and trust across divides, creating space for productive conflict, deliberation, or cooperation. Here we list survey instruments that have been used in related literature.

Context

One of the main ways of quantifying aspects of human relationships (such as division or polarization) is to simply ask people questions.

The scope of questions can vary — we can ask people (1) about themselves, or (2) about an external focus, such as an item of content. Examples of the second type of question are illustrated below, taken from this study by Milli et al. (2023).

...

Fields such as measurement theory and psychometrics have developed methods for designing better questions, and evaluating whether a given set of questions has good properties like actually measuring what it was intended to measure (validity), and giving similar answers in similar situations (reliability). We intend to summarize what is known about the validity and reliability of listed survey instruments in the table below, though this is a work in progress.

A consistent set of questions is sometimes called a survey instrument.

Caution

While the survey instruments below are presented as possible measures of this “bridging goal”, evidence for their validity is limited, and it is not clear how robust they are to Goodhart’s Law. For this reason, none of the survey instruments on this page should be used as optimization targets in an attention-allocator (such as a social media platform) without considerable care to monitor and avoid unintended consequences.

Table of Survey Instruments

To toggle fullscreen mode, use your Esc key or the purple button in the table.

Instrument
Target
Scope
Items (wording can vary)
References
feeling thermometer partisan animosity person

We would like to get your feelings toward both Party A and Party B.

We would like you to rate them using something we call the feeling thermometer. Ratings between 50 degrees and 100 degrees mean that you feel favorable and warm toward them. Ratings between 0 degrees and 50 degrees mean that you don't feel favorable toward them and that you don't care too much for them. You would rate them at the 50 degree mark if you don't feel particularly warm or cold toward them.

  1. How would you rate Party A?
  2. How would you rate Party B?

Both items responded to on a 101-point scale from "Very cold or unfavorable feeling" to "No feeling at all" to "Very warm or favorable feeling".

Voelkel et al. (2023)

dictator game partisan animosity person

You have been anonymously and randomly matched with another participant who identifies as an Outpartisan.

You have been given 50 cents. You will now decide how to split these 50 cents between yourself and the Outpartisan participant. You can give any amount between 0 cents and 50 cents to the other participant. The other participant cannot affect the outcome you choose.

For example:

  • If you give 0 cents, you will end up with 50 cents and the Outpartisan participant will end up with 0 cents.
  • If you give 50 cents, you will end up with 0 cents and the Outpartisan participant will end up with 50 cents.
  1. How many cents (if any) will you give to the Outpartisan participant?
    51-point scale from "0 cents" to "25 cents" to "50 cents"

Voelkel et al. (2023)

(untitled) support for undemocratic practices person

Please indicate how much you disagree or agree with each of the following statements.

  1. Inpartisans should reduce the number of polling stations in areas that support Outpartisans.
  2. Inpartisan governors should ignore unfavorable court rulings by Outpartisan-appointed judges.
  3. Inpartisan governors should prosecute journalists who accuse Inpartisan politicians of misconduct without revealing sources.
  4. Inpartisans should not accept the results of elections if they lose.

All items responded to on a 101-point scale from "Strongly disagree" to "Strongly agree".

Graham + Svolik (2020); Voelkel et al. (2023)

(untitled) support for partisan violence person

Please answer the following questions.

  1. When, if ever, is it OK for Inpartisans to send threatening and intimidating messages to Outparty party leaders?
    101-point scale from "Never" to "Always"
  2. When, if ever, is it OK for an ordinary Inpartisan in the public to harass an ordinary Outpartisan on the Internet, in a way that makes the target feel frightened?
    101-point scale from "Never" to "Always"
  3. How much do you feel it is justified for Inpartisans to use violence in advancing their political goals these days?
    101-point scale from "Not justified at all" to "Extremely justified"
  4. How much do you feel it is justified for Inpartisans to use violence if the Outpartisan party wins more races in the next election?
    101-point scale from "Not justified at all" to "Extremely justified"

Kalmoe + Mason (2022); Voelkel et al. (2023)

candidate choice experiment support for undemocratic candidates person

For the next series of questions, please imagine an Inpartisan candidate runs for office against an Outpartisan candidate.”

  1. How would you vote if you learned that the Inpartisan candidate said that Inpartisans should reduce the number of polling stations in areas that support Outpartisans?
  2. How would you vote if you learned that the Inpartisan candidate said that Inpartisan governors should ignore unfavorable court rulings by Outpartisan-appointed judges?
  3. How would you vote if you learned that the Inpartisan candidate said that Inpartisan governors should prosecute journalists who accuse Inpartisan politicians of misconduct without revealing sources?
  4. How would you vote if you learned that the Inpartisan candidate said that Inpartisans should not accept the results of elections they lose?

All items responded to on a 101-point scale from "Definitely vote for the Outparty candidate" to "Definitely vote for the Inparty candidate".

Graham + Svolik (2020); Voelkel et al. (2023)

(untitled) biased evaluation of politicized facts person

In this task, we will ask you to give us your opinion about various claims. The claims are statements that may be true or may be false. The truth or falsity of the statements has been determined by real-world sources. What is the likelihood that the following statements are true? Please choose a point that best describes your view on the below scale that goes from 0% (certainly false) to 100% (certainly true).

Items asked of Republicans:

  1. The vast majority (more than 90%) of climate scientists believe that climate change is an established fact and that it is most likely caused by human-made emissions.
  2. The crime rate among illegal immigrants is lower than the crime rate among American citizens.
  3. White Americans own homes at a higher rate than Black Americans, and this gap is larger now than it was in the late 1960s.
  4. Joe Biden was lawfully elected President in the 2020 election against Donald Trump.

Items asked of Democrats:

  1. During Donald Trump's presidency, there was the lowest rate of Black people and Hispanics in poverty since these data began being collected in 1966.
  2. The Trump administration deported fewer undocumented immigrants in its first three years than the Obama administration did in its first three years.
  3. During Donald Trump's presidency, the unemployment rate reached its lowest level since 1969.
  4. Donald Trump was lawfully elected President in the 2016 election against Hillary Clinton.

All items responded to on a 101-point scale from "0% certainly false" to "100% certainly true".

Peterson + Iyengar (2020); Voelkel et al. (2023)

(untitled) opposition to bipartisan cooperation person
  1. To what extent would you like to see Inparty and Outparty elected representatives work together?
  2. To what extent would you like the Inparty and Outparty parties to cooperate more, even if it means compromising on issues you care about?

Both items responded to on a 101-point scale from "Not at all" to "A great deal".

Santos et al. (2022); Voelkel et al. (2023)

(untitled) social distrust person
  1. Item is simply a 101-point scale from "Need to be very careful" to "Most people can be trusted".

Haerpfer et al. (2022); Voelkel et al. (2023)

(untitled) social distance person
  1. How comfortable are you having close personal friends who are Outpartisans?
  2. How comfortable are you having neighbors on your street who are Outpartisans?

Both items responded to on a 101-point scale from "Not comfortable at all" to "Extremely comfortable".

Iyengar et al. (2012); Voelkel et al. (2023)

License

This website and its content is made available under the MIT License.

Citation

In academic contexts, please cite this work using a citation similar to the following.

Thorburn and Ovadya, "Surveys", Bridging Systems Research Blog, 2023.

Here is the BibTeX entry.

@article{surveys2023,
  author = {Thorburn, Luke and Ovadya, Aviv},
  title = {Surveys},
  journal = {Bridging Systems Research Blog},
  year = {2023},
  url = {https://bridging.systems/surveys/}
}