SOC 220 - Sept 29

Overview

Jeff Brassard explains how to design survey questions that respondents can understand and answer reliably, covering wording, response scales, memory limits, question order, sensitive topics, and response sets. Examples ranging from TikTok-use questions to a flawed University of Alberta transportation survey show how leading, ambiguous, double-barreled, or unbalanced questions can distort data.

Key takeaways

A response scale must match the question’s grammar and meaning: strongly agree/disagree fits a statement such as “I support the policy,” not a direct question asking how someone feels.
A balanced five-point scale generally offers two positive options, a neutral midpoint, and two negative options; four positive choices and one negative choice can push neutral respondents into an artificially negative category.
Long recall periods undermine accuracy: respondents are unlikely to reliably report annual egg consumption or social-media use from months earlier, so ask about recent behavior or use appropriate tracking data.
Use “prefer not to answer” when respondents need an opt-out from sensitive questions; reserve “don’t know” for plausible uncertainty or as a screening response.
Put announced-topic questions near the beginning, sensitive items in the middle or later, and easy demographics near the end to reduce confusion and abandonment.
Response-set risk increases when surveys present long grids of similar items; vary question formats, break up repetitive sections, and use carefully designed consistency checks.

Chapters

0:00 Survey-Question Basics: Relevance, Clarity, and Neutral Wording

Brassard reviews core rules: align each question with the research question, use specific language, and test whether respondents can answer it.
Avoid long, ambiguous, or double-barreled questions; “interesting and informative” asks about two traits that may receive different answers.
Do not lead respondents with emotionally loaded context, such as mentioning the monarchy’s taxpayer cost before asking whether it should be abolished.
Use a separate screening question to establish whether respondents have relevant experience, such as reading Stephen King novels.

8:49 Match Likert Scales to Question Wording

Positive phrasing is generally easier to answer than negatively framed questions about rules that ban laptops.
A strongly-agree-to-strongly-disagree scale works with a statement, such as “I support a ban,” rather than a question asking how someone feels.
Oppositely worded items can expose inconsistent response patterns, but they should be used deliberately rather than as routine phrasing.

12:05 Balance Response Options Around a Neutral Middle

A common five-point scale has two positive responses, one neutral response, and two negative responses.
The example “excellent, very good, good, satisfactory, or poor” overweights positive answers and forces middling respondents toward “poor.”
Check that each answer option is both balanced and a sensible match for the question being asked.

15:11 Limit Recall Demands and Use “Don’t Know” Carefully

Questions about how many times someone ate eggs in the last three weeks or exercised at the start of COVID may exceed reliable recall.
Asking about weekly social-media use this week is more answerable than asking respondents to remember their use two months earlier.
Brassard prefers “prefer not to answer” when the goal is to let respondents decline a sensitive question; “don’t know” can mix discomfort with lack of knowledge.
Use “don’t know” when ignorance is plausible or useful for screening, such as asking young respondents about an unformed political identity.

23:18 Question Order: Topic First, Sensitive Items Later

Most surveys should present questions in the same order; shuffling is mainly for testing order effects and is too complex for the class assignment.
Ask about the announced topic early so respondents who expected a barbecue-and-climate survey are not surprised by unrelated election questions.
Place sensitive questions, such as experiences of sexual assault, around the middle or later portion to reduce early survey abandonment.
Use easy demographic questions—age, gender, income, or past votes—near the end, unless they are needed for screening.

28:58 Reduce Response Sets with Varied Questions and Layouts

A response set occurs when answers reflect a repetitive answering habit rather than the respondent’s views on each item.
Break up long runs of similar items, vary question types, and include occasional items requiring a different response.
Contrasting or repeated items can help detect inconsistent answers, while an obvious attention check—such as identifying a dog among a computer, car, and desk—can flag careless responding.
A large grid of strongly-agree-to-strongly-disagree items invites respondents to click one column repeatedly; digital survey layouts can show one question at a time.

32:51 Diagnosing a Leading Question About Alberta’s Government

The question “What upsets you most about Alberta’s provincial government?” assumes respondents are upset.
Its options—policies, ethics, ideology, or politicians—offer no way to express approval or say that none of those things is upsetting.
The class identifies the forced negative premise as a leading-question problem.

36:17 Online-Learning Software: Ambiguous Terms and a Mismatched Scale

“Features and adoption” combines different topics, and “adoption” is unclear about whether it means implementation, uptake, or another measure.
The response options “very good” through “very bad” do not clearly fit a question asking respondents to rate both features and adoption.
The example demonstrates the need to specify the target of evaluation and make the response scale symmetrical with the question.

39:05 Why “How Many Eggs in the Last Year?” Fails

A one-year recall period is too long for many respondents to remember accurately, and the question lacks a clear zero option for people who do not eat eggs.
“Eat eggs” could mean eggs as a meal, eggs in other foods, individual eggs, or meals containing eggs.
The answer ranges are poorly spaced: narrow lower intervals are followed by a broad upper category, making responses difficult to interpret.

42:28 Student Satisfaction Needs Specific Topics and Balanced Options

The scale for satisfaction with the University of Alberta student experience offers four positive choices and only one negative choice.
“Student experience” could refer to professors, courses, residence, campus food, clubs, or campus life, so the question is too broad by itself.
Ask separate questions about specific aspects, then consider an overall satisfaction item after respondents have assessed them.

46:31 Criminal-Conviction Employment Question: Split the Double Barrel

“Denied a job opportunity or promotion” combines two distinct outcomes that could have different answers.
A screening question is needed because respondents without a criminal conviction cannot meaningfully answer the item as written.
The strongly-agree-to-strongly-disagree scale does not fit a direct question about whether an event happened; a clear yes/no or a matching statement would work better.
Whether a conviction caused a denial may be an inference rather than something the respondent knows with certainty.

50:14 Leadership, Breakfast, and TikTok Questions Expose Design Flaws

“The prime minister is competent and charismatic” is double-barreled because respondents may judge competence and charisma differently.
The breakfast question lists only tea and bread, coffee, oats, or cereal, excluding other foods and respondents who eat more than one item.
A “select all that apply” format is more appropriate for breakfast choices, with each food listed as its own option.
The examples reinforce that answer choices should cover realistic responses rather than force respondents into incomplete categories.

1:01:03 TikTok Minutes and “Nutritious Breakfast” Need Operational Definitions

The TikTok options of less than 30, 30–60, and more than 60 minutes leave a broad upper category and do not make non-use explicit.
Asking for a number of minutes, potentially checked against a phone’s usage-tracking settings, can provide more detail; narrower intervals are another option.
“I regularly eat a nutritious breakfast” uses vague terms: both “regularly” and “nutritious” can mean different things to different respondents.
A specific behavior and a defined time period are easier to measure than an undefined judgment about nutrition.

1:03:35 University Opinion Questions and the Flawed U of A Parking Survey

“What is your opinion of the University of Alberta?” is broad, and a scale such as “very good” may rate the respondent’s opinion rather than the university.
The parking and transportation survey omitted students from its role categories and used unclear employment-funding terms such as “operating funded” and “grant/trust funded.”
Travel-time questions were hard to answer for respondents whose commute varied, while travel-mode options inconsistently allowed one or two choices.
The survey asked which amenities might encourage cycling or motorcycling without adequately establishing whether those travel modes applied to each respondent.

1:11:38 Transportation and Cannabis Surveys: Screening and Response-Option Problems

The transportation survey used awkward options such as “walk or wheelchair” and offered commute reasons without first clearly screening for how respondents travel.
Its question order placed an important item about changes in commuting behavior late in the survey.
The cannabis-policy survey was less flawed but still combined knowledge of risks and benefits in one item and could split support for smoking from support for vaping.
Brassard closes by noting that the class will continue discussing survey design on Thursday.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jeff Brassard.