Session Time/Number: 10am
Session Space: E
Notetaker: Tracy F.
Take an online course on “survey design” if you don’t know about this subject
What not to do…
• Think about your study, how am I going to make sure that I tell a true story at the end.
• Be sure your data is accurate, what is my data telling me…beyond the X & Y axis.
• Data can help beyond the scope of the data.
• Look at “no women answered this Q”
• Look at outliers…is it significant or is it a mistake (extremes can tell you that maybe you didn’t set the parameters correctly…you might have a bunch of people hitting a wall)
• Limit your graphs to 3 (no more than 5), then spend time answering questions, frame the info. Take the time to have pretty graphs. Ad campaigns that change us are data driven, have a narrative, have a pretty graph.
• The simplicity of the graph…never want your audience to look at your graph and feel stupid. I’ll say something like “this is a ____ type of graph (match pair analysis) so this data is statistically valid…but the graph is obvious. Position yourself as an authority. Current hires, projected need…highlight 20% more people is the delta…analogy of a the 80/20 rule or ants. You are an expert, I’m an expert lets have a discussion
What you should do…
• Have the question you are trying to answer (scientific method)
• Identify the problem
• Ask why?
• What currently doesn’t work (what you are trying to fix)
• What research has been done, what works?
• I have a great idea, roll out all at once, no one knows if it worked or if your idea was effective or if another idea would have been good.
• treatment v. control group
• what info comes out of the study, do follow-up studies, how effective was this item/question.
• what were the average scores…you’ll have an average, where are you falling compared to your control group
• don’t invest all your resources in one way
• metrics we care about
• how will we measure success
• compare control with your treatment
• how do I make it “randomized” when you have folks that “opt in” (they are statistically different from others)
• managers can’t say “we want this team to get this treatment
• here all the people who could potentially participate, stratified sampling, sample from each group.
• What if you don’t randomize?
• What if your sample group gets picked all the time…the answers they give are different than those who never participate.
Qualifiers/Things to remember
• Survey fatigue (re-sampling same group…give time, recover, stop answering or answers are silly)
• Do you like pandas? Qualifier question is not necessary, you can see if they hit 3s all the way down.
• Have you participated in studies recently?
• Two Qs that ask the same thing to see if you get consistency – can be helpful but what are you going to do about it. Eliminate them, eliminate the Qs. Have rules.
• Qualtrics allow for conditional questions…don’t ask employed/unemployed then ask salary
• Asking someone to break it…should do it before you send the survey.
• Preliminary survey to find the issues and problems.
What is a good size sample
• So many interests how will we narrow it down. Maybe don’t give too much freedom.
Qualitative and quantitative
• Small sample would be qual.
• Use a t score test. Recognize the population. What is statistically significant. Volunteers are different than those who don’t.
• Can use the time and reminders and incentives to see who/when they respond. Track the waves of people.
• Look at your demographics (if 90% of your group is latino but only 10 latinos responded…)
• Narrowing choices for what they want to learn
• Phenom – anchoring – if you only see these values, they will think it is only what is out there. What kinds of things are you interested in. then give a list. Qualitative data really needs fewer answers/options.
• Have a study group is a great way to start.
• Anchoring and how you phrase things is important.
• Don’t ask a Q that you have no ability or intention to do anything about.
• I’m not being heard is the best way to reinforce
• I have no control
• What can we change
• what impact can they have
• I want a lot of structure. I want more freedom.
• Get people excited about the results.
• Create a narrative, the statistics are the picture…the problem is the main character. Spending time with the data…stick to the story.
• Maybe have a analogy like surfing…a metaphor can help
• My first time learning to surf…I kept paddling hard but there was a wave that was right there…that way was so stupid no one wants to stick with that “way”
• No one is convinced by data, they are convinced by the story
• Know when you create a bias…have someone who knows how to build a smart study. Think about how to eliminate bias.