I appreciate both the study and sharing the results, but I'm questioning some of the questionnaire design choices.
-----
> On a scale of 1-5, how long do you plan to stay at your company?
There is literally no reason for this to be "on a scale of 1–5" when one could just ask about number of years directly. Asking people to translate years into "a scale of 1–5" introduces all sorts of errors: for some people, a four might correspond to 2–4 years, whereas others may think about a 10–20 years tenure when they see a four.
What's even weirder is they later translate this 1–5 scale into a binary yes/no answer. How was the threshold selected? Would respondents have replied differently if they knew where the threshold would go?
-----
> On a scale of 1-5, what’s your perception of the value of your stock grant relative to your other compensation?
This could ask about a percentage or a fraction instead, and prevent some of the same errors as above. You might argue that "sure but a four simply means 80 %" but I doubt that – note that the average answer for a subgroup in the study was nearly four, and I don't believe half of respondends of that group get less than 20 % of their perceived comp value in non-equity.
-----
> How likely are you to recommend your company to a friend or colleague?
This is a good question because there's a specific way to combine the data later that has known correlations with company success; there is some reasonable research linking higher eNPS values with better outcomes. But that requires combining the data using that method, which is not just taking the average – what the study authors did!
If they had used the established methods, they would have gotten some reliability and validity testing for free. They did not, so they would have to show that their combined data is valid too.
-----
The other two survey items (growth opportunities and equity comp fairness) could probably have been made more concrete, but I also see how it may be easier to ask on a 1–5 scale for those. However, there's nothing in the report about how they performed reliability and validity testing of these scales. Without testing the scales for these psychometric properties, the results could be meaningless.
-----
I get that this article never intended to reach for scientific rigour and that it is but an advertisement for a software platform. But still, I believe we (as a species) can get better at these things to the point where even advertisements can be informative. I'd like to see that, anyway.
Yeah that's very misleading. You note questions from 1 to 5 then they show you an average of answers from 0 to 4 (you'll never get an average below 1 and of course probably not a linear scale anyway).
We already had that discussion when websites used star ratings where you couldn't rate 0 star...
For this kind of qualitative research I stand by the subjectivity of the questions asked as more informative. Retention is in itself a subjective construct.
Sure, you want "number go up" on average, but staying at a company for two years means something completely different to a restless jobhopper compared to a stability-minded employee with a family. Likert scales are perfect for this because they let respondents be their own judge of how well this job is retaining them.
Same goes for the perception of the value of their stock, but I admit it would be really nice to have the absolute value here in addition to the subjective scale to detect an interaction effect.
I do agree that the "actively looking elsewhere" label on the graph for the retention question is laughable.
> Sure, you want "number go up" on average, but staying at a company for two years means something completely different to a restless jobhopper compared to a stability-minded employee with a family. Likert scales are perfect for this because they let respondents be their own judge of how well this job is retaining them.
But the organisations designing their equity comp structure don't really care about "how well people feel the job is retaining them", even if that could be measured on a valid and reliable subjective scale. Organisations will be optimising for employee churn rate, which is the direct inverse of... you guessed it! Expected length of employment.
If the scale accidentally prioritises stability-minded people over job-hoppers, then (all else equal) that's kind of fine because that is also a way to reduce churn. Now, practically speaking all else will not be equal and that's why we ask more than one question on a questionnaire like this!
Glad to see such a thorough analysis written. Liquidity definitely matters with companies taking so long to exit. Such risk isn’t worth it for many employees, and if something doesn’t change I suspect hiring people will become hard for startups.
I appreciate both the study and sharing the results, but I'm questioning some of the questionnaire design choices.
-----
> On a scale of 1-5, how long do you plan to stay at your company?
There is literally no reason for this to be "on a scale of 1–5" when one could just ask about number of years directly. Asking people to translate years into "a scale of 1–5" introduces all sorts of errors: for some people, a four might correspond to 2–4 years, whereas others may think about a 10–20 years tenure when they see a four.
What's even weirder is they later translate this 1–5 scale into a binary yes/no answer. How was the threshold selected? Would respondents have replied differently if they knew where the threshold would go?
-----
> On a scale of 1-5, what’s your perception of the value of your stock grant relative to your other compensation?
This could ask about a percentage or a fraction instead, and prevent some of the same errors as above. You might argue that "sure but a four simply means 80 %" but I doubt that – note that the average answer for a subgroup in the study was nearly four, and I don't believe half of respondends of that group get less than 20 % of their perceived comp value in non-equity.
-----
> How likely are you to recommend your company to a friend or colleague?
This is a good question because there's a specific way to combine the data later that has known correlations with company success; there is some reasonable research linking higher eNPS values with better outcomes. But that requires combining the data using that method, which is not just taking the average – what the study authors did!
If they had used the established methods, they would have gotten some reliability and validity testing for free. They did not, so they would have to show that their combined data is valid too.
-----
The other two survey items (growth opportunities and equity comp fairness) could probably have been made more concrete, but I also see how it may be easier to ask on a 1–5 scale for those. However, there's nothing in the report about how they performed reliability and validity testing of these scales. Without testing the scales for these psychometric properties, the results could be meaningless.
-----
I get that this article never intended to reach for scientific rigour and that it is but an advertisement for a software platform. But still, I believe we (as a species) can get better at these things to the point where even advertisements can be informative. I'd like to see that, anyway.
Yeah that's very misleading. You note questions from 1 to 5 then they show you an average of answers from 0 to 4 (you'll never get an average below 1 and of course probably not a linear scale anyway).
We already had that discussion when websites used star ratings where you couldn't rate 0 star...
For this kind of qualitative research I stand by the subjectivity of the questions asked as more informative. Retention is in itself a subjective construct.
Sure, you want "number go up" on average, but staying at a company for two years means something completely different to a restless jobhopper compared to a stability-minded employee with a family. Likert scales are perfect for this because they let respondents be their own judge of how well this job is retaining them.
Same goes for the perception of the value of their stock, but I admit it would be really nice to have the absolute value here in addition to the subjective scale to detect an interaction effect.
I do agree that the "actively looking elsewhere" label on the graph for the retention question is laughable.
> Sure, you want "number go up" on average, but staying at a company for two years means something completely different to a restless jobhopper compared to a stability-minded employee with a family. Likert scales are perfect for this because they let respondents be their own judge of how well this job is retaining them.
But the organisations designing their equity comp structure don't really care about "how well people feel the job is retaining them", even if that could be measured on a valid and reliable subjective scale. Organisations will be optimising for employee churn rate, which is the direct inverse of... you guessed it! Expected length of employment.
If the scale accidentally prioritises stability-minded people over job-hoppers, then (all else equal) that's kind of fine because that is also a way to reduce churn. Now, practically speaking all else will not be equal and that's why we ask more than one question on a questionnaire like this!
A commercial for Hiive.
Glad to see such a thorough analysis written. Liquidity definitely matters with companies taking so long to exit. Such risk isn’t worth it for many employees, and if something doesn’t change I suspect hiring people will become hard for startups.
LinkedIn tier comment here.