------------------------------------------------------------------------------------------------------------------------ name: log: G:\Gillooly et al 2021\Zigerell reanalysis of Gillooly et al 2021.log log type: text opened on: 22 Sep 2021, 20:27:54 . use "G:\Gillooly et al 2021\Grad_Experiences_Public_v1.dta" . do "C:\Users\ljzig\AppData\Local\Temp\STD1fe4_000000.tmp" . * [1] Codebook . . codebook q14 female treatment2 q31_1 q31_2 q31_3 q31_4 ------------------------------------------------------------------------------------------------------------------------ q14 Do you feel as if you would be successful if you took this course? ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: A_12 range: [1,5] units: 1 unique values: 5 missing .: 0/294 tabulation: Freq. Numeric Label 76 1 A great deal 99 2 A lot 107 3 A moderate amount 5 4 A little 7 5 Not at all ------------------------------------------------------------------------------------------------------------------------ female RECODE of q2 (What gender do you identify as?) ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: female range: [0,1] units: 1 unique values: 2 missing .: 6/294 unique mv codes: 2 missing .*: 3/294 tabulation: Freq. Numeric Label 151 0 Male 134 1 Female 6 . 3 .a Other/nonbinary ------------------------------------------------------------------------------------------------------------------------ treatment2 Treatment (Two Value) ------------------------------------------------------------------------------------------------------------------------ type: numeric (float) label: treatment2 range: [1,2] units: 1 unique values: 2 missing .: 0/294 tabulation: Freq. Numeric Label 149 1 Low women 145 2 High women ------------------------------------------------------------------------------------------------------------------------ q31_1 Please rate the following: - Your interest in quantitative methods ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: A_14 range: [1,5] units: 1 unique values: 5 missing .: 0/294 tabulation: Freq. Numeric Label 23 1 Very low 22 2 Lower than average 59 3 Average 86 4 Higher than average 104 5 Very high ------------------------------------------------------------------------------------------------------------------------ q31_2 Please rate the following: - Your ability in quantitative methods ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: A_15 range: [1,5] units: 1 unique values: 5 missing .: 0/294 tabulation: Freq. Numeric Label 13 1 Very low 29 2 Lower than average 96 3 Average 122 4 Higher than average 34 5 Very high ------------------------------------------------------------------------------------------------------------------------ q31_3 Please rate the following: - Your interest in qualitative methods ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: A_16 range: [1,5] units: 1 unique values: 5 missing .: 0/294 tabulation: Freq. Numeric Label 17 1 Very low 49 2 Lower than average 82 3 Average 101 4 Higher than average 45 5 Very high ------------------------------------------------------------------------------------------------------------------------ q31_4 Please rate the following: - Your ability in qualitative methods ------------------------------------------------------------------------------------------------------------------------ type: numeric (byte) label: A_17 range: [1,5] units: 1 unique values: 5 missing .: 0/294 tabulation: Freq. Numeric Label 25 1 Very low 65 2 Lower than average 111 3 Average 77 4 Higher than average 16 5 Very high . . * [2] Replicate Model 1 Table S3 from Gillooly et al 2021 . . recode q14 (1=5 "A great deal") (2=4 "A lot") (3=3 "A moderate amount") (4=2 "A little") (5=1 "Not at all"), gen(q14re > code) (187 differences between q14 and q14recode) . tab q14recode RECODE of q14 (Do | you feel as if | you would be | successful if you | took this course | Freq. Percent Cum. ------------------+----------------------------------- Not at all | 7 2.38 2.38 A little | 5 1.70 4.08 A moderate amount | 107 36.39 40.48 A lot | 99 33.67 74.15 A great deal | 76 25.85 100.00 ------------------+----------------------------------- Total | 294 100.00 . . gen quant = (q31_1 + q31_2) / 2 . gen qual = (q31_3 + q31_4) / 2 . . ologit q14recode i.treatment2##i.female quant qual Iteration 0: log likelihood = -354.96327 Iteration 1: log likelihood = -329.65456 Iteration 2: log likelihood = -329.24696 Iteration 3: log likelihood = -329.24568 Iteration 4: log likelihood = -329.24568 Ordered logistic regression Number of obs = 285 LR chi2(5) = 51.44 Prob > chi2 = 0.0000 Log likelihood = -329.24568 Pseudo R2 = 0.0725 ------------------------------------------------------------------------------------ q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------------+---------------------------------------------------------------- treatment2 | High women | -.6093621 .3088889 -1.97 0.049 -1.214773 -.003951 | female | Female | -.4307352 .3126657 -1.38 0.168 -1.043549 .1820782 | treatment2#female | High women#Female | .8157447 .4534668 1.80 0.072 -.0730339 1.704523 | quant | .8884287 .1345037 6.61 0.000 .6248063 1.152051 qual | .1465488 .1308666 1.12 0.263 -.1099451 .4030427 -------------------+---------------------------------------------------------------- /cut1 | -.7141175 .7870001 -2.256609 .8283743 /cut2 | -.1337717 .7515102 -1.606705 1.339161 /cut3 | 2.927465 .7497457 1.457991 4.39694 /cut4 | 4.60208 .7815091 3.070351 6.13381 ------------------------------------------------------------------------------------ . . * [3] Remove the post-treatment controls . . ologit q14recode i.treatment2##i.female Iteration 0: log likelihood = -354.96327 Iteration 1: log likelihood = -353.62256 Iteration 2: log likelihood = -353.62193 Iteration 3: log likelihood = -353.62193 Ordered logistic regression Number of obs = 285 LR chi2(3) = 2.68 Prob > chi2 = 0.4432 Log likelihood = -353.62193 Pseudo R2 = 0.0038 ------------------------------------------------------------------------------------ q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------------+---------------------------------------------------------------- treatment2 | High women | -.3771789 .2969385 -1.27 0.204 -.9591677 .2048098 | female | Female | -.4486088 .3037772 -1.48 0.140 -1.044001 .1467834 | treatment2#female | High women#Female | .5041093 .4361746 1.16 0.248 -.3507772 1.358996 -------------------+---------------------------------------------------------------- /cut1 | -3.973041 .4256306 -4.807261 -3.13882 /cut2 | -3.415977 .3487833 -4.09958 -2.732374 /cut3 | -.668955 .2133117 -1.087038 -.2508718 /cut4 | .8001593 .2147749 .3792081 1.22111 ------------------------------------------------------------------------------------ . . * [4] Limit the analysis to male respondents to test for the "backlash" . . ologit q14recode treatment2 if female==0 Iteration 0: log likelihood = -191.76413 Iteration 1: log likelihood = -190.9323 Iteration 2: log likelihood = -190.93171 Iteration 3: log likelihood = -190.93171 Ordered logistic regression Number of obs = 151 LR chi2(1) = 1.66 Prob > chi2 = 0.1970 Log likelihood = -190.93171 Pseudo R2 = 0.0043 ------------------------------------------------------------------------------ q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.38557 .2994938 -1.29 0.198 -.9725671 .2014271 -------------+---------------------------------------------------------------- /cut1 | -3.960503 .6476979 -5.229968 -2.691039 /cut2 | -3.468531 .5862092 -4.617479 -2.319582 /cut3 | -1.15891 .4783065 -2.096373 -.221446 /cut4 | .4950513 .4705915 -.4272911 1.417394 ------------------------------------------------------------------------------ . ologit q14recode treatment2 if female==0, robust Iteration 0: log pseudolikelihood = -191.76413 Iteration 1: log pseudolikelihood = -190.9323 Iteration 2: log pseudolikelihood = -190.93171 Iteration 3: log pseudolikelihood = -190.93171 Ordered logistic regression Number of obs = 151 Wald chi2(1) = 1.65 Prob > chi2 = 0.1993 Log pseudolikelihood = -190.93171 Pseudo R2 = 0.0043 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.38557 .3004018 -1.28 0.199 -.9743467 .2032067 -------------+---------------------------------------------------------------- /cut1 | -3.960503 .6428403 -5.220447 -2.70056 /cut2 | -3.468531 .5736679 -4.592899 -2.344162 /cut3 | -1.15891 .4778487 -2.095476 -.2223433 /cut4 | .4950513 .470547 -.4272039 1.417306 ------------------------------------------------------------------------------ . reg q14recode treatment2 if female==0 Source | SS df MS Number of obs = 151 -------------+---------------------------------- F(1, 149) = 1.71 Model | 1.55320018 1 1.55320018 Prob > F = 0.1929 Residual | 135.254747 149 .907749979 R-squared = 0.0114 -------------+---------------------------------- Adj R-squared = 0.0047 Total | 136.807947 150 .91205298 Root MSE = .95276 ------------------------------------------------------------------------------ q14recode | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.2030591 .1552358 -1.31 0.193 -.509807 .1036889 _cons | 4.114451 .2420117 17.00 0.000 3.636233 4.59267 ------------------------------------------------------------------------------ . reg q14recode treatment2 if female==0, robust Linear regression Number of obs = 151 F(1, 149) = 1.70 Prob > F = 0.1942 R-squared = 0.0114 Root MSE = .95276 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.2030591 .1557024 -1.30 0.194 -.5107292 .104611 _cons | 4.114451 .237865 17.30 0.000 3.644427 4.584476 ------------------------------------------------------------------------------ . . tab q14recode RECODE of q14 (Do | you feel as if | you would be | successful if you | took this course | Freq. Percent Cum. ------------------+----------------------------------- Not at all | 7 2.38 2.38 A little | 5 1.70 4.08 A moderate amount | 107 36.39 40.48 A lot | 99 33.67 74.15 A great deal | 76 25.85 100.00 ------------------+----------------------------------- Total | 294 100.00 . recode q14recode (1/3=0 "Low") (4/5=1 "High"), gen(q14binary) (294 differences between q14recode and q14binary) . tab q14binary RECODE of | q14recode | (RECODE of | q14 (Do you | feel as if | you would | be | successful | if | Freq. Percent Cum. ------------+----------------------------------- Low | 119 40.48 40.48 High | 175 59.52 100.00 ------------+----------------------------------- Total | 294 100.00 . . tab q14binary treatment2 if female==0, chi2 RECODE of | q14recode | (RECODE of | q14 (Do | you feel | as if you | would be | successful | Treatment (Two Value) if | Low women High wome | Total -----------+----------------------+---------- Low | 25 29 | 54 High | 54 43 | 97 -----------+----------------------+---------- Total | 79 72 | 151 Pearson chi2(1) = 1.2218 Pr = 0.269 . prtest q14binary if female==0, by(treatment2) Two-sample test of proportions Low women: Number of obs = 79 High women: Number of obs = 72 ------------------------------------------------------------------------------ Group | Mean Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- Low women | .6835443 .052327 .5809852 .7861034 High women | .5972222 .0578009 .4839346 .7105099 -------------+---------------------------------------------------------------- diff | .0863221 .0779683 -.0664931 .2391372 | under Ho: .0780934 1.11 0.269 ------------------------------------------------------------------------------ diff = prop(Low women) - prop(High women) z = 1.1054 Ho: diff = 0 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 Pr(Z < z) = 0.8655 Pr(|Z| > |z|) = 0.2690 Pr(Z > z) = 0.1345 . . * [5] Check whether any individual controls were sufficient to reduce the p-value . . ologit q14recode treatment2 q31_1 if female==0, robust // * Iteration 0: log pseudolikelihood = -191.76413 Iteration 1: log pseudolikelihood = -183.39761 Iteration 2: log pseudolikelihood = -183.28819 Iteration 3: log pseudolikelihood = -183.28799 Iteration 4: log pseudolikelihood = -183.28799 Ordered logistic regression Number of obs = 151 Wald chi2(2) = 14.32 Prob > chi2 = 0.0008 Log pseudolikelihood = -183.28799 Pseudo R2 = 0.0442 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.6720068 .3075423 -2.19 0.029 -1.274779 -.0692351 q31_1 | .5514101 .161367 3.42 0.001 .2351366 .8676836 -------------+---------------------------------------------------------------- /cut1 | -2.446687 .7936408 -4.002194 -.8911793 /cut2 | -1.927339 .7666637 -3.429972 -.4247055 /cut3 | .5541263 .7426161 -.9013744 2.009627 /cut4 | 2.324292 .7698944 .8153272 3.833258 ------------------------------------------------------------------------------ . ologit q14recode treatment2 q31_2 if female==0, robust Iteration 0: log pseudolikelihood = -191.76413 Iteration 1: log pseudolikelihood = -183.42435 Iteration 2: log pseudolikelihood = -183.34283 Iteration 3: log pseudolikelihood = -183.34273 Iteration 4: log pseudolikelihood = -183.34273 Ordered logistic regression Number of obs = 151 Wald chi2(2) = 13.17 Prob > chi2 = 0.0014 Log pseudolikelihood = -183.34273 Pseudo R2 = 0.0439 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.4705245 .3034856 -1.55 0.121 -1.065345 .1242962 q31_2 | .6428388 .1889461 3.40 0.001 .2725113 1.013166 -------------+---------------------------------------------------------------- /cut1 | -1.993506 .8316755 -3.62356 -.3634518 /cut2 | -1.483697 .8013206 -3.054256 .0868627 /cut3 | .9559496 .7879545 -.5884129 2.500312 /cut4 | 2.738387 .8254333 1.120568 4.356207 ------------------------------------------------------------------------------ . ologit q14recode treatment2 q31_3 if female==0, robust Iteration 0: log pseudolikelihood = -191.76413 Iteration 1: log pseudolikelihood = -190.80102 Iteration 2: log pseudolikelihood = -190.80021 Iteration 3: log pseudolikelihood = -190.80021 Ordered logistic regression Number of obs = 151 Wald chi2(2) = 1.89 Prob > chi2 = 0.3884 Log pseudolikelihood = -190.80021 Pseudo R2 = 0.0050 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.3562245 .3065864 -1.16 0.245 -.9571227 .2446738 q31_3 | .0701828 .1449181 0.48 0.628 -.2138515 .3542172 -------------+---------------------------------------------------------------- /cut1 | -3.700768 .7848677 -5.239081 -2.162456 /cut2 | -3.209498 .7360283 -4.652087 -1.766909 /cut3 | -.9014591 .6931424 -2.259993 .457075 /cut4 | .7558575 .69079 -.5980661 2.109781 ------------------------------------------------------------------------------ . ologit q14recode treatment2 q31_4 if female==0, robust Iteration 0: log pseudolikelihood = -191.76413 Iteration 1: log pseudolikelihood = -190.93207 Iteration 2: log pseudolikelihood = -190.93148 Iteration 3: log pseudolikelihood = -190.93148 Ordered logistic regression Number of obs = 151 Wald chi2(2) = 1.65 Prob > chi2 = 0.4376 Log pseudolikelihood = -190.93148 Pseudo R2 = 0.0043 ------------------------------------------------------------------------------ | Robust q14recode | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.3866494 .302723 -1.28 0.202 -.9799756 .2066767 q31_4 | -.0031795 .1525865 -0.02 0.983 -.3022437 .2958846 -------------+---------------------------------------------------------------- /cut1 | -3.970944 .7419469 -5.425133 -2.516755 /cut2 | -3.478934 .6954658 -4.842022 -2.115846 /cut3 | -1.169286 .6678299 -2.478209 .139636 /cut4 | .4846082 .6676096 -.8238826 1.793099 ------------------------------------------------------------------------------ . . * [6] Did the treatment affect responses to items used in the controls, among male respondents? . . ologit q31_1 treatment2 if female==0, robust Iteration 0: log pseudolikelihood = -205.14468 Iteration 1: log pseudolikelihood = -202.09994 Iteration 2: log pseudolikelihood = -202.09337 Iteration 3: log pseudolikelihood = -202.09337 Ordered logistic regression Number of obs = 151 Wald chi2(1) = 5.96 Prob > chi2 = 0.0146 Log pseudolikelihood = -202.09337 Pseudo R2 = 0.0149 ------------------------------------------------------------------------------ | Robust q31_1 | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | .7432078 .3043553 2.44 0.015 .1466823 1.339733 -------------+---------------------------------------------------------------- /cut1 | -1.606408 .5338962 -2.652825 -.5599908 /cut2 | -1.020218 .4852522 -1.971295 -.0691409 /cut3 | .1521427 .4816895 -.7919513 1.096237 /cut4 | 1.59545 .5020703 .6114101 2.57949 ------------------------------------------------------------------------------ . ologit q31_2 treatment2 if female==0, robust Iteration 0: log pseudolikelihood = -201.21986 Iteration 1: log pseudolikelihood = -201.19104 Iteration 2: log pseudolikelihood = -201.19104 Ordered logistic regression Number of obs = 151 Wald chi2(1) = 0.06 Prob > chi2 = 0.8106 Log pseudolikelihood = -201.19104 Pseudo R2 = 0.0001 ------------------------------------------------------------------------------ | Robust q31_2 | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | .0719286 .3000605 0.24 0.811 -.5161792 .6600365 -------------+---------------------------------------------------------------- /cut1 | -3.078967 .6357703 -4.325054 -1.83288 /cut2 | -1.893726 .5165519 -2.906149 -.8813029 /cut3 | -.227464 .4800962 -1.168435 .7135073 /cut4 | 1.875915 .5157751 .8650146 2.886816 ------------------------------------------------------------------------------ . ologit q31_3 treatment2 if female==0, robust Iteration 0: log pseudolikelihood = -225.90064 Iteration 1: log pseudolikelihood = -222.82446 Iteration 2: log pseudolikelihood = -222.81644 Iteration 3: log pseudolikelihood = -222.81644 Ordered logistic regression Number of obs = 151 Wald chi2(1) = 5.92 Prob > chi2 = 0.0150 Log pseudolikelihood = -222.81644 Pseudo R2 = 0.0137 ------------------------------------------------------------------------------ | Robust q31_3 | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.7339172 .3015841 -2.43 0.015 -1.325011 -.1428231 -------------+---------------------------------------------------------------- /cut1 | -3.508473 .5909184 -4.666652 -2.350294 /cut2 | -2.017625 .5222798 -3.041275 -.9939759 /cut3 | -.7792349 .4868854 -1.733513 .1750429 /cut4 | 1.10028 .4962221 .1277023 2.072857 ------------------------------------------------------------------------------ . ologit q31_4 treatment2 if female==0, robust Iteration 0: log pseudolikelihood = -215.17326 Iteration 1: log pseudolikelihood = -213.20905 Iteration 2: log pseudolikelihood = -213.20543 Iteration 3: log pseudolikelihood = -213.20543 Ordered logistic regression Number of obs = 151 Wald chi2(1) = 3.98 Prob > chi2 = 0.0461 Log pseudolikelihood = -213.20543 Pseudo R2 = 0.0091 ------------------------------------------------------------------------------ | Robust q31_4 | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- treatment2 | -.5857334 .2936272 -1.99 0.046 -1.161232 -.0102346 -------------+---------------------------------------------------------------- /cut1 | -2.83302 .5087721 -3.830195 -1.835845 /cut2 | -1.412886 .4923241 -2.377824 -.447949 /cut3 | .1584508 .4904857 -.8028834 1.119785 /cut4 | 2.541748 .6791367 1.210665 3.872832 ------------------------------------------------------------------------------ . end of do-file . log close name: log: G:\Gillooly et al 2021\Zigerell reanalysis of Gillooly et al 2021.log log type: text closed on: 22 Sep 2021, 20:28:05 ------------------------------------------------------------------------------------------------------------------------