CALCULATING AND REPORTING EFFECT SIZES

28/04/20

ALCULATING AND

EPORTING

FFECT

IZES

IHRBIOSTATISTICSLUNCHLECTURESERIESPRESENTEDBY

DRPAOLACHIVERS

RESEARCHANDBIOSTATISTICS:INSTITUTEFORHEALTHRESEARCH

THEUNIVERSITYOFNOTREDAMEAUSTRALIA

paola[email protected].au

QuotedinSullivanandFeinn (2012).Usingeffectsize– orwhythepvalueisnotenough.

JournalofGraduateMedicalEducation.September:279‐282.DOI:http://dx.doi.org/10.4300/JGME‐D‐12‐00156.1

28/04/20

WhatareEffectSizes?

Ameasurementofthesize(magnitude)ofaneffect.

‐ independentofsamplesize

‐ standardizedmetric

‐ strengthofassociation

Whyareeffectsizesimportant?

• Effectsizeshelpcommunicatepracticalimportanceofaresult

‐ animportant butnonsignificantresult

‐ anunimportant butsignificant result

• Providesinformationabout‘how’important

• Allowscomparisonsacrossstudies(metaanalysis)

• Usedtoinformplanningforfuturestudiesapriori

• Areajournalrequirement

APA(7

ed.) guidetoeffectsizes

 Recommendinclusionfor“readerstoappreciatethemagnitudeor

importance”

(APA2020p.89)

 Statisticalestimateandshouldincludeconfidenceintervals

GeneralPrincipal

“…providethereaderswithenoughinformationtoassessthe

magnitudeoftheobservedeffect.”

(APA2020p.89)

28/04/20

MeasuringEffectSize

Canbecalculatedindifferentways.

Twomainapproaches:

1. Standardizeddifferencebetweentwo

means

• Cohen’sd

• Hedge’sd

• Glass’sDelta

• Commonlanguageeffectsize(CLES)

2. Strengthofassociation

MeasuringEffectSize

1. Standardizeddifferencebetweentwomeans

‐ d family

‐ differencebetweenobservationsdividedbythestandarddeviation

‐ Standarddeviationsunitsofeffect

2. Strengthofassociation

‐ r family

‐ proportionofvariancethatisexplainedbyitsgroup

Effectsizemeasures:

Betweensubjectsdesigns

Cohen’sd(Cohen,1988)

• Standardizedmeandifferenceofaneffect

• Dependentvariablescanbemeasuredondifferentscalesorbe

completelydifferentmeasurements

(Lakens,2013)

• Uncorrectedeffectsize

‐ Providesabiasedestimateofthepopulationeffectsizeespeciallyn<20

Fig.Cohen’sdequationforthesample (Lakens,2013p3)

28/04/20

Effectsizemeasures:

Betweensubjectsdesigns

d calculatedfromthet‐testdifferencesbetweentwogroups

Fig.Cohen’sdequationforthesamplerelatedtoat‐test (Lakens,2013p3)

Correctionsforbias

Populationeffectsizeestimatesbasedonsampleaverages

overestimate thetruepopulationeffect.

Correctionsforbiascanbeapplied:

• CorrectionsforCohen’sd =Hedges’s g

• Correctionsforetasquared(η

)=omegasquared(ω

)

Effectsizemeasures:

Betweensubjectsdesigns

Hedges’s dStandardizedmeandifferenceofaneffect

• Correctedeffectsize

‐ Providesanunbiased estimateofthepopulationeffectsize

Fig.Hedges’s g

equationforthesample (Lakens,2013p3)

28/04/20

Effectsizemeasures:

Betweensubjectsdesigns

Glass’sDelta

• Usedwhenstandarddeviationsdiffersubstantiallybetween

conditions.

• Choosethestandarddeviationofeitherthepreorpost

measurement.

‐ Chooseeffectsizethatbestrepresentstheeffectofinterest.

Effectsizemeasures:

Twoindependentgroups

Commonlanguageeffectsize(CLES)

• Nonparametriceffectsize

• Probabilityofaz‐scoregreaterthanadifferencebetweengroupsof

zero.

• ConvertsCohen’sdintoapercentage,expressedastheprobability

‐ arandomsampledpersonfromonegroupishigherthanarandomlysampled

personfromtheothergroup(betweendesign)

‐ anindividualhasahighervalueononemeasurementcomparedtotheother

(withindesign)

SeesupplementarymaterialfromLakens,2013forcalculationspreadsheet.

http://www.frontiersin.org/journal/10.3389/fpsyg.2013.00863/abstract

Effectsizemeasures:

Onesampleorcorrelateddata

• Usesthedifferencesbetweenmeasurements

• Standardizedmeandifferenceforwithinsubjectsdesign

• Cohen’sd

Fig.Cohen’sd

equationbasedonmeandifference Fig.Cohen’sd

equationbasedont‐test

(Lakens,2013p4) (Lakens,2013p4)

28/04/20

InterpretingEffectSizes

InterpretingCohen’sd

• Smalld=0.2

• Mediumd=0.5

• Larged=0.8

Notethesearearbitrary

andthebestwayistorelateresultstoothereffectsreportedintheliterature.

Whatistheclinicalimportanceoftheresult!

Reportingexample

Independentt‐test

Anindependentt‐testdidnotreportasignificantdifference

betweenlowerexamscoreformales(M=76.33SD=5.84)compared

tofemales(M=79.18SD=6.89;t(32)=2.11p=.071),althougha

mediumtolargeeffectwasfound(d=0.7395%CI[.53‐.86]).

*Noteexamplesarefictitiouswithvaluespresentedforillustrativepurposesonly.

Effectsizemeasures:

Threeormoreindependentgroups(e.g.ANOVA)

• Etasquaredη

(within)

• PartialEtasquaredη

(betweenstudies)

• Omegasquared

• Intraclass correlation

Degreeofassociation

forsample

Degreeofassociation

forpopulation

28/04/20

InterpretingEffectSizes

InterpretingEtasquared

• Small.01or1%

• Medium.06or6%

• Large.138or13.8%

Pallant (2020)p218

REMEMBERthesearearbitraryandthebestwayistorelateresultstoothereffectsreportedin

theliterature.

Whatistheclinicalimportanceoftheresult!

Reportingexample

Dependentt‐test

Andependentt‐testreportedasignificantincreasein3Ktimetrial

frompreseason(M=189secSD=33)tomidseason(M=151sec

SD=16;t(57)=6.91p=.002),withalargeeffect(η

=.37).

*Noteexamplesarefictitiouswithvaluespresentedforillustrativepurposesonly.

Reportingexample

ANOVA

GroupdifferencesexaminedusingANOVAfoundindependent

schoolshadsignificantlyhigherATARscores(M=89.1SD=5.2),

comparedtoCatholicschools(M=82.3SD=3.5)whowerehigher

thanpublicschools(M=73.7SD=16.2;F(2,1654)=4.53p=.019).

Despitereachingstatisticalsignificance,theactualmeandifference

betweenschooltypeswasquitesmall(η

=.02).

*Noteexamplesarefictitiouswithvaluespresented forillustrativepurposesonly.

28/04/20

Effectsizemeasures:

Chi‐squaretests

 Phi

‐ Twobinaryvariables

‐ RelatedtocorrelationandCohen’sd

‐ InterpretedlikePearson’srandR

 Cramer’sPhiorV

‐ Morethantwocategoricalvariables

‐ Measuresinter‐correlation

‐ Biasedasincreaseswiththenumberofcells

InterpretingEffectSizes

Interpretingphi(2x2)

• Small.10

• Medium.30

• Large.50

Pallant (2020)p228

InterpretingCramer’sV

 Twocategories:

• Small.01,Medium.30,Large.50

 Threecategories:

• Small.07,Medium.21,Large.35

 Fourcategories:

• Small.06,Medium.17,Large.29

REMEMBERthesearearbitraryandthebestwayistorelate

resultstoothereffectsreportedintheliterature.

Whatistheclinicalimportanceoftheresult!

Reportingexample

Chisquare

Achisquaregroupdifferencebetweensexandemploymentstatus

indicatedasignificantassociationofmediumeffect

(ꭓ

2

(1,n=186)=12.56p=.003,phi=.39).Morefemaleswereemployed

part‐timecomparedtomales(53%versus39%),whilemoremales

wereinfulltimeemployment(61%versus47%).

*Noteexamplesarefictitiouswithvaluespresentedforillustrativepurposesonly.

28/04/20

Calculatingeffectsizes

Abundanceofonlinecalculators

Lenhard andLenhard (2016)provideanumberofdifferentonline

calculatorsbasedonyoursampleandstudydesign.

Lenhard,W.&Lenhard,A.(2016). CalculationofEffectSizes.Retrieved

from: https://www.psychometrica.de/effect_size.html.Dettelbach (Germany):Psychometrica.

DOI:10.13140/RG.2.2.17823.92329

Reporting

example–

intext

Piggottetal.,2018p4

Reporting

example‐ Tables

Fleay,Brock,Joyce,Christopher,

Banyard,Harry,andWoods,CarlT.

(2018)Manipulatingfielddimensions

duringsmall‐sidedgamesimpactsthe

technicalandphysicalprofilesof

Australianfootballers.Journalof

StrengthandConditioning Research,

32(7).pp.2039‐2044.

Fleay etal.,2018p2042

28/04/20

CautionswhenusingEffectSizes

• Reportedcut‐pointsforinterpretingeffectsizesarearbitraryandthebestwayistorelateresultsto

othereffectsreportedintheliterature.

• Effectsizesaresensitivetospuriousinfluences,suchas:

‐ whichstandarddeviationisused,withpooledSDbetter

‐ pooledSDbasedonassumptionofestimatesfromthesamepopulation

‐ whethertherehavebeencorrections forbias

‐ whetherdatahasanormaldistribution

‐ reliabilityofthemeasurement

• Carewhencomparingeffectssizesofoutcomes

• ‘effectsize’expressionimpliescausalitysoshouldbeusedappropriately

i.e.isthisimplicationintendedorjustified.

Summary‐ effectsizes

• Effectsizeshelpcommunicatepracticalimportanceofaresult

‐ animportant butnonsignificantresult

‐ anunimportant butsignificant result

• Canbeajournalrequirement

• RecommendedaccordingtoAPAstyle

• GeneralPrincipal“…providethereaderswithenoughinformation

toassessthemagnitudeoftheobservedeffect.”

(APA2020p.89)

Summary

Interpreting

EffectSizes

Tablefrom Lenhard andLenhard

(2016)

SeeCoe(2002)forBESDandCLESinterpretationTableI.

https://www.leeds.ac.uk/educol/documents/00002182.htm

28/04/20

SummaryInterpretingEffectSizes

Table1fromSullivanandFeinn (2012).Usingeffectsize– orwhythepvalueisnotenough.

JournalofGraduateMedicalEducation.September:279‐282.DOI:http://dx.doi.org/10.4300/JGME‐D‐12‐00156.1

Summaryof

EffectSizes

d family

Table1fromLakens,2013p5

Summaryof

EffectSizes

r family

Table2fromLakens,2013p7

28/04/20

Referencesandfurtherreading

AmericanPsychologicalAssociation. (2020).PublicationmanualoftheAmericanPsychological Association(7

ed.).Https://doi.org/10.1037/0000165‐000

Coe,R.(2002).It’stheeffectsize,stupid:Whateffectsizeisandwhyitisimportant.PaperpresentedattheAnnualconferenceoftheBritishEducationResearch

Association, UniversityofExeter,England,12‐14September.https://www.leeds.ac.uk/educol/documents/00002182.htm

Lakens D.(2013).Calculatingandreportingeffectsizestofacilitatecumulativescience:apracticalprimerfort‐testsandANOVAs.FrontiersinPsychology4;836.

https://www.frontiersin.org/article/10.3389/fpsyg.2013.00863

Lenhard,W.&Lenhard,A.(2016).CalculationofEffectSizes.Retrievedfrom:https://www.psychometrica.de/effect_size.html.Dettelbach (Germany):

Psychometrica.DOI:10.13140/RG.2.2.17823.92329

Nandy,K.Understandingandquantifyingeffectsizes.https://www.sonnet.ucla.edu/sites/default/files/son_files/RESEARCH/effect_size_4‐9‐2012.pdf

Palant,J.2020. SPSSsurvivalmanual:AstepbystepguidetodataanalysisusingIBMSPSS(7

ed.).Allen&Unwin.

Sullivan,G.M.,&Feinn,R.(2012).UsingEffectSize‐orWhythePValueIsNotEnough.Journalofgraduatemedicaleducation, 4(3),279–282.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/