pophealthresourceweek6.pdf

Original Article

Evaluating the Impact of EBP Education:
Development of a Modified Fresno Test for
Acute Care Nursing
Margo A. Halm, PhD, RN, NEA-BC

Keywords

modified Fresno,
EBP education/
competencies,

acute care nursing,
novice-to-expert,

psychometrics

ABSTRACT
Background: Proficiency in evidence-based practice (EBP) is essential for relevant research find-
ings to be integrated into clinical care when congruent with patient preferences. Few valid and
reliable tools are available to evaluate the effectiveness of educational programs in advancing
EBP attitudes, knowledge, skills, or behaviors, and ongoing competency. The Fresno test is one
objective method to evaluate EBP knowledge and skills; however, the original and modified
versions were validated with family physicians, physical therapists, and speech and language
therapists.

Aims: To adapt the Modified Fresno-Acute Care Nursing test and develop a psychometrically
sound tool for use in academic and practice settings.

Methods: In Phase 1, modified Fresno (Tilson, 2010) items were adapted for acute care nursing.
In Phase 2, content validity was established with an expert panel. Content validity indices (I-CVI)
ranged from .75 to 1.0. Scale CVI was .95%. A cross-sectional convenience sample of acute care
nurses (n = 90) in novice, master, and expert cohorts completed the Modified Fresno-Acute Care
Nursing test administered electronically via SurveyMonkey.

Findings: Total scores were significantly different between training levels (p < .0001). Novice nurses scored significantly lower than master or expert nurses, but differences were not found between the latter cohorts. Total score reliability was acceptable: (interrater [ICC (2, 1)]) = .88. Cronbach’s alpha was 0.70. Psychometric properties of most modified items were satis- factory; however, six require further revision and testing to meet acceptable standards. Linking Evidence to Action: The Modified Fresno-Acute Care Nursing test is a 14-item test for objectively assessing EBP knowledge and skills of acute care nurses. While preliminary psycho- metric properties for this new EBP knowledge measure for acute care nursing are promising, further validation of some of the items and scoring rubric is needed. INTRODUCTION Over a decade ago, the Institute of Medicine (Institute of Medicine [IOM], 2001) recognized evidence-based practice EBP as a key solution to ensure care delivered has the high- est clinical effectiveness known to science. To reach the IOM’s (2007, p. ix) 2020 goal that “90% of clinical decisions will be supported by accurate, timely and up-to-date clinical informa- tion that reflects the best available evidence,” nurses need EBP competencies to guarantee that relevant research findings are integrated into clinical situations when congruent with patient preferences (Melnyk, Gallagher-Ford, Long, Long, & Fineout- Overholt, 2014). BACKGROUND A recent evidence synthesis reported 10 studies evaluating the effectiveness of educational interventions in building EBP attitudes, knowledge, skills, and behaviors of nurses (Halm, 2014). Interventions were primarily workshop or immersion programs, but seminars, journal clubs, and EBP and research councils were also evaluated via: (a) self-reported EBP attitude, knowledge, and behavior (Chang et al., 2013; Dizon, Somers, & Kumar, 2012; Edward & Mills, 2013; Leung, Trevana, & Waters, 2014); (b) PICO questions and activity diaries (Dizon et al., 2012); (c) Edmonton Research Orientation (Gardner, Smyth, Renison, Cann, & Vicary, 2012) and Clinical Effectiveness or EBP Questionnaire (Sciarra, 2011; Toole, Stichler, Ecoff, & Kath, 2013; White-Williams et al., 2013); and (d) interviews and focus groups to identify qualitative themes about nurses’ expe- rience in EBP programs (Balakas, Sparks, Steurer, & Bryant, 2013; Nesbitt, 2013; Wendler, Samuelson, Taft, & Eldridge, 2011). Varied measurement across studies limited estimation of the effectiveness of EBP training (Dizon et al., 2012). In a systematic review, Shaneyfelt et al. (2006) rec- ommended valid and responsive methods to evaluate the programmatic impact of EBP education and progression in 272 Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International CE http://crossmark.crossref.org/dialog/?doi=10.1111%2Fwvn.12291&domain=pdf&date_stamp=2018-05-14 EBP competencies. As self-report is extremely biased (Lai & Teng, 2011; Shaneyfelt et al., 2006); objective knowledge tests that incorporate multiple-choice or short answers with case-based decision-making like the Berlin Questionnaire (Fritsche, Greenhalgh, Falck-Ytter, Neumayer, & Kunz, 2002) or Fresno test were recommended to evaluate EBP knowledge and skills (Shaneyfelt et al., 2006). The Fresno test, a valid and reliable method to evaluate EBP knowledge and skills using a standardized scoring rubric, has been validated with family physicians (Ramos et al., 2003), physical therapy (Miller, Cummings, & Tomlinson, 2013; Tilson, 2010), and speech language (Spek, de Wolf, van Dijk, & Lucas, 2012). SPECIFIC AIMS As objective methods for assessing EBP knowledge and skills of nurses are lacking, the specific aim of this study was to fill a measurement gap by adapting the modified Fresno test (Tilson, 2010) for acute care nursing. Only with consistent use of psy- chometrically sound methods can useful evidence be generated about the effectiveness of various EBP teaching strategies— new knowledge that can direct effective educational and pro- fessional development programs for students and practicing nurses. The specific research question was: Will an adapted Fresno test discriminate EBP knowledge and skills between novice, master, and expert acute care nurses? METHODS Research Design A cross-sectional cohort design was used to replicate Tilson’s (2010) modified Fresno test (Figure 1). Phase I: Test adaptation. New scenarios on acute care nurs- ing were developed for items #1–8 that remained unchanged. Item #9 (clinical expertise) was retained despite removal due to poor psychometric performance by Tilson (2010). Items #10–13 were modified for acute care although the EBP focus was un- changed. Item #14 was modified to the best design for studying the meaning of experience. Phase 2: Content validity. Content validity was established with a panel of four masters and doctorally prepared acute care EBP experts from practice and academic settings. In round one, panelists rated each item and rubric for clarity, impor- tance, and comprehensiveness on a 5-point Likert scale. Pan- elists provided feedback on whether items should be retained, revised, dropped, or added (Polit & Beck, 2012). In round two, items #10 (mathematical calculations for sensitivity, positive predictive value) and #11 (relative and absolute risk reduction) were replaced because the panel did not believe acute care nurses would be expected to make these calculations without a resource. These items were replaced (and reviewed) with assessing tool reliability/validity and applying qualitative find- ings. The scoring rubric (Figure S1) was modified to reflect item alterations and ensure scoring consistency across subjects and raters (Jonsson & Svingby, 2007). With a single overall score, Figure 1. Study flowchart. a passing score was defined as >50% of available points for in-
dividual items (Tilson, 2010). This passing score was set lower
than that defined as “mastery of material” (Ramos, Schafer, &
Tracz, 2003) to reduce the risk of a floor effect with novices.

A content validity index (I-CVI) was calculated for individ-
ual items by dividing the number of 4–5 ratings by the number
of experts. Mean (M) item ratings were 4.54 (clarity), 4.82 (im-
portance), and 4.75 (comprehensiveness). Only item 12 had an
I-CVI value <0.78 because the panel rated interpreting con- fidence intervals lower on importance for acute care nurses. The scale CVI of .95% was calculated by averaging I-CVIs, exceeding acceptable standards of >.90 (Polit & Beck, 2007;
Table 1).

Phase 3: Validation of modified Fresno. After Institu-
tional Review Board exemption was obtained, invitations were
emailed to three cohorts: (a) novice nurses (less than 2 years of

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280.
© 2018 Sigma Theta Tau International

273

Original Article
Table 1. Modified Fresno Test Items (n = 90)

Scores

Item/EBPstepor
component Topic

Content
validity index
(I-CVI)

Possible
score

Passing
score

Novices
(n = 30)
M (SD)

Masters
(n = 30)
M (SD)

Experts
(n = 30)
M (SD) p value*

1 INQUIRE PICOquestion .92 0–24 >12 13.73 (7.37) 19.47 (3.71) 18.13 (4.55) .001 (N-M,N-E)

2ACQUIRE Sources 1.0 0–24 >12 15.03 (6.53) 20.33 (5.09) 17.53 (6.05) .004 (N-M)

3APPRAISE Treatment
design

1.0 0–24 >12 5.80 (6.77) 10.50 (6.90) 11.90 (5.87) .001 (N-M,N-E)

4ACQUIRE Search .92 0–24 >12 13.93 (5.06) 16.53 (4.69) 15.10 (4.69) .18

5APPRAISE Relevance .92 0–24 >12 7.47 (6.31) 9.77 (6.83) 12.03 (6.72) .03 (N-E)

6APPRAISE Validity .92 0–24 >12 7.30 (6.75) 10.67 (7.77) 10.23 (7.38) .16

7APPRAISE Significance 1.0 0–24 >12 3.40 (3.94) 9.97 (8.18) 7.70 (7.03) .001 (N-M,N-E)

8PATIENT
PREFERENCES

Patient
preference

1.0 0–16 >8 6.13 (4.36) 8.20 (5.59) 9.00 (4.95) .08

9CLINICAL
EXPERTISE

Clinical
expertise

1.0 0–8 >4 4.80 (3.04) 5.60 (2.49) 6.40 (2.49) .08

10APPLY Tools .92 0–12 >6 3.90 (4.18) 8.50 (3.35) 7.00 (4.12) .001 (N-M,N-E)

11APPLY Qualitative 1.0 0–16 >8 12.13 (4.75) 10.93 (5.35) 12.53 (6.19) .50

12APPRAISE Confidence
intervals

.75 0–4 >2 .13 (.73) .40 (1.22) 1.07 (1.80) .02 (N-E)

13APPRAISE Design
diagnosis

1.0 0–4 >2 .27 (1.01) .27 (1.01) .27 (1.01) 1.00

14APPRAISE Design
meaning

1.0 0–4 >2 2.13 (2.03) 3.73 (1.01) 3.87 (.73) .001 (N-M,N-E)

Total scores .95ScaleCVI 0–232 >116 96.17 (26.14) 134.87 (30.76) 132.77 (28.94) .001 (N-M,N-E)

*Scheffe post-hoc analysis: N = Novices;M = Masters; E = Experts.

experience after graduation from a bachelorette program) from
three U.S. Magnet hospitals; (b) master nurses (master’s pre-
pared) recruited via the National Association of Clinical Nurse
Specialists listserv; and (c) expert nurses (doctorally prepared)
recruited via the American Nurses Credentialing Corporation’s
Magnet program director’s listserv and faculty at Bethel Uni-
versity (St. Paul, MN, USA). Nurses in the expert cohort self-
affirmed their EBP expertise and teaching experience. Up to
1 hr (in one sitting) was allowed to complete the test with no
external resources; only notepaper and calculators were per-
mitted. Reminder e-mails were sent at 2 and 4 weeks. A $10
gift certificate incentive was offered upon completion. Some
participants did not answer all the items on the exam; these
participants were not included in the sample for each cohort.
Only participants who had a complete exam were included in
the analysis. Data were collected in 2015.

Two doctorally prepared nurses with expertise teaching EBP
served as raters after an orientation to the test items and scor-

ing rubric. Raters practiced scoring three pilot tests from the
three cohorts and resolved discrepancies that could threaten in-
terrater reliability (IRR; e.g., halo effect, leniency or stringency,
central tendency errors; Castorr et al., 1990; before scoring
commenced. A midway refresher session allowed raters to re-
view scores, reducing the threat of rater drift (Castorr et al.,
1990). Data were analyzed with SPSS Version 23.0 (IBM Corp.,
Armonk, NY, USA).

RESULTS
Descriptive Statistics
The total sample of 90 nurses included cohort (a) new grad-
uates (n = 30); (b) master’s prepared CNSs (n = 30); and
(c) doctorally prepared nurses (n = 30). Seventy-six percent
completed the test within 60 min (83% novices, 70% mas-
ters, 73% experts). Mean min for test completion were 56.43

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280.
© 2018 Sigma Theta Tau International

274

Development of a Modifi ed Fresno Test for Acute Care Nursing

Table 2. Psychometric Properties of Individual Items (n = 90)

%Passedbycohort

Item# Topic ICC IDI ITC
All

(n = 90)
Novices
(n = 30)

Masters
(n = 30)

Experts
(n = 30) χ2 p-value

1 PICOquestion .78 .43 .53 85.6 63.3 100.0 93.3 18.52 .0001

2 Sources .78 .35 .53 84.4 73.3 93.3 86.7 4.74 .09

3 Treatmentdesign .86 .61 .56 44.4 26.7 50.0 56.7 6.03 .05

4 Search .72 .26 .48 80.0 76.7 86.7 76.7 1.25 .54

5 Relevance .48 .65 .63 35.6 26.7 33.3 46.7 2.72 .26

6 Validity .47 .43 .50 32.2 20.0 43.3 33.3 3.76 .15

7 Significance .74 .52 .57 26.7 6.7 40.0 33.3 9.55 .01

8 Patient
preference

.55 .52 .39 52.2 36.7 50.0 70.0 6.77 .03

9 Clinical expertise .23 .22 .40 88.9 80.0 93.3 93.3 3.60 .17

10 Tools .76 .74 .68 68.9 40.0 90.0 76.7 18.77 <.0001 11 Qualitative .68 .17 .31 88.9 93.3 90.0 83.3 1.58 .46 12 Confidence intervals .90 .04 .12 13.3 3.3 10.0 26.7 7.50 .02 13 Designdiagnosis .61 .13 .12 6.7 6.7 6.7 6.7 .00 1.0000 14 Designmeaning .89 .35 .37 81.1 53.3 93.3 96.7 22.77 <.0001 Total score .88 N/A N/A .0001 (standard deviation [SD] 38.21) for novices; 57.20 (SD 42.54) for masters; and 43.21 (SD 26.33) for experts. Reliability Statistics IRR was calculated using intraclass correlation coefficients (ICC) for total score and individual items (Table 2). Total score reliability was high at .88. Of the 14 items, 3 had excellent reliability (>.80), 7 had moderate reliability (.60–.79), and 4
had questionable reliability (<.60). Items with questionable IRR focused on relevance (#5), validity (#6), patient preference (#8), and clinical expertise (#9). A Cronbach’s alpha coefficient of .70 was obtained for internal consistency of the modified exam. Item discrimination index (IDI) was calculated for each item by separating total scores into quartiles and subtracting the pro- portion of nurses in the bottom quartile who passed that item (>50% points per item was passing) from the proportion in the
top quartile who passed the same item. The 50% threshold has
been defined as “mastery of material” (Ramos et al., 2003) and
used in similar validation studies (Tilson, 2010). IDI ranges
from –1.0 to 1.0, representing the difference in passing rate
between nurses with high (top 25%) and low (bottom 25%)

overall scores. Eleven of the 14 items had acceptable IDIs >.2
(Table 2). Correlation between item and total score and cor-
rected item-total correlation (ITC) was assessed using Pearson
correlation coefficients. Twelve of the 14 items had acceptable
ITCs >.3 (Table 2). Low IDI and ITC items focused on con-
fidence intervals (#12) and design for diagnostic tests (#13).
Qualitative findings (#11) also had a low IDI.

Total Score Analysis
No floor or ceiling effect was apparent, indicating the test is ap-
plicable from novice to expert (Figure 2). As shown in Table 1,
total mean scores for novices (M 96.17, SD 26.14) revealed
that a passing score of 116 was not achieved in this cohort as
with the master (M 134.87, SD 30.76) and expert (M 132.71,
SD 28.94) cohorts. One-way analysis of variance (ANOVA)
demonstrated that overall mean scores were significantly dif-
ferent, F (2, 89) = 17.58, p < .0001, between cohorts. A post- hoc Scheffe comparison showed novice total mean scores (M 96.17, SD 26.14) differed significantly from master (M 134.87, SD 30.07, d = 1.36) and expert nurses (M 132.77, SD 28.94, d = 1.33). Cohen’s d is an effect size measure that is used to explain the standardized difference between two means, Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International 275 Original Article Figure 2. Box plots for sum scores. commonly reported with ANOVAs or t tests. There were no significant differences between the master and expert cohorts. Item Score Comparison Post-hoc Scheffe analysis also revealed significant cohort dif- ferences in eight items (Table 1). Novice nurses scored sig- nificantly lower than master and expert nurses on PICO (#1), sources (#2), treatment design (#3), relevance (#5), significance (#7), tools (#10), confidence intervals (#12), and design mean- ing (#14). On the other hand, the mean scores for four items increased progressively across cohorts from novice to master, and then from master to expert. These items were treatment design (#3), relevance (#5), patient preference (#8), and con- fidence intervals (#12). While not all items performed in this manner, these items demonstrated mastery of EBP material across cohorts. Item Difficulty Item difficulty (IDI) was calculated via the proportion of nurses who achieved a passing score for each item (Table 2). Of the 14 items, none were easy (IDI > .8). Ten items (71%) were
moderate (IDI > .3), and 4 (29%) were difficult (IDI < .3; Janda, 1998; Nunnally & Bernstein, 1994). In testing individual items, all three cohorts scored below the passing cutoff for five items: Treatment design (#3), validity (#6), significance (#7), confidence intervals (#12), and diagnosis design (#13). Novice and master nurses did not achieve a passing score for relevance (#5), while novices did not pass patient preferences (#8) and tools (#10). Using chi-square analysis, seven items showed significant differences in the proportion of passing scores between cohorts (Table 2). Masters scored highest on PICO (#1), significance (#7), and tools (#10). Experts performed best on treatment de- sign (#3), design meaning (#14), patient preferences (#8), and confidence intervals (#12). In examining item discrimination based on the propor- tion of nurses who passed the test (Table 2), some significant items did not discriminate well between masters and experts: (a) PICO (#1); (b) treatment design (#3); (c) significance (#7); and (d) design meaning (#14). Items on sources (#3), search (#4), relevance (#5), validity (#6), and expertise (#9) discrim- inated on the IDI but did not assess unique EBP knowledge and skills among the three cohorts (p > .05).

DISCUSSION
The Modified Fresno-Acute Care Nursing test is a 14-item test
for assessing EBP knowledge and skills. While the original
test assessed core principles of EBP steps, this replication val-
idated patient preferences and clinical expertise to fully assess
all EBP domains. The test has excellent content validity with
I-CVIs ranging from .75 to 1.0. Overall scale CVI was .95. In-
ternal consistency was acceptable at .70. Table 3 compares the
psychometric properties of the Modified Fresno-Acute Care
Nursing test with the original and modified tests.

Total scale reliability for the two independent raters was
excellent (.88). IRR for individual items was good to excellent
for 10 of 14 items (71%). One reason IRR may have been lower
for relevance (#5) and validity (#6) was the rubric complexity
that required raters to consider responses for both items when
scoring. Like Tilson (2010), IRR was less than desirable for pa-
tient preference (#8) and clinical expertise (#9). Some leniency
in scoring may have occurred with #8 when a nurse offered a
phrase that could elicit patient preferences, rather than stating
it as a question as specified in the rubric. As recommended by
Tilson (2010), clinical expertise should be retained as it covers
an essential EBP domain, but further revision and validation is
needed.

Item difficulty was moderate to high. Two items retained
from Tilson’s (2010) version had low IDI and ITC: Confidence
intervals (#12) and design for diagnosis (#13). These items were
difficult across cohorts and did not discriminate. Of the new
items, tools (#10) had acceptable psychometrics across ICC,
IDI, and ITC. The second qualitative item (#11) had accept-
able ICC and ITC but low IDI and did not discriminate across
cohorts. This finding may demonstrate that qualitative find-
ings have a rich tradition of emphasis across levels of nursing
education and practice.

While some items did not perform ideally, these items re-
main valuable to the larger research goal of developing an
objective and responsive method to evaluate EBP knowledge
and skills. Reasons for poor item performance may include
item characteristics, unknown sample characteristics, scoring
concerns, or a combination of these factors. Six items (#5, #6,
#9, #11, #12, and #13) need to be revised and retested before be-
ing removed. Although Tilson (2010) dropped clinical expertise
(#9), it covers an important EBP domain that other researchers
recognized as essential for measurement (Miller et al., 2013).

A range in item difficulty is best so that the high and low
range of ability can be evaluated. For item #12 (confidence

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280.
© 2018 Sigma Theta Tau International

276

Development of a Modifi ed Fresno Test for Acute Care Nursing

Table 3. Comparison of Reliability and Validity of Fresno Tests

Performance

Measure/acceptable
results

Original Fresno (Ramos
et al., 2003)

DutchadaptedFresno
(Speket al., 2012)

ModifiedFresno-physical
therapy (Tilson, 2010)

ModifiedFresno-AcuteCare
Nursing test (Halm, 2018

current study)

Population � Familyphysicians � Speech
language, clinical
epidemiology
students

� Physical therapy � Acute care nurses

Total score/# items � 212/12 � 212/12 � 224/13 � 232/14

Content validity

� ScaleCVI/>.90 � Not reported � .92 � Not reported � .95

Interrater reliability

� Interrater
correlation/

�
>.60

� Items: .72–.96
� Total score: .97

� Not reported
� Total score: .99

� Items: .41–.99
� Total score: .91

� Items: .23–.90
� Total score: .88

Internal reliability

� Cronbach’s/>.70
� Item-total
correlation
(ITCs)/>.30

� .88
� .47–.75 (items)

� .83
� .31–.76

� .78
� .20–.66

� .70
� .12–.68

Itemdiscrimination

� Item
discrimination
index (IDI)/>.20

� .41–.86; no items
hadweakor
negative
discrimination

� Not reported � .25–.68; no items
hadweakor
negative
discrimination

� .04–.74; 3 itemshad
weakdiscrimination

Construct validity

� Comparisonof
meancohort
scores

� Novice = 95.6+
� Expert = 147.5;
morepassedall
items (p < .05) � Year 1 students = 26.3* � Year 2 students = 69.3* � Year 3 students = 89.1* � Masters students = 154.2* � Novice = 92.8 � Trained = 118.5 � Expert = 149.0++; morepassed 11 items (p < .03–.01) � Novices = 96.17++ � Masters = 134.87; morepassed3 items (p < .01–.0001) � Experts = 132.77; morepassed4 items (p < .01–.0001) *p < .05; +p < .001; ++ p < .0001. intervals), the IDI was low, most likely due to the low base success rate; however, it did discriminate the high end of EBP knowledge among cohorts. This item replaced a mathemati- cal calculation and should be retained because of the growing importance of understanding confidence intervals, although it may need to be revised. Similarly, item #13 (design diagnosis) was difficult. This item should be retained but reworded to in- crease clarity that it is referring to selection and interpretation of diagnostic tests. Item #14 (design meaning) may have been too easy. This item should be retained but reworded, so it is more difficult. Since item #11 was labeled qualitative, it may have primed nurses, and so item #14 (design meaning) should be moved earlier in the test. Based on ITC performance, the rubric for item #11 (qualitative) needs to be more difficult, requiring more specific or unusually helpful or insightful advice to better differentiate between a best possible (16 points) answer versus a more limited (8 points) answer. No floor or ceiling effects were evident, indicating that EBP knowledge and skills, and not clinical experience, influenced mean score differences (Tilson, 2010). Mastery of EBP material was evident from novice to expert nurses on four items. The Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International 277 Original Article Table 4. Uses of the Modified Fresno-Acute Care Nursing Test Self-assessment Pre–post assessment Academic settings 1. Students could use individual itemsand scoring rubric asaguidewhen learningeach EBPstep/component 2. Educators couldperiodically take the test before andafter teachingEBPcourses to identify areas for continual learning to advance levels of EBPexpertise 1. Faculty could usepre–post scores to evaluate EBP education in academicprograms (BSN,MSN,DNP, PhD). Test scores could assist curriculum design/redesign, andassessment of thequality/ rigor of course content, teaching styles, and methods 2. Objective test scores could showhowstudent outcomesare improving, data that canbeused for accreditationpurposes Acute care settings 1. Clinical/advancedpractice nurses canuse individual itemsand scoring rubric as a guide for learning eachEBP step/component 2. Clinical nurses could take the test to assess EBPstrengths andareas for improvement before attendingEBPeducational activities (Ramoset al., 2003) 1. Acute care educators and researchers could use pre–post scores to evaluate EBPeducation for clinical nurses � Identifiedgapswould informneeds for orientation/ongoing staff development opportunities that advanceEBPcompetencies 2. Scores could be tracked tomonitor EBP knowledge/skill progressionof nurses in attaining higher levels of EBPcompetency. A 10%change is meaningful in evaluating improvement in EBPskills over time (McCluskey&Bishop, 2009) � EBPknowledge/skills could beassessed for new hires, existing nurses, aswell asmembers of journal clubs, EBP/researchandpolicy/ procedure committees responsible for revising policies/procedures/protocols/guidelines based onbest available evidence ability of the test to differentiate between novice nurses and masters or experts was high but not across all three cohorts. Historical threats to validity may be one explanation. As an evolving concept, some nurses may not have had similar ex- posure to EBP in doctoral education. Interestingly, acute care nurses had longer times to completion (M 56.43, SD 38.21 for novices; M 57.20, SD 42.54 for masters; M 43.21, SD 26.33 for experts) than those reported by Tilson (M 33.2, SD 8.7 for novices; M 34.8, SD 10.0 for masters; M 40.5, SD 15.5 for ex- perts). These differences may be due to the sample or changes in the Fresno test. EVIDENCE TO ACTION The findings from this sample suggest EBP topics need re- inforcement with acute care nurses in academic and practice settings. Acute care nurses at all levels would benefit from more education on appropriateness of designs for different research questions, as well as assessment of validity, clinical and statistical significance, and confidence intervals. Novice nurses need more guidance in assessing patient preferences and applicability of tools for practice. Both novice and master nurses need more education on assessing study relevance. Ar- eas for EBP education or reinstruction should align with the national EBP competencies developed by Melnyk et al. (2014) for clinical and advanced practice nurses. These competencies provide the road map for expected levels of EBP in the clinical setting. Scores derived from the Modified Fresno-Acute Care Nurs- ing test have many uses in both the academic and prac- tice setting. As described in Table 4, the test and scoring rubric can be used as self-study and assessment guides. While test scores could be used in a pre–post fashion to docu- ment the impact of educational programs in advancing EBP knowledge and skills and competencies of acute care nurses, the Modified Fresno-Acute Care Nursing test needs to un- dergo further validation before such use occurs in practice or academia. LIMITATIONS The first limitation is the lack of demographic information for this small U.S. sample. Length of time since graduation and years of EBP experience were not captured and may have Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International 278 Development of a Modifi ed Fresno Test for Acute Care Nursing influenced performance in the test. The sample of doctorally prepared nurses who were recruited as EBP experts is a further limitation because the test did not differentiate well between experts and masters. Experts spent on average 13 min less time to complete the test and thus, may not have thoroughly docu- mented their EBP knowledge. The scores obtained in these sample cohorts are not generalizable globally to acute care nurses because the emphasis and amount of EBP education may differ in general and across levels of nursing education in developing or developed countries (Ciliska, 2005; Deng, 2015; Holland & Magama, 2017). Secondly, the scoring rubric is complex. Raters need EBP experience and training to ensure reliable use of the rubric. Pilot testing with opportunities to clarify scoring procedures is essential for IRR. At least 10–15 min per test should be allocated (Ramos et al., 2003; Tilson, 2010). This scoring time could be a limitation if an educator or researcher desires an easy assess- ment to evaluate …