Aisyah Ali (1), Singgih Bektiarso (2), Auldry Fransje Walukow (3), Erlia Narulita (4), Akhmad Kadir (5)
(General Background) The alignment of science assessment with students’ socio-cultural contexts is essential to ensure fairness and meaningful measurement of learning outcomes. (Specific Background) However, most contextualized assessments in science education emphasize content validity without empirically confirming their construct structure, limiting their interpretive strength. (Knowledge Gap) There remains a lack of studies that integrate content, empirical, and construct validity evidence in culturally responsive instruments, particularly those designed in parallel pre–post forms. (Aims) This study aimed to develop and validate an ethnoscience-based pre–post instrument by linking Aiken’s Content Validity Ratio (CVR, 4-point scale) with Confirmatory Factor Analysis (CFA, CR/AVE). (Results) Findings from five expert reviews showed 22 of 40 items exceeded the conservative threshold (Aiken’s V ≥ 0.80; CVR = 1.00). Field trials (N = 50) demonstrated moderate difficulty and positive discrimination, while CFA confirmed a three-factor structure with good fit (χ² = 34.203, df = 24, p = 0.083; CFI = 0.94; TLI = 0.92; RMSEA = 0.065). Composite reliability ranged from 0.718–0.797, and AVE was adequate for two factors (0.506; 0.568) and marginal for one (0.459). (Novelty) The study presents a transparent “content–empirical–construct” decision trail rarely reported in ethnoscience assessment. (Implications) This integrative validation framework demonstrates that cultural responsiveness and psychometric rigor can coexist, guiding fair and contextual science learning evaluations.
Highlights:
Integrates Aiken–CVR and CFA for comprehensive validity evidence.
Confirms three-factor model with strong reliability and moderate AVE.
Demonstrates synergy between cultural relevance and measurement rigor.
Keywords: Content Validity, Confirmatory Factor Analysis, Ethnoscience, Culturally Responsive Assessment, Psychometric Validation
H. J. Boon and B. Lewthwaite, “Development of an Instrument to Measure a Facet of Quality Teaching: Culturally Responsive Pedagogy,” International Journal of Educational Research, vol. 72, pp. 38–58, 2015, doi: 10.1016/ijes.2015.05.002.
A. Ali and E. Kulimbang, “Efektivitas Bahan Ajar Tematik Terintegrasi Kearifan Lokal Khombow untuk Meningkatkan Hasil Belajar Siswa Sekolah Dasar di Kecamatan Sentani Timur Kabupaten Jayapura,” Pendas: Jurnal Ilmiah Pendidikan Dasar, vol. 10, no. 1, pp. 370–387, 2025, doi: 10.23969/jp.v10i01.22484.
N. Khairiyatul Mar’ah, A. Rusilowati, and E. Purwanti, “Development of Science Literature Instruments Containing Ethnoscience in Science Subject for Class IV Elementary School Students,” International Journal of Research and Review, vol. 8, no. 9, pp. 423–435, 2021, doi: 10.52403/ijrr.20210954.
L. R. Aiken, “Three Coefficients for Analyzing the Reliability and Validity of Ratings,” Educational and Psychological Measurement, vol. 45, no. 1, pp. 131–142, 1985, doi: 10.1016/j.ijer.2015.05.002.
H. Hendryadi, “Validitas Isi: Tahap Awal Pengembangan Kuesioner,” Jurnal Riset Manajemen dan Bisnis, vol. 2, no. 2, pp. 25–33, 2017.
C. H. Lawshe, “A Quantitative Approach to Content Validity,” Personnel Psychology, vol. 28, no. 4, pp. 563–575, 1975, doi: 10.1111/j.1744-6570.1975.tb01393.x.
D. F. Polit and C. T. Beck, “The Content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations,” Research in Nursing & Health, vol. 29, no. 5, pp. 489–497, 2006, doi: 10.1002/nur.20147.
S. BouJaoude, “Balance of Scientific Literacy Themes in Science Curricula: The Case of Lebanon,” International Journal of Science Education, vol. 24, no. 2, pp. 139–156, 2002, doi: 10.1080/09500690110066494.
C. A. Wynd, B. Schmidt, and M. A. Schaefer, “Two Quantitative Approaches for Estimating Content Validity,” Western Journal of Nursing Research, vol. 25, no. 5, pp. 508–518, 2003, doi: 10.1177/0193945903252998.
H. Baharum et al., “Validating an Instrument for Measuring Newly Graduated Nurses’ Adaptation,” International Journal of Environmental Research and Public Health, vol. 20, no. 4, 2023, doi: 10.3390/ijerph20042860.
H. G. Sakti, M. A. Rizka, I. W. Ayu, and F. Ariany, “Development of Student Pancasila Character Instruments: Evidence of the EFA, CFA and Rasch Models,” Jurnal Kependidikan: Jurnal Hasil Penelitian dan Kajian Kepustakaan di Bidang Pendidikan, Pengajaran dan Pembelajaran, vol. 9, no. 4, pp. 1092–1105, 2023, doi: 10.33394/jk.v9i4.9178.
R. B. Kline, Principles and Practice of Structural Equation Modeling, 4th ed., New York, NY: Guilford Publications, 2023.
T. A. Brown, Confirmatory Factor Analysis for Applied Research, 2nd ed., New York, NY: Guilford Publications, 2015.
L. T. Hu and P. M. Bentler, “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives,” Structural Equation Modeling, vol. 6, no. 1, pp. 1–55, 1999, doi: 10.1080/10705519909540118.
R. C. MacCallum, M. W. Browne, and H. M. Sugawara, “Power Analysis and Determination of Sample Size for Covariance Structure Modeling,” Psychological Methods, vol. 1, no. 2, pp. 130–149, 1996, doi: 10.1037/1082-989X.1.2.130.
C. Fornell and D. F. Larcker, “Evaluating Structural Equation Models with Unobservable Variables and Measurement Error,” Journal of Marketing Research, vol. 18, no. 1, pp. 39–50, 1981, doi: 10.1177/002224378101800104.
M. R. Lynn, “Determination and Quantification of Content Validity,” Nursing Research, vol. 35, no. 6, pp. 382–386, 1986, doi: 10.1016/nr.2015.05.002.
D. Lewis and R. Cook, “Embedded Standard Setting: Aligning Standard-Setting Methodology with Contemporary Assessment Design Principles,” Educational Measurement: Issues and Practice, vol. 39, no. 1, pp. 8–21, 2020, doi: 10.1111/emip.12318.
K. Brauer, J. Ranger, and M. Ziegler, “Confirmatory Factor Analyses in Psychological Test Adaptation and Development: A Non-Technical Discussion of the WLSMV Estimator,” Psychological Test Adaptation and Development, vol. 4, no. 1, pp. 4–12, 2023, doi: 10.1027/2698-1866/a000034.
H. Yulianto, “Maslach Burnout Inventory-Human Services Survey (MBI-HSS) Bahasa Indonesia Version: Construct Validation Study Among Police Officers,” Jurnal Pengukuran Psikologi dan Pendidikan Indonesia, vol. 9, no. 1, pp. 19–29, 2020, doi: 10.15408/jp3i.v9i1.13329.
E. B. Smit et al., “Development of a Patient-Reported Outcomes Measurement Information System Short Form for Measuring Physical Function in Geriatric Rehabilitation Patients,” Quality of Life Research, vol. 29, no. 9, pp. 2563–2572, 2020, doi: 10.1007/s11136-020-02506-5.
W. N. F. Abdol Jani, F. Razali, N. Ismail, and N. Ismawi, “Exploratory Factor Analysis: Validity and Reliability of Teacher’s Knowledge Construct Instrument,” International Journal of Academic Research in Progressive Education and Development, vol. 12, no. 1, pp. 1–14, 2023, doi: 10.6007/ijarped/v12-i1/16236.
F. H. M. Hatta, E. Z. Samsudin, N. Aimran, and Z. Ismail, “Development and Validation of Questionnaires to Assess Workplace Violence Risk Factors (QAWRF): A Tripartite Perspective of Worksite-Specific Determinants in Healthcare Settings,” Risk Management and Healthcare Policy, vol. 16, pp. 1229–1240, 2023, doi: 10.2147/RMHP.S411335.
C. Brown and J. Templin, “Modification Indices for Diagnostic Classification Models,” Multivariate Behavioral Research, vol. 58, no. 3, pp. 580–597, 2022, doi: 10.1080/00273171.2022.2049672.
C. Ayre and A. J. Scally, “Critical Values for Lawshe’s Content Validity Ratio: Revisiting the Original Methods of Calculation,” Measurement and Evaluation in Counseling and Development, vol. 47, no. 1, pp. 79–86, 2014, doi: 10.1016/mec.2015.05.002.
E. Bumbálková, “Test-Taking Strategies in Second Language Receptive Skills Tests: A Literature Review,” International Journal of Instruction, vol. 14, no. 2, pp. 647–664, 2021, doi: 10.29333/iji.2021.14236a.
G. Cho, H. Hwang, M. Sarstedt, and C. M. Ringle, “Cutoff Criteria for Overall Model Fit Indexes in Generalized Structured Component Analysis,” Journal of Marketing Analytics, vol. 8, no. 4, pp. 189–202, 2020, doi: 10.1057/s41270-020-00089-1.
A. K. Montoya and M. C. Edwards, “The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation,” Educational and Psychological Measurement, vol. 81, no. 3, pp. 413–440, 2021, doi: 10.1177/0013164420942899.
D. Shi and A. Maydeu-Olivares, “The Effect of Estimation Methods on SEM Fit Indices,” Educational and Psychological Measurement, vol. 80, no. 3, pp. 421–445, 2020, doi: 10.1177/0013164419885164.
S. C. Bourke et al., “Developing Aboriginal and Torres Strait Islander Cultural Indicators: An Overview from Mayi Kuwayu, the National Study of Aboriginal and Torres Strait Islander Wellbeing,” International Journal for Equity in Health, vol. 21, no. 1, p. 1, 2022, doi: 10.1186/s12939-022-01710-8.
A. Nguyen et al., “Measuring the Psychosocial Wellbeing of Rohingya Mothers in Cox’s Bazar, Bangladesh: Psychometric Properties of an MHPSS Assessment Battery,” F1000Research, vol. 11, p. 1402, 2022, doi: 10.12688/f1000research.125654.1.
A. Rasmussen and N. Jayawickreme, “Introduction to the Special Collection: Developing Valid Psychological Measures for Populations Impacted by Humanitarian Disasters,” Conflict and Health, vol. 14, no. 1, p. 1, 2020, doi: 10.1186/s13031-020-00260-6.