Evaluating the reliability of mathematical literacy assessment for pre-service teachers: A generalizability theory approach
DOI:
https://doi.org/10.30606/absis.v9i1.3585Keywords:
Assessment Reliability, Mathematical Literacy, Pre-Service Teachers, reliabilitas asesmen, Generalizability Theory, literasi matematisAbstract
Reliable assessment of pre-service teachers' mathematical literacy skills is crucial to ensure their readiness to teach. However, conventional reliability indices often fail to disentangle various sources of measurement error simultaneously. This study applies Generalizability Theory (G-Theory) to evaluate and diagnose the reliability of a mathematics literacy instrument. A two-facet nested design, denoted as , was employed involving 30 undergraduate students (p) with six open ended essay item (i) evaluated by three independent raters (r). A G-study was conducted to estimated variance components, revealing that the largest source of variance originated from raters (92.8%). The initial analysis yielded d generalizability coefficient of 0.27 and dependability coefficient of 0.01, indicating low reliability due to substantial rater inconsistency. A D-study was conducted to evaluate alternative designs, however the results showed that even by increasing the number of items and raters, the G-coefficient 0.38 above the threshold of 0.80. These findings highlight that rater effects pose a major threat to measurement stability and suggest that the current instrument requires fundamental refinement of its scoring rubrics and procedures. This study serves as a critical diagnostic for teacher education programs, providing an empirical basis for optimizing assessment design and institutionalizing rater calibration to improve the evaluation of future educators.
Downloads
References
Allen, M. J., & Yen, W. M. (2001). Introduction to measurement theory. Waveland Press.
Ayuningtyas, N., & Sukriyah. (2020). Analisis pengetahuan numerasi mahasiswa matematika calon guru. Delta-Pi: Jurnal Matematika dan Pendidikan Matematika, 12(1), 39–48.
Basri, H., Kurnadi, B., & Tafriliyanto, C. F. (2021). Investigasi kemampuan numerasi. Proximal: Jurnal Penelitian Matematika dan Pendidikan Matematika, 4(2), 72–79.
Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
Bulková, Z., Gašparík, J., Mašek, J., & Zitrický, V. (2022). Analytical procedures for the evaluation of infrastructural measures for increasing the capacity of railway lines. Sustainability, 14(21), Article 14430. https://doi.org/10.3390/su142114430
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley & Sons.
Huebner, A., & Lucht, M. (2019). Generalizability theory in R. Practical Assessment, Research & Evaluation, 24(5), 1–15.
Mardapi, D. (2012). Pengukuran, penilaian, dan evaluasi pendidikan. Nuha Medika.
Moore, C. T. (2016). gtheory: Apply generalizability theory with R [Computer software]. http://evaluationdashboard.com
Mullis, I. V. S., & Martin, M. O. (2017). TIMSS 2019 assessment framework. International Association for the Evaluation of Educational Achievement.
OECD. (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematics, financial literacy and collaborative problem solving (Rev. ed.). OECD Publishing.
Ojose, B. (2011). Mathematics literacy: Are we able to put the mathematics we learn into everyday use? Journal of Mathematics Education, 4(1), 89–100.
Southam-Gerow, M. A., Bonifay, W., McLeod, B. D., Cox, J. R., Violante, S., Kendall, P. C., & Weisz, J. R. (2020). Generalizability and decision studies of a treatment adherence instrument. Assessment, 27(2), 321–333. https://doi.org/10.1177/1073191118765365
Tan, H. (2023). The generalizability of explanations. Proceedings of the International Joint Conference on Neural Networks, 2023-June. https://doi.org/10.1109/IJCNN54540.2023.10191972
Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018). Applications of generalizability theory and their relations to classical test theory and structural equation modeling. Practical Assessment, Research & Evaluation, 23(1), 1–26.
Whitney-Smith, R., Hurrell, D., & Day, L. (2022). The role of mathematics education in developing students’ 21st century skills, competencies and STEM capabilities. In Proceedings of the annual conference of the Mathematics Education Research Group of Australasia (pp. 554–561). Mathematics Education Research Group of Australasia. https://merga.net.au/Public/Public/Publications/Conference_Proceedings.aspx
Wijaya, A., & Dewayani, S. (2021). Framework asesmen kompetensi minimum (AKM). Badan Penelitian dan Pengembangan dan Perbukuan, Kementerian Pendidikan dan Kebudayaan.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Lusi Eka Afri, Nurrahmawati, Arcat
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
