“Complexity” and “Difficulty” of Questions in Research Consultations: Predictability and Identification of Features (A systematic review)

Document Type : Original Article

Authors

Mariam Keshvari ¹

Monireh Shakeri Far ²

¹ Assistant Professor, Department of Knowledge and Information Science, Faculty of Education and Psychology, University of Isfahan, Isfahan, Iran

² PhD Candidate in Knowledge and Information Science, AlZahra University, Tehran, Iran

10.22067/infosci.2026.95492.1254

Abstract

Purpose: Research consultation has emerged as a modern and indispensable responsibility of librarians, particularly those in academic settings. The importance of this role has been underscored to the extent that the sustainability of academic libraries is frequently contingent upon the provision of research consultation services. Central to the efficacy of research consultation is the concept of the “question.” According to the extant literature, only questions characterized as both “complex” and “difficult” necessitate research consultation; such questions are often referred to in various sources as “ill-structured problems.” Nevertheless, considerable ambiguities persist within this area of inquiry. Notably, the information science literature—especially within the domain of reference services—has not clearly delineated between the constructs of “difficulty” and “complexity.” Furthermore, no explicit or consensually accepted criteria or attributes have been established to identify such questions. In light of this gap, the present systematic review seeks to investigate the feasibility of librarians predicting question difficulty and complexity prior to the initiation of a consultation, as well as to identify the defining features of such questions as they emerge both during and after research consultations.
Methods: This study employed a qualitative approach alongside a systematic review methodology. The research corpus comprised 26 relevant sources, which were identified and analyzed in accordance with the PRISMA guidelines.
Findings: Broadly speaking, three subfields within information science have addressed the difficulty and complexity of reference questions: (a) reference and information services, (b) information-seeking behavior, and (c) information retrieval. Among these, reference and information services alongside information-seeking behavior have concentrated on complex and challenging questions with the objective of facilitating interviews and research consultations. In contrast, the field of information retrieval adopts a somewhat distinct focus and serves a broader, more heterogeneous target population—frequently referred to collectively as “searchers.” Consequently, literature from the latter domain was excluded from the present study.
This review aimed to address two primary research questions. The first examined whether librarians are able to predict the difficulty and complexity of questions prior to conducting research consultations. Findings indicate that no strong correlation exists between librarians’ predicted assessments of question difficulty and complexity and the actual levels observed. In essence, anticipating the difficulty of questions for research consultation appears to be a challenging endeavor in itself, one that necessitates librarians’ active engagement with the questions and the implementation of pre-consultation interviews. The level of librarians’ experience emerged as a key factor emphasized across multiple studies, suggesting that, when interpreted with caution, librarian experience may help determine the difficulty and complexity of questions posed for research consultation purposes. With respect to the second research question—pertaining to the characteristics of difficult and complex questions—a review of the literature revealed that several classification models for reference questions, including those proposed by Katz (2002), Warner (2001), and Garlick and Berard (2010), have addressed this issue. These models position reference questions along a continuum ranging from simple and easy to complex and difficult. However, an analysis of these models indicates that the characteristics of difficult and complex questions are primarily examined either during or after the consultation process, with limited attention given to pre-consultation features that could enable prediction. Among the features identified as critical for challenging questions within these classification models, the time required to respond, as well as the diversity and number of sources consulted, are particularly emphasized. Furthermore, within the body of related literature that addresses question complexity and difficulty, the relevant characteristics are generally classifiable into two primary categories:

Pre-Consultation Features: these enclude the timing of the question’s receipt within the library context (i.e., inquiries submitted outside standard operating hours and those received during the mid-semester period); The intrinsic characteristics of the question itself (including its length, the number of conceptual elements it contains, the number of constituent components, and its degree of ill-structuredness); The librarian’s profile (including educational qualifications, professional experience and domain-specific expertise, and level of interest in the subject matter); The library’s institutional capacity (including the availability of informational resources and the presence or absence of a dedicated research consultation service); The questioner’s background (specifically their educational history and level of academic attainment)
Features During and After Consultation: The key determinants encompass the duration of the consultation process—frequently identified as the most critical factor—alongside the number and diversity of sources utilized, the librarian’s domain knowledge, practical experience, and subject expertise, the volume of actions undertaken, the level of interactivity and user engagement achieved, and the overall effectiveness in delivering accurate answers and meaningful guidance.

Discussion and conclusion: Predicting the difficulty and complexity of questions posed by librarians remains a challenging endeavor and requires considerable caution. Nevertheless, several pre-consultation features may serve as useful facilitators and guides for professional judgment. The experience and expertise of librarians are decisive factors both prior to and during consultations. While certain features have been identified as potential indicators of question complexity and difficulty, integrating an initial user interview with these features can yield a more comprehensive and accurate assessment of whether a research consultation is warranted.
Originality: Identifying challenging and complex questions is a crucial competency for librarians, offering substantial potential advantages, including time efficiency, the enhancement of the profession’s scholarly reputation, and improved user satisfaction. This study is one of the first to examine this issue systematically, and its findings provide meaningful insights to support librarians in the delivery of research consultation services.

Keywords

Research consultation

difficult questions

complex questions

reference questions

Subjects

• Information Management

---

References

Anderson, R., Fisher, K., & Walker, J. (2021). Library consultations and a global pandemic: An analysis of consultation difficulty during COVID-19 across multiple factors. The Journal of Academic Librarianship, 47(1), 102273. https://doi.org/10.1016/j.acalib.2020.102273 
Aromataris, E., Lockwood, C., Porritt, K., Pilla, B., & , & Jordan, Z. (2024). JBI Manual for Evidence Synthesis. JBI https://doi.org/10.46658/JBIMES-24-01 
Bagheri, M., Shabani, A., & Abedi, M. R. (2014). Effect of information counseling on library anxiety and library behaviors among users of public libraries. Research on Information Science and Public Libraries, 19(4), 569–581. 
Beckmann, J. F., Birney, D. P., & Goode, N. (2017). Beyond psychometrics: the difference between difficult problem solving and complex problem solving. Frontiers in psychology, 8, 1739. 
Belanger, J., Collins, K., Deutschler, A., Greer, R., Huling, N., Maxwell, C., Papadopoulou, E., Ray, L., & Roemer, R. C. (2016). University of Washington Libraries chat reference transcript assessment  https://lib.uw.edu/wp-content/uploads/assessment_projects_chat-reference-assessment-project-2014-2016-final-report.pdf. 
Bradley, D., Oehrli, A., Rieh, S. Y., Hanley, E., & , & Matzke, B. S. (2020). Advancing the reference narrative: Assessing student learning in research consultations. Evidence based library and information practice, 15(1), 4–19. https://doi.org/10.18438/eblip29634 
Bravender, P., Lyon, C., & Molaro, A. (2011). Should chat reference be staffed by librarians? An assessment of chat reference at an academic library using LibStats. Internet Reference Services Quarterly, 16(3), 111–127. https://doi.org/10.1080/10875301.2011.595255 
Brown, D. M. (1985). Telephone reference questions: A characterization by subject, answer format, and level of complexity. RQ, 1, 290–303. 
Bunge, C. A. (1990). Factors Related to Output Measures for Reference Services in Public Libraries: Data from Thirty-Six Libraries. Public Libraries, 29(1), 42–47. 
Butler, K., & Byrd, J. (2016). Research consultation assessment: Perceptions of students and librarians. The Journal of Academic Librarianship, 42(1), 83–86. https://doi.org/10.1016/j.acalib.2015.10.011 
Childers, T., Lopata, C., & Stafford, B. (1991). Measuring the difficulty of reference questions. RQ, 31(2), 237–243. 
Dempsey, P. R. (2016). “Are you a computer?” Opening exchanges in virtual reference shape the potential for teaching. College & Research Libraries, 77(4), 455–468. 
Dennison, R. F. (1999). Usage-based staffing of the reference desk: A statistical approach. Reference & User Services Quarterly, 158–165. 
Estrada, S., Floren, M., Harding, J., & Wroblewski, M. (2021). What is your research question? A mixed methods evaluation of an academic statistical consulting center. Evaluation and Program Planning, 89, 102001. https://doi.org/10.1016/j.evalprogplan.2021.102001 
Faix, A., MacDonald, A., & Taxakis, B. (2014). Research consultation effectiveness for freshman and senior undergraduate students. Reference Services Review, 42(1), 4–15. https://doi.org/10.1108/RSR-05-2013-0024 
Fields, A. M. (2006). Ill‐structured problems and the reference consultation: The librarian's role in developing student expertise. Reference Services Review, 34(3), 405–420. https://doi.org/10.1108/00907320610701554 
Firsová, N., Hinke, J., & Smutka, L. (2024). Complexity of reference consultations for undergraduate and graduate students in an academic library. Knowledge and Performance Management, 8(2), 76. 
Frank, D. G., Raschke, G. K., Wood, J., & Yang, J. Z. (2001). Information consulting: The key to success in academic libraries. The Journal of Academic Librarianship, 27(2), 90–96. 
Gerlich, B. K., & Berard, G. L. (2007). Introducing the READ scale: Qualitative statistics for academic reference services. Georgia Library Quarterly, 43(4), 4. 
Gerlich, B. K., & Berard, L. G. (2010). Testing The Viability Of The READ Scale (Reference Effort Assessment Data)®: Qualitative Statistics For Academic Reference Services. 
Harrison, R., Jones, B., Gardner, P., & Lawton, R. (2021). Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed-or multi-method studies. BMC health services research, 21(1), 144. https://doi.org/10.1186/s12913-021-06122-y 
Hauff, C., Kelly, D., & Azzopardi, L. (2010, October). A comparison of user and system query performance predictions. Proceedings of the 19th ACM international conference on Information and knowledge management, 979–988. https://doi.org/10.1145/1871437.1871562 
Hirsh, S. (2018). Information services today: An introduction (2ed ed.). Rowman & Littlefield. 
Jastram, I., & Zawistoski, A. G. (2008). Personalizing the library via research consultations. The Desk and beyond: Next generation reference services, 14–24. 
Johnston, L., & Kennedy, M. (2024). An Assessment of Research Consultation Practices. https://www.libraryassessment.org/wp-content/uploads/2025/01/Johnston-An-Assessment-Project-of-Research-Consultations.pdf. 
Kani, J. (2017). Evernote in the research consultation: a feasibility study. Reference Services Review, 45(1), 67–78. https://doi.org/10.1108/RSR-05-2016-0034 
Katz, W. A. (2002). Introduction to reference work. McGraw-Hill. 
Kayongo, J., & Van Jacob, E. (2011). Burning the midnight oil: Librarians, students, and late-night chat reference at the University of Notre Dame. Internet Reference Services Quarterly, 16(3), 99–109. https://doi.org/10.1080/10875301.2011.597632 
Kemp, J. H., Ellis, C. L., & Maloney, K. (2015). Standing by to help: Transforming online reference with a proactive chat system. The Journal of Academic Librarianship, 41(6), 764–770. https://doi.org/10.1016/j.acalib.2015.08.018 
Keshvari, M., & Amiri, S. (2025). Virtual Reference Services: The Evolution of Questions; Diversity of Answerers (A Scoping Review). Library and Information Sciences, 28(1), 37–61. 
Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. Journal of the American society for information science, 42(5), 361–371. https://doi.org/https://www.doi.org/10.1002/(SICI)1097-4571(199106)42:5 
Kuhlthau, C. C. (1993). A principle of uncertainty for information seeking. Journal of documentation, 49(4), 339–355. https://doi.org/https://doi.org/10.1108/eb026918 
Kuhlthau, C. C. (1994). Students and the information search process: Zones of intervention for librarians. Advances in librarianship, 18, 57–72. 
Kuhlthau, C. C. (1996). The Concept of a Zone of Intervention for Identifying the Role of Intermediaries in the Information Search Process. Proceedings of the ASIS Annual Meeting, 33, 91–94. 
Kuhlthau, C. C. (1999). The role of experience in the information search process of an early career information worker: Perceptions of uncertainty, complexity, construction, and sources. Journal of the American society for information science, 50(5), 399–412. 
Kuhlthau, C. C. (2007). Reflections on the development of the model of the information search process (ISP): Excerpts from the Lazerow Lecture, University of Kentucky, April 2, 2007. Bulletin of the american society for information science and technology, 33(5), 32–37. 
Lai, K. (2023). How well does ChatGPT handle reference inquiries? An analysis based on question types and question complexities. College & Research Libraries, 84(6), 974. 
Maloney, K., & Kemp, J. H. (2015). Changes in Reference Question Complexity Following the Implementation of a Proactive Chat System: Implications for Practice. 2015(7), 16. https://doi.org/10.5860/crl.76.7.959 
Michell, G., & Dewdney, P. (1998). Mental Models Theory: Applications for Library and Information Science. Journal of Education for Library and Information Science, 39(4), 275–281. https://doi.org/10.2307/40324303 
Mizzaro, S., & Mothe, J. (2016). Why do you Think this Query is Difficult? A User Study on Human Query Prediction. 39th International ACM SIGIR conference on research and development in Information Retrieval, 1073-1076. https://doi.org/10.1145/2911451.2914696, 
Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & , & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology, 18(1), 143. https://doi.org/https://www.doi.org/10.1186/s12874-018-0611-x 
Oberg, K. J. (2011). An Investigation of Reference Question Difficulty over Time University of Norht Carolina https://doi.org/10.17615/j4pv-hw82]. 
Peck, T. P. (1975). Counseling skills applied to reference services. RQ, 233–235. 
Pelánek, R., Effenberger, T., & & Čechák, J. (2020). Complexity and difficulty of items in learning systems. International Journal of Artificial Intelligence in Education, 32(1), 232–196. https://doi.org/https://doi.org/10.1007/s40593-021-00252-4 
Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & , & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371–385. https://doi.org/https://doi.org/10.1002/jrsm.1123 
Pomerantz, J., & McClure, C. R. (2004). Evaluation of a statewide collaborative chat‐based reference service: Approaches and directions. Proceedings of the American Society for Information Science and Technology, 41(1), 109–102. https://doi.org/10.1002/meet.1450410112 
Robinson, B. M. (1989). Reference services: a model of question handling. RQ, 61(1), 29–48, https://www.jstor.org/stable/25828432. 
Rogers, E., & Carrier, H. S. (2017). A qualitative investigation of patrons’ experiences with academic library research consultations. Reference Services Review, 45(1), 18–37. https://doi.org/10.1108/rsr-04-2016-0029 
Ryan, S. M. (2008). Reference transactions analysis: The cost-effectiveness of staffing a traditional academic reference desk. The Journal of Academic Librarianship, 34(5), 389–399. https://doi.org/https://doi.org/10.1016/j.acalib.2008.06.002 
Saint Clair, J. W., Aluri, R., & Pastine, M. (1977). Staffing the reference desk: Professionals or Nonprofessionals? Journal of Academic Librarianship, 3(3), 149–153. 
Saracevic, T., & Kantor, P. (1988). A study of information seeking and retrieving. III. Searchers, searches, and overlap. Journal of the American society for information science, 39(3), 197–216. 
Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes Harvard university press. 
Warner, A., & Hurley, D. A. (2021). Does the READ scale work for chat? A review of the literature. Evidence based library and information practice, 16(3), 122–139. https://doi.org/10.18438/eblip29947 
Warner, D. G. (2001). A new classification for reference statistics. Reference & User Services Quarterly, 51–55. 
White, M. D., & Iivonen, M. (2002). Assessing level of difficulty in web search questions. The Library Quarterly, 72(2), 205–233. 
Wood, R. E. (1986). Task complexity: Definition of the construct. Organizational behavior and human decision processes, 37(1), 60–82. https://doi.org/10.1016/0749-5978(86)90044-0 
Xun, G., & Land, S. M. (2004). A conceptual framework for scaffolding III-structured problem-solving processes using question prompts and peer interactions. Educational technology research and development, 52(2), 5–22. https://doi.org/10.1007/BF02504836