| 陈阳,孟旭阳,陈熙.内嵌RAG的科技文献发现系统测评方法研究[J].数字图书馆论坛,2026,22(6):59~69 |
| 内嵌RAG的科技文献发现系统测评方法研究 |
| Evaluation Methods for Scientific and Technical Literature Discovery System with Embedded RAG |
| 投稿时间：2025-05-20 |
| DOI：10.3772/j.issn.1673–2286.2026.06.006 |
| 中文关键词: 科技文献；检索增强生成；指标体系；自动化测评；NSTL |
| 英文关键词: Scientific and Technical Literature; Retrieval-Augmented Generation; Indicator System; Automated Evaluation; National Science and Technology Library |
| 基金项目:本研究得到国家社会科学基金青年项目“大模型赋能下的学术文献智慧服务新模式与应用研究”（编号：24CTQ008）资助。 |
| 作者 | 单位 | | 陈阳 | 中国科学技术信息研究所 | | 孟旭阳 | 中国科学技术信息研究所 | | 陈熙 | 中国科学技术信息研究所 |
|
| 摘要点击次数: 9 |
| 全文下载次数: 8 |
| 中文摘要: |
| 检索增强生成（Retrieval-Augmented Generation，RAG）技术已广泛应用于科技文献发现系统。但当前发现系统测评方法仍存在人工成本高、难以适配RAG架构特性等问题。因此，本研究提出一种内嵌RAG的科技文献发现系统测评方法，从检索质量、生成质量和溯源质量三个维度构建测评指标体系。首先利用自动化测评获得各指标的量化结果，再采用融合大模型智能评分与人工核对的混合评估方法进行评价，最后通过斯皮尔曼、肯德尔相关性检验验证指标量化结果与混合评估结果的一致性。本研究以国家科技图书文献中心（National Science and Technology Library，NSTL）发现系统为实证对象进行验证。研究结果表明，本研究提出的测评方法能有效识别系统在科技文献检索、生成和溯源方面的差异，且自动化测评结果与混合评估结果呈显著正相关，验证了本研究所构建指标体系的可靠性与有效性，为科技文献发现系统测评与迭代升级提供了实践支撑。 |
| 英文摘要: |
| Retrieval-Augmented Generation (RAG) technology has been widely applied in scientific and technical literature discovery systems. However, current evaluation methods for these systems still face issues such as high labor costs and difficulties in adapting to the RAG architecture. Therefore, this research proposes an evaluation approach for scientific and technical literature discovery systems with embedded RAG, constructing a metric system based on three dimensions: retrieval quality, generation quality, and traceability quality. First, quantitative results for each indicator are obtained through automated assessment. Next, a hybrid evaluation method combining intelligent scoring by a large language model with manual verification is employed. Finally, the consistency between the quantitative results of the indicators and the hybrid evaluation results is verified using Spearman’s and Kendall’s correlation tests. This
research uses the discovery system of National Science and Technology library (NSTL) as its empirical subject. The results indicate that this evaluation method can effectively identify differences in the system’s performance regarding scientific and technical literature retrieval, generation, and attribution. Furthermore, the indicator calculation results show a significant positive correlation with the hybrid evaluation results, validating the reliability and effectiveness of the indicator system developed in this study and providing practical support for the evaluation and iterative improvement of scientific and technical literature discovery systems. |
|
查看全文
查看/发表评论 下载PDF阅读器 |
| 关闭 |
