218 S. Ito et al.: Radioprotection 2025, 60( 3), 211 – 220
the difficulty of Japanese with the reading level prompt is similar to the findings of previous studies( Cheong et al., 2024; Dihan et al., 2024; Shen et al., 2024). For example, Dihan et al. used ChatGPT-3.5, ChatGPT-4.0, and Google Bard to assess the comprehension and text difficulty of pediatric glaucoma patient education materials( Dihan et al., 2024). Only ChatGPT-4.0 improved comprehension when prompted to respond at a 6th grade level, but the sentence difficulty decreased for all AI chatbots. The study by Shen et al. used ChatGPT-3.5 and Google Search to generate answers to patients’ questions about medical guidelines( Shen et al., 2024). When prompted to respond at a 6th grade level, sentence difficulty decreased more than without prompting. A study by Cheong et al. evaluated patient education materials for obstructive sleep apnea using ChatGPT-3.5 and Google Bard. The results reported that Google Bard was able to produce 5th grade level text( Cheong et al., 2024). These results suggest that, to varying degrees depending on the type of AI chatbot, AI chatbots may be able to reduce the difficulty of sentences by prompting them to produce 5th or 6th grade level documents, but may have difficulty improving comprehension. The sentences produced by the chatbot are patterned compositions and layout designs unless specifically instructed by prompts. Therefore, the prompt“ teach me at a 6th grade level” may have decreased the difficulty of the sentences, but may not have improved comprehension unless the composition and layout were changed.
This study had various strengths and limitations. One strength of this study is that it is the first study to numerically evaluate the understandability, actionability, and difficulty of documents produced by AI chatbots related to nuclear power and radiation disaster prevention. It is also the first study to comprehensively extract the most frequently searched radiation-related keywords in Fukushima Prefecture during the week of the Fukushima Daiichi Nuclear Power Station accident. Next, we discuss the limitations of this study. First, the prompt,“ Please teach me at a 6th grade level,” was simple but lacked sufficient content to confirm the usefulness of the AI chatbot in obtaining radiation disaster prevention information for the general public. This prompt is simple and did improve readability. However, AI chatbots do not actively display layouts and designs that promote comprehension, charts and summaries, suggestions for action, or explanations of technical terms without prompting. We believe that only those who can enter sophisticated search queries will be able to obtain useful information through the use of AI chatbots and search engines. This could lead to a widening of the human knowledge gap. We believe that prompts should be considered that allow many people to get the maximum benefit from AI chatbots. Second, it is difficult to compare the results for AI chatbots with those of previous studies: new versions of AI chatbots are released every year, and their performance varies greatly depending on the version and type. In recent years, guidelines for papers using AI chatbots and other types of chatbots have been published, but before that, many papers did not specify the time of the study or the version of the AI chatbot. Therefore, it is difficult to know what results regarding the quality evaluation of texts created with an AI chatbot are caused by the prompt, the type of material evaluated, the type of AI chatbot, and the chatbot version. We believe it is important to develop and adhere to guidelines regarding the use of AI chatbots.
Third, there is the possibility of generalization to other languages. Some time after the Fukushima Daiichi Nuclear Power Station accident, many radiation-related materials for the general public were created and published online in Japan. However, immediately after the nuclear accident, there were almost no easy-to-understand radiation-related materials for the general public, and the Web contained a lot of misinformation, making it difficult to judge the appropriateness of the information. Therefore, it may be difficult to use AI chatbots to obtain appropriate radiation-related information in countries and languages where radiation-related information for the general public is not published online. Prompts for translations and summaries by AI chatbots may be important when radiation-related information for the general public is needed. Finally, AI chatbots present challenges in terms of information reliability, as they occasionally produce incorrect information. In this study, 48( 5.9 %) of the sentences generated by the AI chatbots contained outright errors. There are prior studies that analyzed academic literature and specialized books using AI tools to build large language models tailored to the medical field, enabling reliable information extraction and generation( Wu et al., 2024). These studies included examples such as the PMC-LLaMA model, which was developed using medical papers and textbooks. While concerns remain about the reliability of text generated by AI chatbots, these studies provide a foundation for generating trustworthy content. Future research should focus on mitigating the risks of misinformation and further improving the accuracy and reliability of generated content.
5 Conclusions
The radiation-related sentences generated by AI chatbots were easier to understand, and the Japanese sentences generated by chatbots were easier to read than the sentences on webpages. We also found that the prompt,“ Please teach me at a 6th grade level,” could further reduce the reading difficulty of the Japanese documents. These results suggest that the AI chatbot is an effective tool for promoting the public’ s understanding of radiation disaster prevention. However, further research is needed to provide more understandable, practical, and readable materials, including the creation of effective prompts.
Funding
This work was supported by the Japan Society for the Promotion of Science under KAKENHI Grant Number JP 24K06389 and the Program of the Network-type Joint Usage / Research Center for Radiation Disaster Medical Science.
Conflicts of interest
The authors declare that they have no conflict of interest.