NDSS2025併催のUSEC2025に研究論文が採択されました

研究成果の発表

2025年2月

Evaluating LLMs Towards Automated Assessment of Privacy Policy Understandability

学会・研究会：Symposium on Usable Security and Privacy (USEC) 2025
著者：Keika Mori (DTCY/Waseda University), Daiki Ito, Takumi Fukunaga, Takuya Watanabe, Yuta Takata, Masaki Kamizono (DTCY), and Tatsuya Mori (Waseda University/NICT/RIKEN AIP)
https://www.ndss-symposium.org/ndss2025/co-located-events/usec/（外部サイト）

論文概要

Companies publish privacy policies to improve transparency regarding the handling of personal information. A discrepancy between the description of the privacy policy and the user's understanding can lead to a risk of a decrease in trust. Therefore, in creating a privacy policy, the user's understanding of the privacy policy should be evaluated. However, the periodic evaluation of privacy policies through user studies takes time and incurs financial costs. In this study, we investigated the understandability of privacy policies by large language models (LLMs) and the gaps between their understanding and that of users, as a first step towards replacing user studies with evaluation using LLMs. Obfuscated privacy policies were prepared along with questions to measure the comprehension of LLMs and users. In comparing the comprehension levels of LLMs and users, the average correct answer rates were 85.2% and 63.0%, respectively. The questions that LLMs answered incorrectly were also answered incorrectly by users, indicating that LLMs can detect descriptions that users tend to misunderstand. By contrast, LLMs understood the technical terms used in privacy policies, whereas users did not. The identified gaps in comprehension between LLMs and users, provide insights into the potential of automating privacy policy evaluations using LLMs.