Evaluating Transparency in Large Language Models: A Four-Dimensional Framework and Comparative Analysis

The increasing availability of open and open-weight large language models (LLMs) has been framed as a means of improving accessibility, reproducibility, and accountability in AI development. However, the extent to which model openness provides meaningful transparency about training data, technical development, evaluation practices, and ethical risks remains unclear. In this work, we develop a four-dimensional framework for systematically evaluating transparency in LLM development across Dataset Transparency, Model Architecture, Evaluation and Benchmarks, and Ethical Accountability. Each dimension is evaluated using a structured 0–3 rubric based on the availability, specificity, and acknowledgment of relevant information in publicly available technical reports and documentation. We apply the framework to four LLMs representing different approaches to openness and documentation: Meta Llama 2, OLMo, Mistral, and MAP-Neo. Our comparative analysis reveals substantial variation in transparency across models and dimensions. OLMo and MAP-Neo demonstrate the strongest overall transparency, particularly through detailed dataset documentation, architectural specifications, benchmark reporting, and discussion of ethical considerations. In contrast, Llama 2 and Mistral provide more limited information about training data and ethical accountability, despite providing substantial technical and benchmark information. Across all four models, architectural transparency is generally stronger than dataset transparency, suggesting that information about model design is more readily disclosed than information necessary to examine training-data provenance, representation, and potential data-related risks. We further find that benchmark disclosure is relatively consistent across models, but the benchmarks primarily reflect established evaluation practices, with substantial emphasis on English-language and Western-centered tasks. While some models acknowledge limitations of these benchmarks, systematic evaluation of sociocultural representation and bias remains limited. Ethical accountability also varies considerably in both scope and framing, ranging from technical safety measures to broader commitments to transparency, auditability, and democratization. These findings demonstrate that LLM openness is multidimensional: increased disclosure in one area does not necessarily imply comprehensive transparency or ethical accountability. Our framework provides a structured approach for comparing transparency practices across LLMs and highlights areas where greater disclosure and critical evaluation are needed to support meaningful auditing and responsible AI development.