There are two questions you can ask about a model's safeguards, and for an open-weight release only one of them matters much.
The first is whether the safeguards hold under adversarial pressure — whether a determined user, prompting in bad faith, can get the model to produce something the developer intended to refuse. This is the question most evaluation protocols are built around, and it is the right question for a model served through an API, because the provider controls the deployment and can respond to what the evaluation finds.
The second is what the model can do once the safeguards are removed entirely. For open weights this is the operative question, because refusal behaviour trained into a model is not a property of the weights in the way that capability is. Fine-tuning it away is well documented, cheap relative to the cost of training the model in the first place, and requires no privileged access. Anyone who has the weights can do it.
Why the distinction changes what a pre-release evaluation is for
If safeguards can be removed, then an evaluation finding that they can be circumvented is not really a finding about safety. It is a finding about how much effort circumvention takes — which is worth knowing, since friction has real effects on who actually does a thing, but it is not the same as prevention.
The evaluation that matters before an open-weight release is therefore an evaluation of the underlying capability, conducted on a version with the safeguards deliberately stripped. That is uncomfortable work: it requires the evaluating body to build the very artefact everyone is worried about, in order to measure it. Institutes that do this properly are doing something more difficult than running a red-team exercise against a chat interface.
Cyber-offence evaluation and the publication problem
Cyber capability evaluations create a reporting difficulty that most other domains do not. A finding that a model materially assists with a class of offensive operation is only fully informative if you describe the class, the assistance and the conditions — and that description is itself a contribution to the problem.
The convention that has emerged is to publish the shape of the finding and the assessed severity while withholding the specifics, with fuller detail shared between institutes and relevant national bodies. It is a reasonable compromise and an unsatisfying one, because it asks the public to accept a conclusion whose evidence it cannot inspect. Publications covering this work should be clear about which parts of a finding they have seen and which parts they are relaying on trust.
Joint evaluation as a governance mechanism
Evaluations conducted jointly by institutes in different jurisdictions have a property single-institute evaluations lack: the finding is harder to walk back. A conclusion published by two bodies answering to different governments, with different commercial pressures around them, is a more robust artefact than the same conclusion published by one.