Hugging Face's logo

ibm-granite
/
granite-guardian-3.3-8b

Text Generation
Transformers
Safetensors
English
granite
conversational
Model card Files
xet
Community
4
ink-pad commited on
Commit
7660e9b
·
verified ·
1 Parent(s): d8f3d57

Updated table style

Browse files
Files changed (1) hide show
  1. README.md +254 -26
README.md CHANGED
@@ -21,7 +21,7 @@ It outperforms other open-source models in the same space on standard benchmarks
21
  - **Cookbook:** [Granite Guardian Recipes](https://github.com/ibm-granite/granite-guardian/tree/main/cookbooks/granite-guardian-3.3)
22
  - **Website**: [Granite Guardian Docs](https://www.ibm.com/granite/docs/models/guardian/)
23
  - **Paper:** [Granite Guardian](https://arxiv.org/abs/2412.07724)
24
- - **Release Date**: July 30, 2025
25
  - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
26
 
27
  ## Usage
@@ -141,7 +141,7 @@ score, _ = parse_response(response)
141
  print(f"# score: {score}\n") # score: yes
142
  ```
143
 
144
- #### Example 3: Detect lack of groundednedss of model's response in RAG settings
145
 
146
  Here you see how how to use the Granite Guardian in thinking mode by passing ```think=True``` in the ```apply_chat_template``` method.
147
 
@@ -242,44 +242,272 @@ The model is also equipped to detect hallucinations in agentic workflows, such a
242
  Following the general harm definition, Granite-Guardian-3.3-8B is evaluated across the standard benchmarks of [Aeigis AI Content Safety Dataset](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-1.0), [ToxicChat](https://huggingface.co/datasets/lmsys/toxic-chat), [HarmBench](https://github.com/centerforaisafety/HarmBench/tree/main), [SimpleSafetyTests](https://huggingface.co/datasets/Bertievidgen/SimpleSafetyTests), [BeaverTails](https://huggingface.co/datasets/PKU-Alignment/BeaverTails), [OpenAI Moderation data](https://github.com/openai/moderation-api-release/tree/main), [SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) and [xstest-response](https://huggingface.co/datasets/allenai/xstest-response).
243
  The following table presents the F1 scores for various harm benchmarks, along with the aggregate F1 score.
244
 
245
- | Method | AggregateF1 | AegisSafetyTest | BeaverTails | HarmBench_Prompt | OAI_hf | SafeRLHF_test | simpleSafetyTest | toxic_chat | xstest_RH | xstest_RR | xstest_RR(h) |
246
- |---|---|---|---|---|---|---|---|---|---|---|---|
247
- | granite-guardian-3.1-8b | 0.79 | 0.88 | 0.81 | 0.80 | 0.78 | 0.81 | 0.99 | 0.73 | 0.87 | 0.45 | 0.83 |
248
- | granite-guardian-3.2-5b | 0.78 | 0.88 | 0.81 | 0.80 | 0.73 | 0.80 | 0.99 | 0.73 | 0.90 | 0.43 | 0.82 |
249
- | granite-guardian-3.3-8b (no_think) | 0.81 | 0.87 | 0.84 | 0.80 | 0.77 | 0.80 | 0.99 | 0.76 | 0.90 | 0.49 | 0.87 |
250
- | granite-guardian-3.3-8b (think) | 0.79 | 0.86 | 0.82 | 0.80 | 0.78 | 0.78 | 0.99 | 0.69 | 0.86 | 0.50 | 0.86 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
251
 
252
 
253
  ### RAG Hallucination Benchmarks
254
  For detecting hallucinations in RAG settings, the model is evaluated on [LM-AggreFact](https://llm-aggrefact.github.io/) benchmarks. We report balanced accuracy scores on LM AggreFact below:
255
 
256
- | Method | AVG | AggreFact-CNN | AggreFact-XSum | ClaimVerify | ExpertQA | FactCheck-GPT | Lfqa | RAGTruth | Reveal | TofuEval-MediaS | TofuEval-MeetB | Wice |
257
- |---|---|---|---|---|---|---|---|---|---|---|---|---|
258
- | granite-guardian-3.1-8b | 0.709 | 0.532 | 0.570 | 0.724 | 0.597 | 0.759 | 0.855 | 0.768 | 0.877 | 0.725 | 0.761 | 0.635 |
259
- | granite-guardian-3.2-5b | 0.665 | 0.508 | 0.530 | 0.650 | 0.596 | 0.743 | 0.808 | 0.630 | 0.872 | 0.691 | 0.685 | 0.604 |
260
- | granite-guardian-3.3-8b (no_think) | 0.761 | 0.669 | 0.738 | 0.767 | 0.596 | 0.729 | 0.878 | 0.831 | 0.894 | 0.736 | 0.815 | 0.720 |
261
- | granite-guardian-3.3-8b (think) | 0.765 | 0.661 | 0.749 | 0.759 | 0.597 | 0.766 | 0.870 | 0.821 | 0.896 | 0.739 | 0.789 | 0.773 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
262
 
263
 
264
  We also report performance on TRUE benchmark (balanced accuracy) that measures faithfulness of LLM responses to the context.
265
 
266
- | Method | AVG | begin | dialfact | frank | mnbm | paws | q2 | qags_cnndm | qags_xsum | summeval |
267
- |---|---|---|---|---|---|---|---|---|---|---|
268
- | granite-guardian-3.1-8b | 0.725 | 0.714 | 0.630 | 0.835 | 0.648 | 0.780 | 0.710 | 0.756 | 0.717 | 0.733 |
269
- | granite-guardian-3.2-5b | 0.710 | 0.740 | 0.694 | 0.791 | 0.635 | 0.749 | 0.727 | 0.723 | 0.660 | 0.672 |
270
- | granite-guardian-3.3-8b (no_think) | 0.777 | 0.733 | 0.684 | 0.886 | 0.660 | 0.825 | 0.801 | 0.814 | 0.796 | 0.796 |
271
- | granite-guardian-3.3-8b (think) | 0.773 | 0.732 | 0.722 | 0.864 | 0.680 | 0.813 | 0.799 | 0.792 | 0.798 | 0.761 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
272
 
273
 
274
  ### Function Calling Hallucination Benchmarks
275
  The model performance is evaluated on the [FC Reward Bench evaluation](https://huggingface.co/datasets/ibm-research/fc-reward-bench) dataset. We use balanced accuracy as the metric to compare the various models.
276
 
277
- | Method | fc_reward_bench |
278
- |---|---|
279
- | granite-guardian-3.1-8b | 0.64 |
280
- | granite-guardian-3.2-5b | 0.61 |
281
- | granite-guardian-3.3-8b (no_think) | 0.74 |
282
- | granite-guardian-3.3-8b (think) | 0.71 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
283
 
284
 
285
  ## Training Data
 
21
  - **Cookbook:** [Granite Guardian Recipes](https://github.com/ibm-granite/granite-guardian/tree/main/cookbooks/granite-guardian-3.3)
22
  - **Website**: [Granite Guardian Docs](https://www.ibm.com/granite/docs/models/guardian/)
23
  - **Paper:** [Granite Guardian](https://arxiv.org/abs/2412.07724)
24
+ - **Release Date**: August 1, 2025
25
  - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
26
 
27
  ## Usage
 
141
  print(f"# score: {score}\n") # score: yes
142
  ```
143
 
144
+ #### Example 3: Detect lack of groundedness of model's response in RAG settings
145
 
146
  Here you see how how to use the Granite Guardian in thinking mode by passing ```think=True``` in the ```apply_chat_template``` method.
147
 
 
242
  Following the general harm definition, Granite-Guardian-3.3-8B is evaluated across the standard benchmarks of [Aeigis AI Content Safety Dataset](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-1.0), [ToxicChat](https://huggingface.co/datasets/lmsys/toxic-chat), [HarmBench](https://github.com/centerforaisafety/HarmBench/tree/main), [SimpleSafetyTests](https://huggingface.co/datasets/Bertievidgen/SimpleSafetyTests), [BeaverTails](https://huggingface.co/datasets/PKU-Alignment/BeaverTails), [OpenAI Moderation data](https://github.com/openai/moderation-api-release/tree/main), [SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) and [xstest-response](https://huggingface.co/datasets/allenai/xstest-response).
243
  The following table presents the F1 scores for various harm benchmarks, along with the aggregate F1 score.
244
 
245
+ <table>
246
+ <caption style="text-align:center"><b>Harm</b></caption>
247
+ <thead>
248
+ <tr>
249
+ <th style="text-align:left; background-color: #001d6c; color: white;">Model</th>
250
+ <th style="text-align:center; background-color: #001d6c; color: white;">AggregateF1</th>
251
+ <th style="text-align:center; background-color: #001d6c; color: white;">AegisSafetyTest</th>
252
+ <th style="text-align:center; background-color: #001d6c; color: white;">BeaverTails</th>
253
+ <th style="text-align:center; background-color: #001d6c; color: white;">HarmBench_Prompt</th>
254
+ <th style="text-align:center; background-color: #001d6c; color: white;">OAI_hf</th>
255
+ <th style="text-align:center; background-color: #001d6c; color: white;">SafeRLHF_test</th>
256
+ <th style="text-align:center; background-color: #001d6c; color: white;">simpleSafetyTest</th>
257
+ <th style="text-align:center; background-color: #001d6c; color: white;">toxic_chat</th>
258
+ <th style="text-align:center; background-color: #001d6c; color: white;">xstest_RH</th>
259
+ <th style="text-align:center; background-color: #001d6c; color: white;">xstest_RR</th>
260
+ <th style="text-align:center; background-color: #001d6c; color: white;">xstest_RR(h)</th>
261
+ </tr></thead>
262
+ <tbody>
263
+ <tr>
264
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.1-8b</td>
265
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.79 </td>
266
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.88 </td>
267
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.81 </td>
268
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
269
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.78 </td>
270
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.81 </td>
271
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.99 </td>
272
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.73 </td>
273
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.87 </td>
274
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.45 </td>
275
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.83 </td>
276
+ </tr>
277
+ <tr>
278
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.2-5b</td>
279
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.78 </td>
280
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.88 </td>
281
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.81 </td>
282
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
283
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.73 </td>
284
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
285
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.99 </td>
286
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.73 </td>
287
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.90 </td>
288
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.43 </td>
289
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.82 </td>
290
+ </tr>
291
+ <tr>
292
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (no_think)</td>
293
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.81 </td>
294
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.87 </td>
295
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.84 </td>
296
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
297
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.77 </td>
298
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
299
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.99 </td>
300
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.76 </td>
301
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.90 </td>
302
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.49 </td>
303
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.87 </td>
304
+ </tr>
305
+ <tr>
306
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (think)</td>
307
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.79 </td>
308
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.86 </td>
309
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.82 </td>
310
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.80 </td>
311
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.78 </td>
312
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.78 </td>
313
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.99 </td>
314
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.69 </td>
315
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.86 </td>
316
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.50 </td>
317
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.86 </td>
318
+ </tr>
319
+ </tbody></table>
320
+
321
 
322
 
323
  ### RAG Hallucination Benchmarks
324
  For detecting hallucinations in RAG settings, the model is evaluated on [LM-AggreFact](https://llm-aggrefact.github.io/) benchmarks. We report balanced accuracy scores on LM AggreFact below:
325
 
326
+ <table>
327
+ <caption style="text-align:center"><b>LM-Aggrefact</b></caption>
328
+ <thead>
329
+ <tr>
330
+ <th style="text-align:left; background-color: #001d6c; color: white;">Models</th>
331
+ <th style="text-align:center; background-color: #001d6c; color: white;">AVG</th>
332
+ <th style="text-align:center; background-color: #001d6c; color: white;">AggreFact-CNN</th>
333
+ <th style="text-align:center; background-color: #001d6c; color: white;">AggreFact-XSum</th>
334
+ <th style="text-align:center; background-color: #001d6c; color: white;">ClaimVerify</th>
335
+ <th style="text-align:center; background-color: #001d6c; color: white;">ExpertQA</th>
336
+ <th style="text-align:center; background-color: #001d6c; color: white;">FactCheck-GPT</th>
337
+ <th style="text-align:center; background-color: #001d6c; color: white;">Lfqa</th>
338
+ <th style="text-align:center; background-color: #001d6c; color: white;">RAGTruth</th>
339
+ <th style="text-align:center; background-color: #001d6c; color: white;">Reveal</th>
340
+ <th style="text-align:center; background-color: #001d6c; color: white;">TofuEval-MediaS</th>
341
+ <th style="text-align:center; background-color: #001d6c; color: white;">TofuEval-MeetB</th>
342
+ <th style="text-align:center; background-color: #001d6c; color: white;">Wice</th>
343
+ </tr></thead>
344
+ <tbody>
345
+ <tr>
346
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.1-8b</td>
347
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.709 </td>
348
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.532 </td>
349
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.570 </td>
350
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.724 </td>
351
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.597 </td>
352
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.759 </td>
353
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.855 </td>
354
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.768 </td>
355
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.877 </td>
356
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.725 </td>
357
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.761 </td>
358
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.635 </td>
359
+ </tr>
360
+ <tr>
361
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.2-5b</td>
362
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.665 </td>
363
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.508 </td>
364
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.530 </td>
365
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.650 </td>
366
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.596 </td>
367
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.743 </td>
368
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.808 </td>
369
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.630 </td>
370
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.872 </td>
371
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.691 </td>
372
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.685 </td>
373
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.604 </td>
374
+ </tr>
375
+ <tr>
376
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (no_think)</td>
377
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.761 </td>
378
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.669 </td>
379
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.738 </td>
380
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.767 </td>
381
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.596 </td>
382
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.729 </td>
383
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.878 </td>
384
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.831 </td>
385
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.894 </td>
386
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.736 </td>
387
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.815 </td>
388
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.720 </td>
389
+ </tr>
390
+ <tr>
391
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (think)</td>
392
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.765 </td>
393
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.661 </td>
394
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.749 </td>
395
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.759 </td>
396
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.597 </td>
397
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.766 </td>
398
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.870 </td>
399
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.821 </td>
400
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.896 </td>
401
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.739 </td>
402
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.789 </td>
403
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.773 </td>
404
+ </tr>
405
+ </tbody></table>
406
+
407
 
408
 
409
  We also report performance on TRUE benchmark (balanced accuracy) that measures faithfulness of LLM responses to the context.
410
 
411
+ <table>
412
+ <caption style="text-align:center"><b>TRUE</b></caption>
413
+ <thead>
414
+ <tr>
415
+ <th style="text-align:left; background-color: #001d6c; color: white;">Models</th>
416
+ <th style="text-align:left; background-color: #001d6c; color: white;">AVG</th>
417
+ <th style="text-align:center; background-color: #001d6c; color: white;">begin</th>
418
+ <th style="text-align:center; background-color: #001d6c; color: white;">dialfact</th>
419
+ <th style="text-align:center; background-color: #001d6c; color: white;">frank</th>
420
+ <th style="text-align:center; background-color: #001d6c; color: white;">mnbm</th>
421
+ <th style="text-align:center; background-color: #001d6c; color: white;">paws</th>
422
+ <th style="text-align:center; background-color: #001d6c; color: white;">q2</th>
423
+ <th style="text-align:center; background-color: #001d6c; color: white;">qags_cnndm</th>
424
+ <th style="text-align:center; background-color: #001d6c; color: white;">qags_xsum</th>
425
+ <th style="text-align:center; background-color: #001d6c; color: white;">summeval</th>
426
+ </tr></thead>
427
+ <tbody>
428
+ <tr>
429
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.1-8b</td>
430
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.725 </td>
431
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.714 </td>
432
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.630 </td>
433
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.835 </td>
434
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.648 </td>
435
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.780 </td>
436
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.710 </td>
437
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.756 </td>
438
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.717 </td>
439
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.733 </td>
440
+ </tr>
441
+ <tr>
442
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.2-5b</td>
443
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.710 </td>
444
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.740 </td>
445
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.694 </td>
446
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.791 </td>
447
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.635 </td>
448
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.749 </td>
449
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.727 </td>
450
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.723 </td>
451
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.660 </td>
452
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.672 </td>
453
+ </tr>
454
+ <tr>
455
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (no_think)</td>
456
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.777 </td>
457
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.733 </td>
458
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.684 </td>
459
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.886 </td>
460
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.660 </td>
461
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.825 </td>
462
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.801 </td>
463
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.814 </td>
464
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.796 </td>
465
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.796 </td>
466
+ </tr>
467
+ <tr>
468
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (think)</td>
469
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.773 </td>
470
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.732 </td>
471
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.722 </td>
472
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.864 </td>
473
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.680 </td>
474
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.813 </td>
475
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.799 </td>
476
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.792 </td>
477
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.798 </td>
478
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.761 </td>
479
+ </tr>
480
+ </tbody></table>
481
 
482
 
483
  ### Function Calling Hallucination Benchmarks
484
  The model performance is evaluated on the [FC Reward Bench evaluation](https://huggingface.co/datasets/ibm-research/fc-reward-bench) dataset. We use balanced accuracy as the metric to compare the various models.
485
 
486
+ <table>
487
+ <caption style="text-align:center"><b>fc-reward-bench</b></caption>
488
+ <thead>
489
+ <tr>
490
+ <th style="text-align:left; background-color: #001d6c; color: white;">Models</th>
491
+ <th style="text-align:center; background-color: #001d6c; color: white;">AVG</th>
492
+ </tr></thead>
493
+ <tbody>
494
+ <tr>
495
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.1-8b</td>
496
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.64 </td>
497
+ </tr>
498
+ <tr>
499
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.2-5b</td>
500
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.61 </td>
501
+ </tr>
502
+ <tr>
503
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (no_think)</td>
504
+ <td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;"> 0.74 </td>
505
+ </tr>
506
+ <tr>
507
+ <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">granite-guardian-3.3-8b (think)</td>
508
+ <td style="text-align:center; background-color: #FFFFFF; color: black;"> 0.71 </td>
509
+ </tr>
510
+ </tbody></table>
511
 
512
 
513
  ## Training Data