A Comparative Study of Llama-3.2-3B and Mistral-Nemo-12B on News Summarization

Abstract

We present a comparative evaluation of two open-source large language models — Meta's Llama-3.2-3B-Instruct and Mistral AI's Mistral-Nemo-Instruct-2407 (12B) — on the task of abstractive news summarization. Using a zero-shot instruction-tuned prompting strategy, we evaluate both models across three complementary dimensions: lexical fidelity (ROUGE-L), semantic alignment (BERTScore F1), and inference efficiency (average latency). Our results indicate that Mistral-Nemo achieves superior summarization quality across both automatic metrics, while Llama-3.2-3B offers a substantial latency advantage under CPU-constrained inference conditions. We discuss the quality-deployability tradeoff surfaced by this benchmark and outline directions for future work including faithfulness evaluation and human-in-the-loop auditing.

1. Introduction

Abstractive summarization — the task of generating a concise, coherent summary that captures the salient meaning of a source document — has emerged as a canonical benchmark for evaluating the natural language understanding capabilities of large language models (LLMs). Unlike extractive methods, which select and concatenate spans from the source, abstractive models must paraphrase, compress, and synthesize information, making the task both linguistically demanding and practically valuable for downstream applications in journalism, legal analysis, and information retrieval.

The rapid proliferation of open-source LLMs has created a practical need for rigorous comparative benchmarks that extend beyond raw parameter count or perplexity. Practitioners selecting models for deployment must balance multiple competing objectives: summarization quality, inference cost, hardware requirements, and robustness to hallucination. Reference-based metrics such as ROUGE and BERTScore have become standard proxies for quality, yet neither directly penalises factual fabrication — a critical failure mode in real-world summarization pipelines.

This paper addresses two research questions:

1. RQ1: To what extent does model scale (3B vs. 12B parameters) translate to measurable gains in summarization quality under zero-shot instruction tuning?

2. RQ2: What is the practical inference cost of those quality gains, and how does it interact with hardware constraints?

We conduct our evaluation on the CNN/DailyMail benchmark (v3.0.0), a widely adopted dataset for news summarization, using 20 samples from the held-out test split.

2. Related Work

Automatic summarization evaluation. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) remains the dominant evaluation framework for summarization, with ROUGE-L measuring longest common subsequence overlap between generated and reference summaries (Lin, 2004). BERTScore extends this paradigm by computing token-level similarity using contextual embeddings from pre-trained language models, enabling evaluation of semantic equivalence beyond surface-level n-gram matching (Zhang et al., 2020).

LLM-based summarization. Prior work has demonstrated that instruction-tuned LLMs achieve competitive performance on abstractive summarization under zero-shot conditions (Goyal et al., 2022; Zhang et al., 2023), often outperforming fine-tuned sequence-to-sequence models on out-of-domain inputs. However, performance gains tend to correlate strongly with model scale, raising questions about the practical viability of large models in resource-constrained deployment settings.

Faithfulness in summarization. A well-documented limitation of reference-based evaluation is its insensitivity to hallucination — the generation of plausible but factually unsupported content (Maynez et al., 2020). Recent work has proposed factual consistency metrics based on natural language inference (Falke et al., 2019) and question-answering generation (QAG) frameworks (Wang et al., 2020) to address this gap. We identify faithfulness evaluation as a critical direction for future work in our setting.

3. Experimental Setup

3.1 Dataset

We evaluate on the CNN/DailyMail dataset (Hermann et al., 2015), version 3.0.0, using 20 samples drawn from the test split. Each sample consists of a news article paired with human-written highlight bullet points, which serve as the reference summary. While the subset size limits statistical power, it enables a controlled comparison under the computational constraints of local CPU inference.

3.2 Models

Both models are open-source instruction-tuned variants accessed via the HuggingFace Hub. Mistral-Nemo was loaded with 4-bit NF4 quantization (Dettmers et al., 2023) to reduce its memory footprint from approximately 24 GB (bfloat16) to approximately 7 GB, enabling inference on consumer hardware.

Note on experimental confounding: Llama-3.2-3B was loaded in full bfloat16 precision without quantization — a deliberate decision reflecting its smaller parameter count and commensurately lower memory requirements. As a consequence, the two models operate under different numerical precision regimes, which may modestly influence both output quality and latency. We report this as a known limitation and recommend controlled hardware benchmarking as future work.

3.3 Prompt Strategy

Both models were prompted under a zero-shot instruction-tuned regime. Each model's native chat template was applied via HuggingFace's apply_chat_template, with a shared system instruction:

"You are a professional news editor. Summarize the following article in exactly 3 concise bullet points. Each bullet must start with '•'. Output only the bullet points, nothing else."

Llama-3.2-3B used a pre-formatted prompt string incorporating Llama-3 special tokens (<|begin_of_text|>, <|start_header_id|>), while Mistral-Nemo received a plain-text article input through the structured messages API. This asymmetry reflects each model's native prompt format requirements.

3.4 Generation Parameters

3.5 Evaluation Metrics

ROUGE-L measures the F1 score of the longest common subsequence between the generated summary and the reference, capturing both recall and precision of lexical overlap while remaining sensitive to word order.

BERTScore F1 computes token-level cosine similarity between generated and reference summaries using roberta-large embeddings, capturing semantic equivalence beyond surface-level wording.

Both metrics are computed over the 20-sample test subset, filtering to successful generations only.

4. Results

4.1 Summarization Quality

Mistral-Nemo outperforms Llama-3.2-3B on both quality metrics. The ROUGE-L gap (+40.1% relative improvement) is substantial, suggesting that Nemo's generated summaries exhibit meaningfully stronger lexical alignment with human-written references. The BERTScore gap is more modest (+1.5%), indicating that both models preserve semantic meaning at a broadly similar level despite the lexical difference — Llama more frequently paraphrases where Nemo reproduces phrasing closer to the source.

4.2 Inference Latency

Llama-3.2-3B is approximately 23× faster under CPU-constrained inference. This gap reflects both the raw parameter count difference (3B vs. 12B) and the inference backend differences noted in Section 3.2. Both models were evaluated on CPU without GPU acceleration; with a CUDA-capable GPU and 4-bit NF4 quantization, Nemo's latency is expected to reduce to approximately 15–30 seconds per sample, substantially narrowing the gap.

5. Discussion

5.1 Quality vs. Deployability

Our results surface a fundamental tradeoff in applied NLP: larger models produce higher-quality outputs, but at inference costs that may be prohibitive in resource-constrained or latency-sensitive deployments. Mistral-Nemo's ROUGE-L advantage (+40.1%) is operationally meaningful — a model that better reproduces the salient content of an article in reference-aligned language is genuinely more useful for summarization downstream tasks. However, a 23× latency penalty under CPU inference renders it impractical for real-time applications without dedicated GPU hardware.

For practitioners, the choice between these models is therefore not a quality question but a deployment question: latency requirements and hardware availability should drive model selection at least as much as benchmark scores.

5.2 Limitations of Reference-Based Evaluation

A critical limitation of both ROUGE-L and BERTScore is their insensitivity to hallucination. Both metrics evaluate similarity to a human-written reference summary; neither penalises a fluent, plausible summary that fabricates facts not present in the source article. A model could, in principle, achieve high scores on both metrics while generating factually unsupported content. This is a well-documented failure mode of neural abstractive summarization (Maynez et al., 2020), and motivates the faithfulness evaluation planned for future work.

5.3 Prompt Format Asymmetry

As noted in Section 3.3, Llama and Nemo were prompted using different structural formats — a consequence of their differing native chat templates. While both prompts convey an equivalent instruction, the formatting difference introduces a potential confound: differences in output quality may reflect prompt sensitivity as much as model capability. A controlled experiment with a unified prompting strategy would provide a cleaner comparison.

6. Future Work

GPU benchmarking. Rerunning Nemo inference on a CUDA-capable GPU with 4-bit NF4 quantization active would provide a latency comparison under equivalent hardware conditions, isolating the effect of model size from hardware constraints.

Faithfulness and QAG evaluation. Complementing ROUGE and BERTScore with faithfulness scoring (e.g. SummaC) and question-answering generation (QAG) metrics would directly penalise hallucination — a dimension absent from the current evaluation. We consider this the highest-priority extension of this work.

Human-in-the-loop audit. A manual review of approximately 10% of outputs by a domain expert would surface hallucinations, "lost in the middle" phenomena, and verbosity bias not captured by automated metrics.

Dataset scale. Extending the evaluation from 20 to 200+ samples would reduce variance and improve the statistical reliability of metric comparisons, enabling significance testing between models.

Fine-tuning. Evaluating the effect of supervised fine-tuning on the CNN/DailyMail training split for both models would establish whether the observed quality gap narrows under task-specific adaptation.

Bias analysis. Applying a critical audit prompt to flag emotionally loaded language and framing asymmetries in generated summaries would extend this work into the sociotechnical evaluation of LLM-generated journalism.

7. Conclusion

We have presented a comparative benchmark of Llama-3.2-3B-Instruct and Mistral-Nemo-Instruct-2407 on zero-shot abstractive news summarization. Mistral-Nemo achieves superior summarization quality across both ROUGE-L (+40.1%) and BERTScore F1 (+1.5%), consistent with the expected scaling effect of a 4× parameter advantage. However, this quality gain comes at a 23× latency cost under CPU inference — a tradeoff with direct implications for production deployment decisions.

Our evaluation also highlights a methodological gap in standard summarization benchmarks: reference-based metrics cannot detect hallucination, and any production summarization system should treat faithfulness as a first-class evaluation criterion. We release our code and evaluation pipeline to support reproducibility and further benchmarking in this space.

References

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314.

Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019). Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. ACL 2019.

Goyal, T., Li, J. J., & Durrett, G. (2022). News summarization and evaluation in the era of GPT-3. arXiv preprint arXiv:2209.12356.

Hermann, K. M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. NeurIPS 2015.

Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. ACL Workshop on Text Summarization Branches Out.

Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. ACL 2020.

Wang, A., Cho, K., & Lewis, M. (2020). Asking and answering questions to evaluate the factual consistency of summaries. ACL 2020.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. ICLR 2020.

Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2023). Benchmarking large language models in complex instruction-following. arXiv preprint arXiv:2307.09007.

Contact

Sue Huynh

• Email: nguyen_huynh@brown.edu

• GitHub: https://github.com/suehuynh

Report completed: Summer 2026