# Synthetic Preference Data Generation with Distilabel and Llama 3.1

> Pipeline pattern for building preference datasets using Distilabel and local LLMs

A Distilabel workflow pattern that automates the generation of synthetic instruction-response datasets. It prompts two different Llama 3.1 models via Ollama and uses a larger model as an UltraFeedback judge to score and rank outputs. It is designed to quickly produce domain-specific preference pairs for fine-tuning smaller models.

## How it works

Use Distilabel (open-source framework) with Llama 3.1 via Ollama to generate synthetic datasets. Pattern: two LLMs (llama3.1 8B and llama3.1:70b-instruct) each generate a response to an instruction loaded from the Hub; responses are combined into one column; an LLM-as-judge (UltraFeedback, aspect 'overall-rating') rates them; the best response is paired with the instruction. Pipeline: load\_dataset \>\> generate \>\> combine \>\> evaluate. Code imports: from distilabel.llms import OllamaLLM; from distilabel.pipeline import Pipeline; from distilabel.steps import LoadDataFromHub, CombineColumns, tasks. Key snippets: llama8B=OllamaLLM(model="llama3.1", host=host\_llama\_8b); llama70B=OllamaLLM(model="llama3.1:70b-instruct-q2\_k", host=host\_llama\_70b); with Pipeline(name="preference-datagen-llama3.1") as pipeline: load\_dataset=LoadDataFromHub(name="load\_dataset", output\_mappings={"prompt":"instruction"}); generate=\[tasks.TextGeneration(name='text\_generation\_8B', llm=llama8B), tasks.TextGeneration(name='text\_generation\_70B', llm=llama70B)\]; combine=CombineColumns(columns=\["generation","model\_name"\], output\_columns=\["generations","model\_names"\]); evaluate=tasks.UltraFeedback(aspect="overall-rating", llm=llama70B).

## What it is for

- Generate preference datasets for DPO or RLHF fine-tuning
- Compare outputs between smaller and larger quantized LLM variants
- Run automated LLM-as-a-judge scoring on synthetic data pipelines

## Key points

- Loads seed prompts directly from Hugging Face Hub using LoadDataFromHub
- Runs multi-model inference simultaneously via OllamaLLM backends
- Combines multi-model outputs into unified tabular records using CombineColumns
- Evaluates candidate responses using the UltraFeedback task for automated ranking
- Runs fully locally or self-hosted to keep dataset generation private

## Pricing

Open source

## Details

- Type: Tip
- Category: [AI](https://xiry.app/topics/ai.md)
- Open source: yes
- Platforms: Code library, Linux, macOS
- For: Machine learning engineers and researchers fine-tuning open-weight LLMs
- Tags: distilabel, synthetic data, llama3, rlhf, dpo, ollama

## Why it was saved

- [@akshay_pachaar, 9 December 2024](https://x.com/akshay_pachaar/status/1866098026361880805): Produces (instruction, response) pairs and judge-rated preferences for fine-tuning small LLMs.

## Related

- [Local Deep Researcher](https://xiry.app/resources/local-deep-researcher-2395): Fully local web research and report writing assistant using Ollama or LMStudio (GitHub repo, Open source)
- [AgenticSeek](https://xiry.app/resources/agenticseek): Open-source, fully local autonomous AI agent that browses and writes code (GitHub repo, Open source)
- [Browser Use WebUI](https://xiry.app/resources/browser-use-webui): Open-source graphical interface for running autonomous AI browser agents locally. (Web app, Open source)
- [Dolphin 3.0](https://xiry.app/resources/dolphin-3-0): Uncensored, open-source fine-tuned LLM designed for local execution without guardrails (AI model, Open source)
- [Llama-3.1-Nemotron-70B-Instruct](https://xiry.app/resources/llama-3-1-nemotron-70b-instruct): NVIDIA alignment fine-tune of Llama 3.1 70B for enhanced instruction following (AI model, Free)
- [OpenRAG](https://xiry.app/resources/openrag): All-in-one local stack for building and running RAG applications (GitHub repo, Open source)

Updated 11 October 2026. Page: https://xiry.app/resources/synthetic-preference-data-generation-with-distilabel-and-lla

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.