Ora

How to Run Llama 2 with Python

Published in Large Language Models 8 mins read

Running Llama 2 with Python involves a few key steps, primarily centered around acquiring the model, setting up your environment, and using a suitable library or framework for inference. This guide covers two popular methods: using community-optimized implementations like llama.cpp with Python bindings for efficient local execution, and leveraging the user-friendly Hugging Face Transformers library.

Understanding Llama 2 and Its Requirements

Llama 2 is a powerful large language model developed by Meta AI. To run it effectively on your local machine, especially the larger versions, you'll need significant computational resources, particularly a Graphics Processing Unit (GPU) with ample Video RAM (VRAM).

Hardware Considerations

  • GPU: An NVIDIA GPU is highly recommended for optimal performance, along with CUDA drivers.
  • VRAM: The amount of VRAM required depends on the model size and whether you use quantization (e.g., 4-bit, 8-bit).
  • CPU & RAM: While less critical than VRAM, a modern CPU and sufficient system RAM are still important, especially for CPU-only inference or larger models.

Llama 2 Model Sizes and VRAM Estimates

Understanding the VRAM requirements is crucial for choosing which Llama 2 model variant you can run. Quantization significantly reduces VRAM usage.

Llama 2 Model Parameters Approximate VRAM (FP16) Approximate VRAM (4-bit Quantized)
Llama 2 7B 7 Billion ~14 GB ~5-6 GB
Llama 2 13B 13 Billion ~26 GB ~9-10 GB
Llama 2 70B 70 Billion ~140 GB ~40-45 GB

Note: These are approximations. Actual usage can vary based on batch size, context length, and other factors.

Method 1: Running with llama.cpp and Python Bindings for Local Optimization

This method is excellent for efficient CPU or GPU (via cuBLAS) inference, often making larger models runnable on less powerful hardware through aggressive quantization. It directly incorporates building the code, as suggested by the make instruction.

Step 1: Download the Llama 2 Model Files

You need to obtain the Llama 2 model weights.

  1. Request Access: Visit the official Meta AI Llama 2 page and request access. Once approved, you'll receive a link to download the weights.
  2. Download Specific Format: For llama.cpp, you will need the model weights converted to the GGUF (GPT-Generated Unified Format) format. These are often available on community platforms like Hugging Face (search for "llama 2 gguf"). Download the specific GGUF file for the model size and quantization level you desire (e.g., llama-2-7b-chat.Q4_K_M.gguf).

Step 2: Set Up Your Python Virtual Environment

It's best practice to isolate your project dependencies.

  1. Create Virtual Environment:
    python3 -m venv llama2_env
  2. Activate Virtual Environment:
    • On Linux/macOS:
      source llama2_env/bin/activate
    • On Windows:
      llama2_env\Scripts\activate

Step 3: Clone the llama.cpp Repository

The llama.cpp project provides a C++ implementation of Llama that can run efficiently on various hardware, with Python bindings available.

  1. Clone the Repository:
    git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

Step 4: Build the llama.cpp Code

This step compiles the C++ code for llama.cpp and is crucial for enabling optimized execution.

  1. Build with make:
    • For CPU-only:
      make
    • For GPU acceleration (NVIDIA CUDA):
      make LLAMA_CUBLAS=1

      Ensure you have CUDA Toolkit installed for GPU acceleration.

  2. Install Python Bindings: After building llama.cpp, install its Python bindings (llama-cpp-python).
    • For CPU-only:
      pip install llama-cpp-python
    • For GPU acceleration (ensure you are in the llama.cpp directory where make was run for cuBLAS):
      pip install llama-cpp-python --force-reinstall --no-cache-dir --verbose
# If the above fails to pick up cuBLAS, you might need to specify the build flags:
# CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python

Step 5: Run Inference with Python

Now you can use Python to load your downloaded GGUF model and interact with Llama 2.

from llama_cpp import Llama

# Make sure to replace 'path/to/your/llama-2-7b-chat.Q4_K_M.gguf'
# with the actual path to your downloaded GGUF model file.
llm = Llama(model_path="./llama-2-7b-chat.Q4_K_M.gguf", n_ctx=2048) # n_ctx is context window size

prompt = "Q: What is the capital of France? A:"
output = llm(prompt, max_tokens=32, stop=["Q:", "\n"], echo=True)

print(output["choices"][0]["text"])

# Example for a chat-like interaction
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me a short story about a brave knight."},
]

# The 'create_chat_completion' method is more suitable for chat models
chat_output = llm.create_chat_completion(messages=messages, max_tokens=100)
print(chat_output["choices"][0]["message"]["content"])

Method 2: Simplified Approach with Hugging Face Transformers Library

The Hugging Face transformers library offers a streamlined Python-only experience for loading and running Llama 2, especially when using models pre-converted for the library.

Step 1: Prerequisites & Model Access

  1. Ensure GPU Drivers: Make sure your NVIDIA GPU drivers and CUDA Toolkit are properly installed.
  2. Hugging Face Access: Llama 2 models on Hugging Face require you to accept their terms and conditions.
    • Go to the Meta Llama 2 page on Hugging Face and click "Access Repository."
    • You might need to log in to Hugging Face and use a Hugging Face token in your environment or code.
      huggingface-cli login
# Enter your token when prompted

Step 2: Install Required Python Libraries

Using your activated virtual environment from Step 2 above (or a new one):

pip install transformers torch accelerate bitsandbytes sentencepiece
  • transformers: The core library for loading models.
  • torch: PyTorch, the deep learning framework.
  • accelerate: Helps with efficiently loading and running large models, especially across multiple GPUs or with offloading.
  • bitsandbytes: Essential for loading models in 4-bit or 8-bit quantized formats to save VRAM.
  • sentencepiece: Tokenizer used by Llama 2.

Step 3: Load and Run the Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

# Specify the model ID from Hugging Face
# You'll need to use a specific model, e.g., 'meta-llama/Llama-2-7b-chat-hf'
# Make sure you have access to this model on Hugging Face.
model_id = "meta-llama/Llama-2-7b-chat-hf"

# Configure 4-bit quantization for efficient memory usage
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load the model with quantization
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto" # Automatically map model layers to available devices (GPU/CPU)
)

# Prepare your prompt
prompt = "Tell me a fun fact about giraffes."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda") # Send inputs to GPU

# Generate response
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        num_return_sequences=1,
        pad_token_id=tokenizer.eos_token_id # Important for generation
    )

# Decode and print the output
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

# Example for a chat-like interaction (Llama 2 Chat models often use a specific chat template)
chat_prompt = [
    {"role": "user", "content": "What are the benefits of learning Python?"}
]
tokenized_chat = tokenizer.apply_chat_template(chat_prompt, tokenize=True, add_generation_prompt=True, return_tensors="pt")
tokenized_chat = tokenized_chat.to("cuda")

with torch.no_grad():
    chat_outputs = model.generate(
        tokenized_chat,
        max_new_tokens=200,
        pad_token_id=tokenizer.eos_token_id
    )

chat_response = tokenizer.decode(chat_outputs[0][tokenized_chat.shape[-1]:], skip_special_tokens=True)
print(chat_response)

Key Considerations for Running Llama 2

  • Model Variants: Llama 2 comes in base and chat-tuned versions (e.g., Llama-2-7b vs. Llama-2-7b-chat). Choose the one appropriate for your task. Chat models are fine-tuned for conversational interactions.
  • Prompt Engineering: The way you phrase your input (prompt) significantly impacts the model's output. For chat models, follow their specific conversation templates (e.g., [INST] ... [/INST]).
  • Performance Tuning:
    • n_ctx (context window): For llama.cpp, this controls how much history the model remembers. Increase if needed, but it consumes more RAM.
    • max_tokens/max_new_tokens: Limits the length of the generated response.
    • temperature, top_p, top_k: Control the creativity and randomness of the output.
  • Safety and Responsible AI: Llama 2 is a powerful tool; always consider ethical implications and potential biases in its outputs.

Troubleshooting Common Issues

  • Out of Memory (OOM) Errors:
    • Reduce max_new_tokens.
    • Use a smaller Llama 2 model (e.g., 7B instead of 13B).
    • Utilize quantization (4-bit or 8-bit).
    • If using Hugging Face, ensure device_map="auto" and quantization_config are correctly set.
  • Installation Problems:
    • Ensure your Python virtual environment is active.
    • Verify pip is up-to-date (pip install --upgrade pip).
    • For llama.cpp with GPU, double-check CUDA toolkit installation and make commands.
  • Model Loading Errors:
    • Ensure you have proper access (Meta AI agreement, Hugging Face login/token).
    • Verify the model_path (for llama.cpp) or model_id (for Hugging Face) is correct.
    • Check for corrupted model files.

By following these steps, you can successfully run Llama 2 with Python, choosing the method that best suits your hardware and desired level of control.

← Previous Article
What is an Arytenoid Lateralization (Tie-Back) Procedure?
Next Article →
How Do You Equip Palamute Gear?