AI Tips

Practical ways to train, run, or shrink AI models — explained for people new to AI. 1 new in last 30d.

New here? Each card answers one question: what is this and why should I care? Click a card to read the full explanation, including any new words. The command at the bottom is what you would type to try it on your own machine.

The AI flow — where each tip fits

Read left to right

An AI model goes through these five phases. Click a phase to see the tips that apply there.

1Pre-training
A model first learns language by reading huge amounts of text. This costs millions of dollars and runs on thousands of GPUs.
2Fine-tuning
You take that pre-trained model and teach it your own data, your own task, or your own writing style. Hours to days, on a few GPUs.
e.g. QLoRA · Unsloth · DeepSpeed ZeRO-3 / FSDP
See Training tips →
3Preference tuning
After fine-tuning, you teach the model which answers humans prefer. This makes it polite, helpful, and on-topic.
e.g. DPO / GRPO / KTO
See Training tips →
4Quantization
The trained model is huge. Quantization shrinks it about 4× by storing its numbers with less precision, so it fits on cheap hardware.
e.g. GGUF + llama.cpp · AWQ / GPTQ · EXL2
See Quantization tips →
5Inference / serving
Running the model so users can ask it questions. This is what your app actually does in production.
e.g. vLLM (PagedAttention) · Ollama · Speculative decoding
See Inference tips →

Live

Mixture of Experts (MoE) — huge model, fast model latency

The model is a team of small experts. For each word it only uses two of them. So a 141B model answers as fast as a 39B one.

Techniquedepends on top-k

A normal ('dense') model uses all of its parameters for every word. An MoE replaces one block with several smaller experts, plus a router that picks the top-k experts (usually 2) per word. So Mixtral-8x22B has 141B total parameters but only ~39B are active per word — answers come at the speed of a 39B model. The trade-off: you still have to load all experts in GPU memory, so VRAM is high. DeepSeek-V3 and Mixtral are the well-known open MoEs.

Try it

vllm serve mistralai/Mixtral-8x22B-Instruct-v0.1 --tensor-parallel-size 4

Source

Speculative decoding — make the big model faster, for free

A tiny model guesses the next words. The big model just checks the guesses in one batch. 2–3× faster, same answer quality.

Techniqueany inference target

Big models are slow because they generate one word (or token) at a time. With speculative decoding you also load a small, cheap 'draft' model. The draft model writes the next 5 tokens. The big model then runs once and checks all 5 in parallel. Tokens it agrees with are kept; tokens it does not agree with are replaced. The final answer is identical to what the big model would have written alone — you just spent fewer expensive runs to get there. Built into vLLM, TGI, and llama.cpp.

Try it

vllm serve meta-llama/Llama-3.1-70B --speculative-model meta-llama/Llama-3.2-1B --num-speculative-tokens 5

Source

Implement carryless multiplication in CUDA 13.3 for cryptographyNewAuto

NVIDIA CUDA 13.3 introduces carryless multiplication for faster cryptography.

TechniqueNVIDIA GPUs

NVIDIA CUDA 13.3 includes a dedicated hardware instruction for carryless multiplication, which can be utilized to build faster cryptography solutions on NVIDIA GPUs.

Try it

# Example CUDA code snippet for carryless multiplication
#include <cuda_fp16.h>
__global__ void carryless_multiplication(float16_t *a, float16_t *b, float16_t *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        c[i] = __mul24(a[i], b[i]);
    }
}

Source

Optimize Memory Bandwidth and Reduce Launch Overhead with Kernel Fusion in CUDAAuto

Improve memory bandwidth and reduce kernel launch overhead using kernel fusion

TechniqueRTX 3090 24GB

NVIDIA's blog post explains how kernel fusion can be used to optimize memory traffic and reduce kernel launch overhead in CUDA. By combining multiple kernels into a single kernel launch, data transfer between the CPU and GPU is minimized, and the overhead of launching multiple kernels is reduced, leading to performance improvements.

Try it

# Example CUDA code snippet for kernel fusion
__global__ void fusedKernel(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        a[i] = b[i] + c[i];
        b[i] = a[i] * c[i];
    }
}

Source

Optimize memory traffic and reduce launch overhead with kernel fusion in CUDAAuto

Improve GPU performance by fusing multiple kernels into a single one

TechniqueRTX 3090 24GB

Kernel fusion is a technique that combines multiple kernels into a single one to optimize memory bandwidth and reduce kernel launch overhead, as explained in NVIDIA's blog post.

Try it

# Example of kernel fusion in CUDA
__global__ void fusedKernel(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        a[i] = b[i] + c[i];
        b[i] = a[i] - c[i];
    }
}

Source