LFM2.5-350M-Python-Math

A fine-tune of LiquidAI/LFM2.5-350M (instruct) focused on Python code generation and math word-problem solving, while retaining general chat ability through a balanced mixed dataset.

Why this exists

The previous 230M fine-tune (lfm2.5-230m-code-math) showed strong potential but suffered from catastrophic forgetting (e.g., confusing baking cookies with HTTP cookies, failing negative constraints like "no dairy"). This 350M version addresses those issues by:

Mixing general chat data (yahma/alpaca-cleaned, 30k samples) to prevent knowledge loss.
Injecting custom fix-it examples targeting specific failure modes (negative constraints, complete Pygame scripts).
Using longer context (2048 tokens) so code outputs aren't truncated mid-function.
Reducing epochs to 2 with a lower learning rate (2e-5) to prevent overfitting observed in earlier runs.

Fine-tuning started from the instruct checkpoint rather than base. Testing confirmed that at 350M scale, starting from base with a mixed dataset still produced alignment failures (refusals, identity confusion, math regression), while the instruct checkpoint with the same dataset produced consistently strong results.

Training details

Base model: LiquidAI/LFM2.5-350M (instruct)
Method: Full fine-tune (96GB VRAM, no LoRA needed)
Datasets:
- Python Code: iamtarun/python_code_instructions_18k_alpaca (Python-focused, replacing the multi-language 120k set)
- Math: openai/gsm8k (main split)
- General Chat: yahma/alpaca-cleaned (30k sample subset)
- Custom Fix-It: Hand-crafted examples for negative constraints ("no dairy", "no eggs") and complete runnable Pygame scripts (duplicated 50x for weight)
Checkpoint selection: Best by eval_loss
Sequence length: 2048 tokens (increased from 1024 to accommodate full scripts)
Max response chars: 3500 (prevents code truncation)
Epochs: 2 (reduced from 4; overfitting observed past epoch 2 in prior runs)
Learning rate: 2e-5 (reduced from 5e-5 for 350M stability)
Loss: Completion-only

What it's good at

Python Code: Complete, runnable scripts including Pygame game loops, file I/O, classes, list comprehensions, and algorithmic implementations (e.g., two-pointer palindrome check). No more placeholder pass statements or truncated functions.
Math: GSM8K-style word problems with step-by-step reasoning annotations (<<...>>). Reliable on algebra, percentages, geometry, and multi-step arithmetic.
General Chat: Retains coherent conversational ability. Correctly handles negative constraints (e.g., "breakfast without eggs" returns egg-free options). Knows the difference between baking cookies and browser cookies.
Speed: At 350M parameters, achieves ~157 t/s generation on laptop CPU (i5-12450H) with Q5_K_S quantization via llama.cpp.

Known limitations

Python only: Trained exclusively on Python code instructions. Other languages were not included in this fine-tune.
Sentence counting: May not strictly adhere to "exactly N sentences" constraints.
Identity: May occasionally claim to be developed by Google (artifact from Alpaca-Cleaned training data).
Still 350M parameters: Do not expect deep multi-step reasoning or long-form creative writing at the level of larger models.
Not evaluated on safety-critical, medical, or legal use cases.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hauser458original/lfm2.5-350m-python-math"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

output = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.5, top_p=0.9)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

GGUF quantized versions (Q4_K_M, Q5_K_S, Q5_K_M, Q8_0, F16) for llama.cpp/Ollama/LM Studio are available at: hauser458original/lfm2.5-350m-python-math-GGUF