PriceLLaMAA-2025-04-28_07.20.50

This repository contains a fine-tuned LLaMa model for predicting product prices based on descriptions. It's trained using the ed-donner/pricer-data dataset and the trl library for Supervised Fine-Tuning (SFT) with LoRA (Low-Rank Adaptation).

Model description

The base model used is meta-llama/Meta-Llama-3.1-8B. It's quantized to 4 bits using bitsandbytes for memory efficiency. The model is fine-tuned using LoRA, targeting specific layers (q_proj, v_proj, k_proj, o_proj) for efficient adaptation.

Intended Uses

• Price Prediction:
  The model is designed to predict or estimate the price of a product based on its textual description.

• E-commerce Applications:
  Can be used by online sellers, marketplaces, or catalog management systems to suggest initial pricing based on product descriptions.

• Data Augmentation:
  Helpful for generating synthetic price labels for datasets during training of other machine learning models.

• Market Research:
  Can assist analysts in comparing how similar product descriptions could correlate with price estimates.

---

Limitations

• Domain-Specific:
  The model is trained primarily on e-commerce-style product descriptions. It may not perform well outside typical retail scenarios (e.g., luxury items, collectibles, services).

• No Real-Time Market Awareness:
  The model does not have access to real-time pricing, supply-demand factors, or current market trends.

• Approximate Predictions:
  Outputs are estimates based on learned patterns in the training data and are not guaranteed to be accurate for production financial decisions.

• Bias from Training Data:
  If the training dataset contains biases (e.g., certain product categories being overpriced/underpriced), the model may inherit those biases.

• Language and Format Sensitivity:
  Descriptions that are extremely short, poorly written, or in languages/formats very different from the training data may yield poor predictions.

---

Training Details

• Dataset: ed-donner/pricer-data
• Base Model: meta-llama/Meta-Llama-3.1-8B
• Quantization: 4-bit NF4
• Fine-tuning Method: LoRA with SFT
• Library: trl
• Hyperparameters: See the training script in the repository for detailed hyperparameter values.

Training Procedure

The model was fine-tuned using Supervised Fine-Tuning (SFT) combined with LoRA for parameter-efficient adaptation. The base model meta-llama/Meta-Llama-3.1-8B was loaded in 4-bit precision to optimize memory usage.

The training steps were:

1. Model Preparation:

   • Loaded the base model (Meta-Llama-3.1-8B) in 4-bit NF4 quantization using bitsandbytes.
   • Applied a LoRA configuration targeting the following modules:
     • q_proj
     • k_proj
     • v_proj
     • o_proj

2. Dataset:

   • Used the ed-donner/pricer-data dataset, which consists of product descriptions and corresponding prices.

3. Training Setup:

   • Fine-tuned using the trl library's SFTTrainer.
   • Optimizer: PagedAdamW with betas=(0.9, 0.999) and epsilon=1e-08.
   • Learning rate scheduler: Cosine decay schedule with 3% warmup ratio.
   • Random seed: 42 for reproducibility.

4. Hyperparameters:

   • Learning Rate: 1e-4
   • Training Batch Size: 2
   • Evaluation Batch Size: 1
   • Number of Epochs: 1

5. Monitoring:

   • Tracked training loss and evaluation metrics using Weights & Biases (wandb).

6. Saving:

   • Only the LoRA adapters were saved, keeping the base model frozen to ensure lightweight deployment.

The entire training was optimized for fast prototyping and low GPU memory usage without sacrificing too much performance.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 1

Framework versions

• PEFT 0.14.0
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1

Demo Usage

You can use the model for inference like this:

from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

# Load the base model (Meta-Llama-3.1-8B)
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B")

# Load the fine-tuned model with PEFT
model_name = "Sameer2407/PriceLLaMAA-2025-04-28_07.20.50"  # Replace with your model path
model = PeftModel.from_pretrained(base_model, model_name)

# Load the tokenizer
tokenizer = base_model.get_tokenizer()

# Define a product description
product_description = "A sleek, modern stainless steel electric kettle with 1.5-liter capacity and auto shut-off feature."

# Prepare input
inputs = tokenizer(f"Predict the price: {product_description}", return_tensors="pt").to(model.device)

# Generate output
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

# Decode and print the predicted price
predicted_price = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(predicted_price)