Model Overview
vortex-3b is a 2.78 billion parameter causal language model created by OEvortex that is derived from EleutherAI's Pythia-2.8b and fine-tuned on Vortex-50k dataset'
from transformers import pipeline
# Initialize the pipeline
pipe = pipeline("text-generation", model="OEvortex/vortex-3b")
# Use the pipeline
text = "Once upon a time"
generated_text = pipe(text, max_length=100, do_sample=True)[0]['generated_text']
print(generated_text)
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | vortex 3b | vortex 3b-v2 | dolly-v2-3b | pythia-2.8b-deduped |
|---|---|---|---|---|
| Avg. | 35.76 | 37.46 | 25.26 | 36.72 |
| AI2 Reasoning Challenge (25-Shot) | 31.91 | 39.68 | 22.83 | 36.26 |
| HellaSwag (10-Shot) | 56.89 | 65.04 | 26.55 | 60.66 |
| MMLU (5-Shot) | 27.32 | 25.09 | 24.7 | 26.78 |
| TruthfulQA (0-shot) | 37.39 | 33.80 | 0 | 35.56 |
| Winogrande (5-shot) | 60.14 | 59.12 | 59.43 | 60.22 |
| GSM8k (5-shot) | 0.91 | 2.05 | 1.86 | 0.83 |
- Downloads last month
- 910
Model tree for OEvortex/vortex-3b
Dataset used to train OEvortex/vortex-3b
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard31.910
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard56.890
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard27.320
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard37.390
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard60.140
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard0.910