PrimeIntellect/search-data
Viewer • Updated • 16.4k • 31
AI & ML interests
Commoditizing intelligence through a decentralized AI lab to advance scientific progress and human flourishing.
Recent Activity
View all activity
Fareso
updated a
dataset 5 days ago
Fareso
published a
dataset 5 days ago
rasdani
published a
dataset 23 days ago
milkkarten
authored a
paper about 1 month ago
milkkarten
submitted a
paper to Daily Papers about 1 month ago
milkkarten
authored a
paper about 2 months ago
milkkarten
submitted a
paper to Daily Papers about 2 months ago
milkkarten
authored a
paper about 2 months ago
milkkarten
submitted a
paper to Daily Papers about 2 months ago
milkkarten
authored a
paper 3 months ago
milkkarten
submitted a
paper to Daily Papers 3 months ago
milkkarten
authored a
paper 4 months ago
milkkarten
submitted a
paper to Daily Papers 4 months ago
siro1
authored a
paper 4 months ago
milkkarten
authored a
paper 4 months ago
Johannes
authored a
paper 6 months ago
eliebak
submitted a
paper to Daily Papers 6 months ago
Post
4547
Super excited to announce that our research team at Hugging Face will be doing an AMA on reddit r/LocalLLaMA.
Come ask any questions to the team behind SmolLM, FineWeb and more! And who knows, maybe there’ll be a shiny new release to talk about?
Thursday 4th September, 8AM-11AM PST 🤗
science
Come ask any questions to the team behind SmolLM, FineWeb and more! And who knows, maybe there’ll be a shiny new release to talk about?
Thursday 4th September, 8AM-11AM PST 🤗
Post
776
Motif 2.6B tech report is pretty insane, first time i see a model with differential attention and polynorm trained at scale!
> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.
Motif-Technologies/Motif-2.6B
> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.
Motif-Technologies/Motif-2.6B
milkkarten
authored a
paper 11 months ago