Lambent/Mira-v1.24-27B-swcm

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SWCM (Signal‑Weighted Consensus Merge) merge method using Lambent/Mira-v1.23.1-27B-dpo as a base.

Models Merged

The following models were included in the merge:

• Lambent/Mira-v1.23.1-27B-dpo + ../Mira-v1.24-27B-adapters/sft3/
• Lambent/Mira-v1.23.1-27B-dpo + ../Mira-v1.24-27B-adapters/sft2/
• Lambent/Mira-v1.23.1-27B-dpo + ../Mira-v1.24-27B-adapters/sft1/

Configuration

The following YAML configuration was used to produce this model:

merge_method: swcm
base_model: Lambent/Mira-v1.23.1-27B-dpo

models:
  # The Base Model (The Anchor)
  # Essential for SWCM to calculate the 'delta' (Adapter - Base) correctly.
  - model: Lambent/Mira-v1.23.1-27B-dpo

  # SFT 1 (Run A)
  - model: Lambent/Mira-v1.23.1-27B-dpo+../Mira-v1.24-27B-adapters/sft1/
    parameters:
      weight: 1.0

  # SFT 2 (Run B - attention only)
  - model: Lambent/Mira-v1.23.1-27B-dpo+../Mira-v1.24-27B-adapters/sft2/
    parameters:
      weight: 1.0

  # SFT 3 (Run C - mlp only)
  - model: Lambent/Mira-v1.23.1-27B-dpo+../Mira-v1.24-27B-adapters/sft3/
    parameters:
      weight: 1.0

parameters:
  # Signal-Weighted Parameters
  density_strength: 1.5   # Boosts the faint signal from your 5e-7 runs
  regularization: 0.0     # CRITICAL: 0.0 ensures we don't decay the tiny updates
  max_iter: 10            # Enough iterations to resolve the consensus
  tol: 1e-4

dtype: bfloat16
tokenizer_source: base