Hugging Face's logo

Shuu12121
/
CodeSearch-codebert-base-30k-1e5

Sentence Similarity
sentence-transformers
Safetensors
roberta
feature-extraction
dense
Generated from Trainer
dataset_size:180000
loss:MultipleNegativesRankingLoss
text-embeddings-inference
Model card Files
xet
Community

SentenceTransformer based on microsoft/codebert-base

This is a sentence-transformers model finetuned from microsoft/codebert-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: microsoft/codebert-base
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Parses an Edm.Date value and returns the corresponding JavaScript Date value.\n\n@param {string} sValue\nthe Edm.Date value to parse\n@returns {Date}\nthe JavaScript Date value or <code>null</code> in case the input could not be parsed',
    'function (sValue) {\n\t\t\treturn DateFormat.getDateInstance({\n\t\t\t\t\tpattern : "yyyy-MM-dd",\n\t\t\t\t\tstrictParsing : true,\n\t\t\t\t\tUTC : true\n\t\t\t\t}).parse(sValue);\n\t\t}',
    'func (r *Request) Build() {\n\tif !r.built {\n\t\tr.Handlers.Build.Run(r)\n\t\tr.built = true\n\t\tif r.Error != nil {\n\t\t\treturn\n\t\t}\n\t\tr.Handlers.Sign.Run(r)\n\t}\n}',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.9022, -0.0791],
#         [ 0.9022,  1.0000, -0.0990],
#         [-0.0791, -0.0990,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

  • Size: 180,000 training samples
  • Columns: sentence_0, sentence_1, and label
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 label
    type string string float
    details
    • min: 3 tokens
    • mean: 66.78 tokens
    • max: 512 tokens
    • min: 31 tokens
    • mean: 206.6 tokens
    • max: 512 tokens
    • min: 1.0
    • mean: 1.0
    • max: 1.0
  • Samples:
    sentence_0 sentence_1 label
    // NewFeature is the normal factory method for a feature
    // Note that id is expected to be a string or number    		 func NewFeature(geometry interface{}, id interface{}, properties map[string]interface{}) *Feature {
    if properties == nil {
    properties = make(map[string]interface{})
    }
    return &Feature{Type: FEATURE, Geometry: geometry, Properties: properties, ID: id}
    }    		 1.0
    // Endpoints returns the endpoints used by this connection. func (c *agencyConnection) Endpoints() []string {
    c.mutex.RLock()
    defer c.mutex.RUnlock()

    var result []string
    for _, x := range c.connections {
    result = append(result, x.Endpoints()...)
    }
    return result
    }    		 1.0
    // Build validates the configuration options provided then builds the command func (builder *MapReduceCommandBuilder) Build() (Command, error) {
    if builder.protobuf == nil {
    panic("builder.protobuf must not be nil")
    }
    if builder.streaming && builder.callback == nil {
    return nil, newClientError("MapReduceCommand requires a callback when streaming.", nil)
    }
    return &MapReduceCommand{
    protobuf: builder.protobuf,
    streaming: builder.streaming,
    callback: builder.callback,
    }, nil
    }    		 1.0
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 40
  • per_device_eval_batch_size: 40
  • num_train_epochs: 1
  • fp16: True
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 40
  • per_device_eval_batch_size: 40
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • project: huggingface
  • trackio_space_id: trackio
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: no
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: True
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss
0.1111 500 1.1925
0.2222 1000 0.1974
0.3333 1500 0.1708
0.4444 2000 0.1574
0.5556 2500 0.1466
0.6667 3000 0.1385
0.7778 3500 0.1344
0.8889 4000 0.1347
1.0 4500 0.135

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 5.2.0
  • Transformers: 4.57.3
  • PyTorch: 2.8.0+cu128
  • Accelerate: 1.12.0
  • Datasets: 3.6.0
  • Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
12
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Shuu12121/CodeSearch-codebert-base-30k-1e5

Base model

microsoft/codebert-base
Finetuned
(131)
this model

Papers for Shuu12121/CodeSearch-codebert-base-30k-1e5