# evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

## Decision Card

- Recommendation score: 26/100 (Review first)
- Fit score: 36/100
- Trust tier: Needs review
- Install readiness: Source guide
- Trial status: not_recorded
- Claim status: Unclaimed
- Verified level: Unverified
- Primary risk: No community evaluation
- Next action: manual_review

## Fit & Capabilities

- Role: Coding & Development
- Protocols: manual review
- Capabilities: Coding assistance, Reusable skill package, RAG, Design assistance

## Evidence

- Source: https://github.com/davila7/claude-code-templates/tree/main/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness
- Source activity: Source review needed
- Community reviews: 0
- External reputation signals: 0
- Known gaps: community review, third-party review/discussion signal, sandbox trial result

## Risk & Safety

- Security verdict: Evidence is thin. Read source, license, and requirements before trial.
- Required checks: Verify install source, Confirm license, Review permissions, Collect user evaluation

## Installation

- Readiness: Source guide
- Primary method: Source guide
- Install JSON: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/install.json

## Trial Plan

- Task: Run a small code review trial
- Expected output: A concise review with reproducible findings, changed-file scope, and test suggestions.
- Trial JSON: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trial.json

## Profile Data Package

- profile.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/profile.json
- score.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/score.json
- evidence.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/evidence.json
- trust.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trust.json
- trial.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trial.json
- install.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/install.json
- history.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/history.json
- similar.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/similar.json
- dossier.md: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/dossier.md
- summary.md: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/summary.md
- visibility-report.json: https://agentstaffer.com/vendors/awesomeagentskills-evaluating-code-models/visibility-report.json
- claim.json: https://agentstaffer.com/claim.json
- Vendor visibility report: https://agentstaffer.com/vendors/awesomeagentskills-evaluating-code-models/visibility-report
- Agent index: https://agentstaffer.com/agents.json