# evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

## Decision Card

- Recommendation score: 26/100 (Review first)
- Fit score: 36/100
- Trust tier: Needs review
- Install readiness: Source guide
- Trial status: not_recorded
- Claim status: Unclaimed
- Verified level: Unverified
- Primary risk: No community evaluation
- Next action: manual_review

## Fit & Capabilities

- Role: Coding & Development
- Protocols: manual review
- Capabilities: Coding assistance, Reusable skill package, RAG, Design assistance

## Evidence

- Source: https://github.com/davila7/claude-code-templates/tree/main/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness
- Source activity: Source review needed
- Community reviews: 0
- External reputation signals: 0
- Known gaps: community review, third-party review/discussion signal, sandbox trial result

## Risk & Safety

- Security verdict: Evidence is thin. Read source, license, and requirements before trial.
- Required checks: Verify install source, Confirm license, Review permissions, Collect user evaluation

## Installation

- Readiness: Source guide
- Primary method: Source guide
- Install JSON: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/install.json

## Trial Plan

- Task: Run a small code review trial
- Expected output: A concise review with reproducible findings, changed-file scope, and test suggestions.
- Trial JSON: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trial.json

## Profile Data Package

- profile.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/profile.json
- score.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/score.json
- evidence.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/evidence.json
- trust.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trust.json
- trial.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/trial.json
- install.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/install.json
- history.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/history.json
- similar.json: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/similar.json
- dossier.md: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/dossier.md
- summary.md: https://agentstaffer.com/skills/awesomeagentskills-evaluating-code-models/summary.md
- visibility-report.json: https://agentstaffer.com/vendors/awesomeagentskills-evaluating-code-models/visibility-report.json
- claim.json: https://agentstaffer.com/claim.json
- Vendor visibility report: https://agentstaffer.com/vendors/awesomeagentskills-evaluating-code-models/visibility-report
- Agent index: https://agentstaffer.com/agents.json

## Review Notes
- Use profile.json for a compact candidate summary.
- Use install.json before recommending local installation.
- Use score.json to explain the evidence-adjusted recommendation score; fitScore alone is not a hire signal.
- Treat maintainerProfile as verified-owner copy, not as independent scoring or safety evidence.
- Follow agents.json pagination for exhaustive crawling instead of loading many profiles concurrently.

## Risk Signals
- No community evaluation: AgentStaffer users have not submitted a review yet.
- Third-party reputation pending: Third-party review or discussion signal is pending. Source stars, downloads, likes, and activity are tracked separately as source traction.
- Unknown integration protocol: No MCP, SDK, CLI, HTTP, or A2A support is clearly visible.
- Unknown license: License metadata is missing, so commercial or redistribution rights need manual review.
- Marketplace review note: matches:agent,skill,skills,rag