Intelligence Analyst
micro1 · 100% remote · Contract · Posted
- Pay
- $80–170/hr
- Location
- Worldwide
- Languages
- English
- Hours
- Flexible
- Openings
- 10
- Level
- Expert
Summary
Author intelligence test cases, grade AI responses on analytic rigor and source handling, and document edge cases and disagreements.
How the pay compares
At the middle of its range, this job pays $125 an hour.
- Generalist jobs: median $60 an hour across 273 open hourly jobs. This job pays $65 more.
- micro1 jobs: median $70 an hour across 346 open hourly jobs. This job pays $55 more.
What you'll do
- Develop and map analytic cases to defined taxonomies
- Author complex test cases for evaluating model performance on source handling, analytic rigor, and uncertainty expression
- Assess and grade AI-generated responses based on reasoning, source use, recognition of limitations, and practical utility
- Identify and flag instances where legitimate analytic exploration is over-refused by the model
- Document edge cases and analytic disagreements
- Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment
- Contribute insights to calibrate analytic standards and enhance AI system benchmarks
Requirements
- All-source analytic experience or deep expertise in a specific analytic discipline
- Strong proficiency in source evaluation, confidence-level assessment, and analytic standards
- Experience in teaching, mentoring, or leading analytic reviews
- Exceptional written and verbal communication skills
- Ability to identify and articulate nuanced model limitations and over-refusal scenarios
- Familiarity with structured analytic techniques and intelligence tradecraft standards
Skills
- Analytic Rigor
- Source Handling
- Over-Refusal Detection
- All-Source Intelligence Analysis
Full description
Role Title: Intelligence Analyst
Role Type: Contractor
Location: Remote
micro1 is engaging Intelligence Analysts to contribute to a customer-facing project focused on evaluating and advancing artificial intelligence capabilities. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.
Scope of Work
- Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios.
- Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression.
- Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility.
- Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback.
- Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies.
- Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
- Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks.
Preferred Qualifications
- Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background.
- Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards.
- Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor.
- Exceptional written and verbal communication skills, with a focus on clarity and precision.
- Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
- Familiarity with structured analytic techniques and intelligence tradecraft standards.
- Previous participation in AI evaluation or model assessment initiatives is a plus.
Learn More About Our Government & Defense Work
At micro1, we work with government and defense experts to evaluate how advanced AI models perform on real-world tasks.
Our MissionReport benchmark assesses AI-generated military and government reports, highlighting the importance of expert judgment, accuracy, and operational understanding.
Learn more: https://www.micro1.ai/benchmark/mission-report
Similar jobs
Special Forces OperatorTactical Operations ExpertiseMission PlanningMilitary Tactics and ProceduresOperational Realism AssessmentRules of Engagement (Roe) KnowledgeLaw of Armed Conflict (Loac) Knowledge+8$80–170/hr
🌍 Worldwide
Human Intelligence Collector (HUMINT)Elicitation StrategyReal-Time Cue ReadingReliability AssessmentInterrogation/Investigative QuestioningHUMINT+4$80–170/hr
🌍 Worldwide
Architecture and Engineering Leaders: Project Management Software InterviewProject Management SoftwareTime TrackingClient BillingSoftware EvaluationBudget ManagementFinancial Strategy+5$175
🌍 Worldwide
Product Managers: 1-on-1 Interview on Software Strategy and RoadmapsProduct StrategyRoadmap PlanningFeature PrioritizationEngineering Cycles+3$175
🌍 Worldwide