Note: The job is a remote job and is open to candidates in USA. Devoted Health is on a mission to improve the health and well-being of older Americans by providing integrated healthcare solutions. The Senior Manager, AI Performance & Operations will oversee the performance and continuous improvement of AI workflows, ensuring they meet operational outcomes and quality standards.
Responsibilities
- Post-Launch Accountability: Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met
- Portfolio Oversight: Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions
- Performance & Dashboards: Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently
- Diagnosis & Incident Response: Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure
- Continuous Improvement & Optimization: Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates. Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changes
- Scalable Evaluation: Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production
- Expert Review: Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models
- Executive Communication: Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it
Skills
- Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems
- Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows
- A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic
- Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage
- Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments
- Deep personal accountability with a drive to own results rather than just producing recommendations
- Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions
Benefits
- Employer sponsored health, dental and vision plan with low or no premium
- Generous paid time off
- $100 monthly mobile or internet stipend
- Stock options for all employees
- Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
- Parental leave program
- 401K program
- Our total rewards package is for full time employees only. Intern and Contract positions are not eligible.
Company Overview
Devoted Health is a healthcare company serving seniors and giving them a health care plan with personal guides and world-class technology. It was founded in 2017, and is headquartered in Waltham, Massachusetts, USA, with a workforce of 1001-5000 employees. Its website is https://devoted.com.Company H1B Sponsorship
Devoted Health has a track record of offering H1B sponsorships, with 1 in 2021. Please note that this does not guarantee sponsorship for this specific role.