[Remote] Staff AI Product Analyst

Note: The job is a remote job and is open to candidates in USA. Angi is a global company that has been transforming the home services industry for over 30 years. The Staff AI Product Analyst will be responsible for defining AI-native product analytics, creating evaluation frameworks, and ensuring alignment between human teams and AI agents in the product analytics function.

Responsibilities

• Flagship Journey Measurement: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes
• Causal Framing & Experimentation: Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship
• Data Layer Systems Thinking: Map how agents interact with metrics, raw tables, documentation, and APIs to ensure requirements for data and analytics engineering are explicit, structured, and testable
• Eval Framework Architecture: Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness
• Taxonomy & Failure Mode Analysis: Systematically categorize wrong answers, tool misuse, context gaps, and data edge cases, feeding those insights back into prompts, retrieval (RAG), tools, and data modeling pipelines
• Performance Monitoring: Monitor live usage of Agentic AI analytics workflows—tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog
• Playbooks & Narrative: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Support enablement and leadership storytelling around agentic AI analytics
• Strategic Collaboration: Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomes
• Defining Success: Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metrics

Skills

• 10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment
• 1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsibly
• Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices
• Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline
• Exceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholders
• Experience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systems
• Demonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iteration
• A systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business value

Benefits

• This position will be eligible for a competitive year end performance bonus & equity package.
• Full medical, dental, vision package to fit your needs
• Flexible vacation policy; work hard and take time when you need it
• Pet discount plans & retirement plan with company match (401K)

Company Overview

Company H1B Sponsorship