Data Scientist

Ellipsis Health is creating cutting-edge AI/ML products that solve healthcare staffing issues and administrative burdens using conversational AI and our patented voice biomarker technology in the delivery of better healthcare for everyone. We are headquartered in Silicon Valley and are funded and supported by some of the most preeminent venture capital teams.

As a Data Scientist specializing in AI Quality & Evaluation, you will play a pivotal role in ensuring the robustness, accuracy, and fairness of our AI models and systems. You will be instrumental in developing comprehensive evaluation frameworks, creating diverse and high-quality datasets, and implementing metrics to drive continuous improvement across our AI initiatives. Your work will directly contribute to accelerating customer onboarding and enhancing the overall user experience.

Ellipsis Health is located in the San Francisco Bay Area, but we are open to remote candidates for this role, as long as they are located in the U.S.

Responsibilities

• Drive High-Impact Projects: Own data science and AI evaluation projects end-to-end—from defining the problem and developing the methodology to executing the analysis, communicating results, and driving implementation.

• Drive Continuous Improvement of LLM-as-a-Judge: Build the data and evaluation infrastructure for a closed-loop AI quality system, including golden datasets, ground-truth benchmarks, judge calibration, error analysis, and production feedback. Continuously identify failure modes and translate insights into improvements to datasets, evaluation methodologies, LLM-as-a-Judge systems, and the underlying AI products.

• Develop Evaluation Frameworks: Design and implement robust evaluation frameworks, methodologies, and metrics for LLM- and speech-based AI systems. Develop scalable approaches that help us measure quality and accelerate customer onboarding.

• Build and Manage Evaluation Datasets: Design, curate, and maintain high-quality datasets for training, testing, benchmarking, and evaluating conversational AI systems and their components.

• Create and Manage Data Pipelines and Dashboards: Build and maintain scalable data pipelines and intuitive dashboards that enable reliable data analysis,ongoing AI quality monitoring, and data-driven decision-making across the organization.

• Collaborate and Innovate: Work closely with AI/ML Engineers and Product Managers to integrate your findings into product enhancements. You'll stay current with the latest advancements in AI evaluation, particularly in Large Language Models (LLMs) and speech systems, contributing to our overall technical strategy.