Site Reliability Engineer

About the Role

iCapital is looking to hire a Site Reliability Engineer to join the Site Reliability Engineering team, which plays a critical role in ensuring the platform delivers consistent, reliable service to clients. The services run on Kubernetes in AWS, and this role is focused on maintaining the health, stability, and performance of production environments.

This position is ideal for a hands-on engineer who thrives in diagnosing complex production issues where the root cause is not immediately apparent. This role transforms insights gained from production incidents into stronger reliability standards, safer deployment practices, more effective alerting, and automation that helps prevent recurring issues. While metrics, logs, and traces are leveraged daily, the core focus is identifying, resolving, and learning from the issues they reveal.

Responsibilities

• Act as a senior escalation point for complex production issues in Kubernetes-based services, leading hands-on investigation across application, container, Kubernetes, and AWS layers through to root cause.
• Partner with the teams that own our cluster infrastructure to resolve platform-level issues, and feed what you learn back into reliability standards and improvements.
• Define and implement reliability and operability standards for Kubernetes-based services, including resource sizing, health checks, scaling patterns, rollout safety, and baseline monitoring, as part of service onboarding.
• Serve as the incident commander or senior technical lead for high-severity incidents, lead postmortems, and drive systemic improvements through action items and measurable follow-through.
• Turn recurring investigations into runbooks, diagnostic tooling, and automated remediation that shorten time to resolution and eliminate toil.
• Coach engineers in systematic troubleshooting and Kubernetes best practices through reviews, game days, and knowledge sharing.
• Define, implement, and iterate service level objectives (SLOs) and service level indicators (SLIs) that reflect customer and business expectations.
• Standardize monitoring and alerting through “monitors as code” (Terraform preferred), including quality gates such as severity, ownership, and runbook links.
• Operate and tune OpenTelemetry pipelines, including sanitizing data, managing cardinality, and enriching telemetry with consistent fields.
• Maintain the infrastructure behind our monitoring stack, including Helm charts, ArgoCD ApplicationSets, Kinesis streams, Lambda functions, and Prometheus.
• Participate in on-call rotations with a focus on improving reliability, reducing alert noise, and increasing signal quality over time.