Software Engineer Observability Instrumentation

Software Engineer Observability Instrumentation Overview

This role sits within the Observability Engineering team, which is responsible for giving engineering groups access to the firm’s telemetry platforms so they can produce and consume telemetry effectively for their services. The team is looking for a technically capable, service-oriented software engineer who can make observability easier to adopt across the business. The emphasis is on the producer side of telemetry: building strong instrumentation patterns, working with OpenTelemetry SDKs, and shaping collector configurations that help teams emit telemetry that is consistent, reliable, and useful.

The position is a good fit for someone who enjoys creating broad engineering leverage through shared patterns, libraries, and guidance rather than working on a single product feature. You will use your software engineering experience, together with developing AI engineering skills, to improve how teams instrument, understand, and run their services across the firm. A major part of the job is to improve service visibility and to work closely with application and infrastructure teams so that instrumentation paths are straightforward and repeatable for C#, Python, and Kubernetes-based services.

What you will do

• Develop, extend, and maintain OpenTelemetry SDKs, supporting libraries, and collector components used across the organisation.
• Create and support instrumentation approaches for both Kubernetes workloads and services running outside Kubernetes.
• Help define, promote, and embed observability standards across platform engineering and application engineering teams.
• Use AI and AI-assisted development methods to improve productivity and increase the impact of the team’s work.
• Scale and refine telemetry backend systems and the pipelines that move observability data through the platform.
• Improve incident response by increasing the quality and coverage of telemetry available to engineers.
• Work with teams to establish practical golden paths for instrumentation across C#, Python, and Kubernetes-based services.
• Take part in the out-of-hours on-call rota and support operational needs when required.

What we are looking for

• Strong problem-solving ability and the judgement to turn engineering standards into usable libraries, examples, and guidance for other teams.
• Programming experience in C# and/or Python, plus a solid understanding of software architecture.
• Practical, hands-on experience with OpenTelemetry SDKs, instrumentation, and collectors.
• Experience working with Kubernetes and cloud-native observability practices.
• A good understanding of metrics, logs, and traces, and how engineering teams use them to debug and operate services.
• Experience with DevOps and delivery tooling such as Terraform, ArgoCD, Helm, or Jenkins.
• Interest in AI engineering and SRE practices, especially where they can improve incident response and root-cause analysis.
• Comfort working in a role that creates shared engineering leverage rather than building a single isolated feature set.

Nice-to-have experience

• Exposure to observability backends such as Prometheus/PromQL, VictoriaMetrics, OpenSearch, Grafana, or similar tools.
• Experience with auto-instrumentation, distributed tracing, structured logging, or correlating traces and metrics.
• Knowledge of Kafka or telemetry pipeline architectures.

Why join

The company offers a highly competitive compensation package with an annual discretionary bonus. Employees receive lunch through Just Eat for Business, access to a dedicated barista bar, 35 days of annual leave, and 9% company pension contributions. The workplace is described as informal, with an excellent work/life balance, and the benefits package also includes comprehensive healthcare, life assurance, a cycle-to-work scheme, and monthly company events.

Beyond the core package, the wider benefits offering includes a company pension scheme, a discretionary bonus scheme, season ticket loans, a give-as-you-earn option, risk protection benefits, and charity fundraising matching. The company also provides generous relocation and immigration assistance. Health-related support includes private health insurance with GP access, dental and vision cover, enhanced support for a range of health and family-forming needs, a healthcare cash plan, and employee assistance and wellbeing programmes. Lifestyle benefits include annual leave, enhanced family leave policies, back-up dependent care, travel insurance for employees and their families, cycle schemes, gym and fitness subsidies, free lunch, a barista bar, regular social events, an informal dress code, and talks from guest speakers.

Working environment

The role is based in London and is part of a collaborative engineering organisation working from the company’s London headquarters. The wider culture is described as dynamic, flexible, and highly stimulating, with an emphasis on deep exploration, methodical execution, long-term thinking, and building systems that support high-impact research and engineering work.

How the hiring process works

• Submit an online application, which is reviewed by the Talent Acquisition team.
• Attend an initial technical interview, or in some cases complete a remote technical test.
• Take part in a behavioural interview focused on you as a person rather than only on technical ability.
• Move on to additional technical interviews that explore your skills and competencies in more depth.
• Finish with management interviews, where you meet team members, your future manager, and functional leadership.

Additional context

The broader engineering organisation focuses on building robust infrastructure and high-performance platforms that turn research ideas into practical results. The company also highlights its collaborative culture, its work with open-source software, and its commitment to supporting local communities and STEM talent. Employee testimonials on the page emphasise learning from colleagues, working with complex systems, and being part of a supportive environment where people can grow technically and professionally.

Degree Requirement: Not Specified

Visa Sponsorship May be