Annotations QA Analyst
Content Platform
Link copied to clipboard.
Apply now
Apply
Permanent
Dublin or London
To power the best audio network in the world by enabling creators and content distributors to deliver their content frictionlessly and equip Spotify teams with the richest possible catalog. That’s what the Content Platform team is all about. We use our deep understanding of consumer expectations to enrich the lives of millions of our users worldwide, bringing the music and audio they love to the devices, apps and platforms they use every day.
The Annotation Platform Ops team works alongside Product and Engineering teams at the heart of the Content Platform R&D studio to develop rich and interconnected datasets to enable delightful consumer experiences through a fusion of machine and human expertise. We focus on use cases across all of Spotify’s verticals - Music, Podcasts, and Audiobooks, with an emphasis on the safety and moderation of our catalogs in accordance with Spotify policies.
We are seeking an Annotation QA Analyst to contribute their human expert knowledge to the wide collection of labeled datasets that Content Platform teams use to train, evaluate, and better understand machine learning models and systems. This is an exciting opportunity to work at the center of ML and AI-driven development.
What You'll Do
- Proactively review and ensure annotated data is in line with Spotify’s standards and policies, including running quality reviews prioritized by business needs.
- Ensure high-quality and timely results for R&D collaborator teams (Product and Engineering) via established quality framework / quality assurance processes, using metrics such as agreement rates and consensus.
- Run QA expert reviews on the most difficult edge cases in projects in order to address and resolve ambiguity.
- Surface themes, suggestions, and feedback for the annotation team per project. Communicate complex concepts and findings in a clear and effective manner to non-technical and technical audiences.
- Contribute to feedback loops between annotation teams, R&D collaborator teams (Product and Engineering), content policy experts and/or Trust & Safety policy experts, in order to continuously iterate on data analysis workflows.
- Collaborate with R&D collaborator teams (Product and Engineering) to ensure proper project guidelines exist per use case prior to annotator training (e.g. ensuring proper in scope vs out of scope examples are generated). If necessary, update or revise project guidelines based on results from initial rounds of annotations.
- Proactive collaborate with other members of the Annotation Platform Ops team across all in-flight projects.
Who You Are
- Well-versed in annotation, data analysis, quality assurance, and human-in-the-loop best practices.
- Familiar with AI / LLM-driven workflows.
- Comfortable with large-scale review and contributing to very large datasets across multi-modal content types (e.g., text, audio, images, video).
- Passionate about catalog safety and moderation (e.g., Trust & Safety) and a desire to apply QA frameworks across diverse domains.
- Self-starter who can collaborate with cross-functional partners in a dynamic environment and deliver results.
- Excellent verbal and written communication skills.
- Experience working outside of the safety and moderation space is a plus.
- Good grasp of machine learning lifecycles from goal setting to data collection/labeling to model development and deployment is a plus.
- Basic knowledge of SQL is a plus.
Where You'll Be
- This role is based in Dublin or London.
- We offer you the flexibility to work where you work best! There will be some in-person meetings, but still allows for flexibility to work from home.
Learn about life at Spotify
Spotify is an equal opportunity employer. You are welcome at Spotify for who you are, no matter where you come from, what you look like, or what’s playing in your headphones. Our platform is for everyone, and so is our workplace. The more voices we have represented and amplified in our business, the more we will all thrive, contribute, and be forward-thinking! So bring us your personal experience, your perspectives, and your background. It’s in our differences that we will find the power to keep revolutionizing the way the world listens.
At Spotify, we are passionate about inclusivity and making sure our entire recruitment process is accessible to everyone. We have ways to request reasonable accommodations during the interview process and help assist in what you need. If you need accommodations at any stage of the application or interview process, please let us know - we’re here to support you in any way we can.
Spotify transformed music listening forever when we launched in 2008. Our mission is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art and billions of fans the chance to enjoy and be passionate about these creators. Everything we do is driven by our love for music and podcasting. Today, we are the world’s most popular audio streaming subscription service.
Our global benefits
Extensive learning opportunities, through our dedicated team, GreenHouse.
Flexible share incentives letting you choose how you share in our success.
Global parental leave, six months off - for all new parents.
All The Feels, our employee assistance program and self-care hub.
Flexible public holidays, swap days off according to your values and beliefs.