Location: Hybrid- NYC (3+ days in office required)
Type: Full-time W-2 employee, subject to a 90-day introductory period\ \ Salary and equity commensurate with experience and performance\ \ About the Role
Aviary AI is looking for a Software Engineer focused on QA and test automation (90-day probationary period) to own quality across our voice agent and software deployments. You’ll sit at the edge of our deployment cycle with real authority over what ships — starting hands-on with manual QA, then building the tooling that automates it, and ultimately owning the systems that keep our agents reliable as we scale.
The Objective
Own quality for our voice agent and software deployments, and automate as much of that QA as possible. You start by manually QA’ing deployments using our existing tooling, then build the systems that make most of that manual work disappear. Once those systems exist, you own and run them.
The Arc of the Role
This role has a clear progression:
- Manual QA (now). Run quality checks on voice agent deployments using our in-platform QA suite in Aviary Web (chat, voice, and adversarial testing) plus standard testing for our software releases. Get deep on where deployments break and what “ready to ship” actually means.
- Build the automation. Turn that manual process into tooling. Extend our adversarial testing framework (Blue-Jay) and our automated campaign-review agent (Coach) so the bulk of QA runs without a human in the loop.
- Own the systems. Once the automation is live, your job becomes managing and evolving these codebases — keeping Blue-Jay, Coach, and the Aviary Web QA suite sharp as our deployment volume grows.
Responsibilities
- Own the quality bar for voice agent and software deployments — you decide what ships, what waits, and what needs more work.
- Design and maintain automated test and evaluation suites across our voice agent stack and standard software pipelines.
- Build tooling that replaces manual QA, focusing on automation, scalability, and reliable release gating.
- Take part in the design and architecture of testing and evaluation infrastructure across the product.
- Help shape team culture, internal tools, and the processes around how we ship.
What You’ll Work With
- Aviary Web QA suite — the in-platform testing tools (React/Next.js, TypeScript) covering chat, voice, and adversarial testing of agent configs, with automated grading against success definitions.
- Blue-Jay — our adversarial testing framework (Python) that generates customer personas and runs text and phone battles against agents to probe edge cases and failure modes.
- Coach — our run-to-completion agent (Python) that reviews campaigns, analyzes flagged calls, proposes config changes, and validates them through Blue-Jay before reporting to human reviewers.
Qualifications
- BA/BS in Computer Science, Engineering, or a related field.
- 1–3 years of software development experience (including internships); strong new grads will be considered.
- Proficiency in React/Next.js, with experience building responsive, scalable web applications.
- Strong general programming ability — Python and TypeScript both matter, since our stack spans both.
- Familiarity with AWS services (e.g. EC2, S3, Lambda, ECS) and cloud architecture for deploying scalable solutions.
- Familiarity with relational databases, particularly PostgreSQL and SQL Server, including writing and optimizing queries.
- Excellent problem-solving skills and the ability to work independently in a fast-paced startup environment.
Nice to Haves
Experience with test automation, CI, or evaluation frameworks. Work in natural language processing, machine learning, recommendation systems, or large-scale production search/ranking systems. Prior experience with LLM or voice agent evaluation and adversarial testing. UX/UI experience a plus (i.e. Figma). \ \ What We Offer
- Equity
- Hybrid work - 3+ days in our NYC office
- Health, dental, and vision coverage
- Unlimited PTO
- In-office meals stipend
- Learning & development budget