Edtech.com's Summary

Crimson Education is hiring a Senior Software QA Engineer to design, develop, and scale end-to-end automation test frameworks for AI-powered applications, ensuring reliable and safe admissions technology. The role involves integrating automated tests into CI/CD pipelines, monitoring AI and ML model performance, and simulating adversarial inputs to maintain robust AI features.

Highlights

Design and build automation test frameworks covering UI, API, and integration layers.
Integrate automated tests within CI/CD pipelines to support rapid deployments.
Define and track test coverage metrics to maintain regression-free software releases.
Conduct API performance, load, and stress testing.
Develop AI/LLM validation strategies focusing on accuracy, bias, safety, and hallucination rates.
Establish prompt regression testing and monitor data/model drift in collaboration with data scientists and ML engineers.
Simulate adversarial inputs to ensure AI feature security and guardrails.
Require 4+ years in Software QA Automation, with 1-2 years testing AI/ML-powered applications.
Proficient in Python, JavaScript/TypeScript, and automation tools such as Playwright, Cypress, Selenium, Postman, RestAssured, and PyTest.
Bachelor’s degree in Computer Science, Engineering, or equivalent experience; familiarity with AI evaluation frameworks and cloud/container technologies is a plus.

Senior Software QA Engineer Full Description

Senior Software QA Engineer

Senior Software QA Engineer at Crimson Education: design and scale end-to-end quality automation for AI-powered apps, own test coverage, and ensure safe, reliable admissions tech.

Apply for this job

Crimson Education is the world’s leading college admissions consulting firm, with over 1,490 Ivy League offers and 2,410 to the US Top 15. We help ambitious students gain admission to the world’s top universities through expert-led guidance and proven, data-driven strategies. Crimson students are 7x more likely to get into the Ivy League than their peers. We were recently featured on the front page of the Wall Street Journal.

Crimson is the only college admissions consultancy that brings together:

Former Ivy League and Top 20 admissions officers to rigorously review and refine applications
Professors and PhD teaching fellows from leading universities to guide students through original, independent research — with pathways to selective peer-reviewed publication or conference presentation
Past ISEF winners and judges who coach students to compete for state, national, and international science fair awards
Capstone project mentors who help students design and scale leadership initiatives with real-world impact, measurable outcomes, and credible external validation

We’re backed by leading VC firms, including Tiger Global, Heal Partners, IceHouse Ventures, and Movac, and recently closed a USD $40M Series D funding round at a USD $640M valuation. We now operate across 21 markets worldwide, including the US, Canada, UK, Singapore, Japan, Hong Kong, Australia, and New Zealand.

The Role

What are the main responsibilities for this role?

Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).
CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.
Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.
Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.
AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.
Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.
Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.
Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.

What skills and experience are required?

4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.
Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.
Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).
- Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).
Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.
Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.
Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).
A healthy obsession with AI safety and ethics.

Why work for Crimson?

Rapidly growing start up, with a flexible working environment where you will be empowered to structure how you work
Limitless development and exposure - our internal promotions/role changes made up 33% of all recruitment last year
$1000 individual training budget per year, we love to ‘Level Up’ (it’s one of our core values)!
Psychologist on staff
Insightful fireside chats and workshops to help support our high-performing and ambitious team
Radical Candor is a feedback approach we live by
We’re a global player with 28 markets (and growing) across the globe. Most roles have the option to work from one of our many offices or remotely!

If you're passionate about growing in a fast-paced, collaborative environment and want to work with cutting-edge technology, then we'd love to hear from you!

Please keep an eye on your spam/junk email folder for correspondence from Team Tailor.

Department
Product & Technology
Locations
Multiple locations
Remote status
Hybrid
Employment type
Full-time

Original Job Description

Senior Software QA Engineer

Senior Software QA Engineer at Crimson Education: design and scale end-to-end quality automation for AI-powered apps, own test coverage, and ensure safe, reliable admissions tech.

Apply for this job

Former Ivy League and Top 20 admissions officers to rigorously review and refine applications
Professors and PhD teaching fellows from leading universities to guide students through original, independent research — with pathways to selective peer-reviewed publication or conference presentation
Past ISEF winners and judges who coach students to compete for state, national, and international science fair awards
Capstone project mentors who help students design and scale leadership initiatives with real-world impact, measurable outcomes, and credible external validation

The Role

What are the main responsibilities for this role?

Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).
CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.
Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.
Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.
AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.
Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.
Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.
Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.

What skills and experience are required?

4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.
Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.
Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).
- Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).
Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.
Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.
Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).
A healthy obsession with AI safety and ethics.

Why work for Crimson?

Rapidly growing start up, with a flexible working environment where you will be empowered to structure how you work
Limitless development and exposure - our internal promotions/role changes made up 33% of all recruitment last year
$1000 individual training budget per year, we love to ‘Level Up’ (it’s one of our core values)!
Psychologist on staff
Insightful fireside chats and workshops to help support our high-performing and ambitious team
Radical Candor is a feedback approach we live by
We’re a global player with 28 markets (and growing) across the globe. Most roles have the option to work from one of our many offices or remotely!

Department
Product & Technology
Locations
Multiple locations
Remote status
Hybrid
Employment type
Full-time

Let me find your next job.

Thanks - you're signed up.

Senior Software QA Engineer

Edtech.com's Summary

Senior Software QA Engineer Full Description

Senior Software QA Engineer

Quality Assurance Analyst

Staff Quality Assurance Engineer

Senior Software Quality Engineer (Mexico Remote)

Compliance Quality Assurance Specialist

UTS - Temporary Quality Assurance Associate at UNC Chapel Hill