Relias
AI Engineer
🇺🇸 Hybrid - Morrisville, NC
🕑 Full-Time
💰 TBD
💻 Information Technology
🗓️ August 4th, 2025
Docker
Kubernetes
Langchain
Edtech.com's Summary
Relias LLC is hiring an AI Engineer to design and deploy scalable large language model (LLM)-based systems and build modular infrastructure supporting generative AI applications. The role involves implementing protocols for intelligent system coordination, developing rigorous testing frameworks, and collaborating with cross-functional teams to integrate AI capabilities into products.
Highlights
- Architect and deploy generative AI systems using large language models with Model Context Protocol (MCP) and Agent-to-Agent (A2A) communication.
- Develop reusable, LLM-agnostic infrastructure supporting prompt chaining, context caching, and retrieval-augmented generation (RAG).
- Design automated frameworks for LLM evaluation, regression testing, bias mitigation, and performance monitoring.
- Collaborate with data scientists, engineers, and product managers to deliver AI-enabled features in user-facing and backend systems.
- Write maintainable, documented code to promote auditability and team collaboration.
- Require 5+ years AI/ML engineering experience and 2+ years specifically deploying LLM systems in production.
- Proficiency with AI protocols (MCP, A2A), Python, LangChain, LlamaIndex, MLflow, FastAPI, Azure cloud, Docker, and Kubernetes.
- Bachelor's degree in Data Science, Computer Science, or related field required; Master’s or PhD preferred.
- Experience with LLM testing pipelines including synthetic evaluation and human-in-the-loop feedback.
- Flexible hybrid work policy requires living within commuting distance to Morrisville, NC headquarters with periodic onsite presence.
AI Engineer Full Description
AI Engineer
Are you looking for a high energy, strategic, and fast-paced position as an AI Engineer? Join Relias, the company changing lives throughout the world by helping healthcare organizations improve their clinical and financial outcomes!
For 11,000+ health care and human service organizations, Relias helps clients deliver better clinical and financial outcomes by elevating the performance of teams. We help organizations across the continuum of care get better at maintaining compliance, developing staff and promoting consistent, high-quality care. Our platform employs assessments to reveal specific gaps in skills and addresses them with personalized and engaging learning, choosing from 7,000+ online courses that meet accrediting board, state and federal requirements. We are passionate about our products and our clients; what we deliver and the impact we have on the world is truly something you can be proud to represent. Join us and make a difference.
WHAT CAN RELIAS OFFER YOU?
- Fantastic health and wellness benefits package, including an outstanding 401k match, a flexible PTO program, and a generous and inclusive parental leave policy. Additionally, Relias pays for the employee portion of the monthly healthcare premium!
- Flexible work environment with onsite and work from home options – you choose when you want to come into the office!
- Active Employee Resource Groups open to all employees!
- Comprehensive onboarding program – a great introduction to our company, customers and culture!
- Growth and career advancement opportunities!
- 20%+ annual employee promotion and transfer rate
- Multiple development program options – leadership development, professional development curriculums, and Nanodegree options in both technology and data science
- Professional development gained from conference attendance and participation in organizations like NC Tech
- Onsite 321 Coffee Shop providing free coffee and pastries to employees
SUMMARY:
We are seeking a highly skilled and forward-thinking AI Engineer to design and deploy scalable LLM-based systems within our Data & Generative AI team. This role is instrumental in building robust, repeatable, and LLM-agnostic infrastructure that supports the full lifecycle of generative AI applications. The AI Engineer will leverage protocols such as the Model Context Protocol (MCP) and Agent-to-Agent (A2A) communication to enable intelligent, autonomous system behavior across our product ecosystem. This is a hands-on engineering role that demands fluency in model orchestration, rigorous validation frameworks, and a deep understanding of production-grade AI infrastructure.
WHAT YOU’LL BE DOING:
•LLM-based System Implementation: Architecting, building, and deploying generative AI systems using large language models. Applying MCP and A2A protocols to ensure dynamic, context-aware, and multi-agent capabilities across AI workflows.
•LLM-Agnostic Infrastructure Development: Developing modular, reusable infrastructure components that support integration with various foundation models (e.g., OpenAI, Azure OpenAI, open-source LLMs). Enabling prompt chaining, context caching, and retrieval-augmented generation (RAG) in a scalable manner.
•Testing and Validation Frameworks: Designing and implementing frameworks for automated evaluation, regression testing, and performance monitoring of LLMs. Ensuring high standards for factual consistency, bias mitigation, hallucination control, and user alignment.
•Cross-functional Integration: Collaborating with Data Scientists, Engineers, and Product Managers to translate AI capabilities into real-world applications. Supporting the integration of LLM functionality into user-facing tools and backend systems.
•Documentation and Maintainability: Writing high-quality, maintainable code with clear documentation to support auditability, version control, and team collaboration. Contributing to internal knowledge sharing and onboarding materials.
•LLM-Agnostic Infrastructure Development: Developing modular, reusable infrastructure components that support integration with various foundation models (e.g., OpenAI, Azure OpenAI, open-source LLMs). Enabling prompt chaining, context caching, and retrieval-augmented generation (RAG) in a scalable manner.
•Testing and Validation Frameworks: Designing and implementing frameworks for automated evaluation, regression testing, and performance monitoring of LLMs. Ensuring high standards for factual consistency, bias mitigation, hallucination control, and user alignment.
•Cross-functional Integration: Collaborating with Data Scientists, Engineers, and Product Managers to translate AI capabilities into real-world applications. Supporting the integration of LLM functionality into user-facing tools and backend systems.
•Documentation and Maintainability: Writing high-quality, maintainable code with clear documentation to support auditability, version control, and team collaboration. Contributing to internal knowledge sharing and onboarding materials.
YOU’VE GOT WHAT IT TAKES IF YOU HAVE/ARE:
•5+ years of experience in AI/ML engineering, with 2+ years specifically in building and deploying LLM-based systems in production.
•Bachelor’s degree in Data Science, Computer Science, or a related field.
•Protocol-Oriented LLM Design: Proficiency in leveraging Model Context Protocol (MCP) and Agent-to-Agent (A2A) architecture to enable intelligent coordination between AI systems.
•LLM Infrastructure Engineering: Strong experience building prompt management, context orchestration, and evaluation frameworks that are model-agnostic and production-ready.
•AI Evaluation & Monitoring: Skilled in designing and operationalizing LLM testing pipelines using synthetic evaluation, unit prompting, and human-in-the-loop feedback loops.
•Role Model and Mentorship: Strong coding skills in Python and AI toolchains (e.g., LangChain, LlamaIndex, MLflow, FastAPI, etc.), with experience in cloud platforms (Azure preferred) and containerization (Docker, Kubernetes).
•Bachelor’s degree in Data Science, Computer Science, or a related field.
•Protocol-Oriented LLM Design: Proficiency in leveraging Model Context Protocol (MCP) and Agent-to-Agent (A2A) architecture to enable intelligent coordination between AI systems.
•LLM Infrastructure Engineering: Strong experience building prompt management, context orchestration, and evaluation frameworks that are model-agnostic and production-ready.
•AI Evaluation & Monitoring: Skilled in designing and operationalizing LLM testing pipelines using synthetic evaluation, unit prompting, and human-in-the-loop feedback loops.
•Role Model and Mentorship: Strong coding skills in Python and AI toolchains (e.g., LangChain, LlamaIndex, MLflow, FastAPI, etc.), with experience in cloud platforms (Azure preferred) and containerization (Docker, Kubernetes).
IT WOULD BE GREAT IF YOU HAD:
•Master’s degree or Ph.D. in Data Science, Computer Science, or a related field.
•Experience with evaluating multiple LLMs and integrating open-source model
•Experience with evaluating multiple LLMs and integrating open-source model
Relias is an Equal Opportunity Employer and a Drug-Free workplace
IN OFFICE REQUIREMENT:
Relias values collaboration and wants to ensure that our team members have opportunities to work with their teams regularly for professional development opportunities. Our flexible hybrid work environment requires that you live in the state of North Carolina, within a commutable distance to our office (~1-hour commute). You would be expected to work in our Morrisville, NC Headquarters approximately 30/40 days/quarter.
Company: Relias LLC
Country: United States of America
State/Region: North Carolina
City: Morrisville
Postal Code: 27560
Job ID: 281901
Date: Aug 4, 2025
Location: Morrisville, NC, US, 27560
Similar Jobs
Systems Engineer
Adtalem Global Education
🇺🇸 Hybrid - Chicago, IL
Director - Digital Business Relationship Partner
Adtalem Global Education
🇺🇸 Hybrid - Chicago, IL
Director - Digital Business Relationship Partner
Adtalem Global Education
🇺🇸 Hybrid - Lisle, IL
Senior Full Stack Node.js/Angular/Mobile Developer
Adtalem Global Education
🇺🇸 Hybrid - Columbia, MD
Engineer II Digital Media Technology Services
University of the Pacific
🇺🇸 Sacramento, CA
