Senior AI/ML Engineer - Forward Deployed Engineer
Posted 2 days ago
The role in short
Serve as a technical lead embedded with Federal Agencies to design, build, and deploy secure, production-grade AI/ML solutions and agents while managing compliance and stakeholder coordination.
Position Summary
The Senior AI/ML Engineer serves as a senior Forward Deployed Engineer and accountable technical lead supporting one or more Federal Agencies. The role translates ambiguous mission needs into secure, usable, accessible, and Government-owned AI capabilities. The incumbent provides hands-on architecture, engineering, AI assurance, delivery leadership, and stakeholder advisory support across discovery, design, development, testing, deployment, operations, and knowledge transfer.
Essential Responsibilities
- Lead agency-specific AI solution discovery, feasibility analysis, architecture, engineering design, integration, deployment, and operational support.
- Translate incomplete mission needs into measurable problem statements, user needs, acceptance criteria, prioritized backlogs, experiments, and release plans.
- Architect and implement AI-enabled applications using appropriate methods, including classical ML, document intelligence, NLP, LLM integrations, RAG, agentic workflows, multi-agent patterns, MCP server-based components, APIs, and deterministic automation.
- Select the simplest safe and effective solution rather than defaulting to generative AI.
- Lead the technical design of data flows, identity and access controls, trust boundaries, fallback and rollback methods, observability, and resilience.
- Direct MLOps/LLMOps practices, including version control, CI/CD, model/artifact registration, evaluation, monitoring, retraining or refresh workflows, and release management.
- Define and oversee AI evaluation methods for accuracy, groundedness, relevance, hallucination, calibration, fairness, robustness, explainability, and operational impact.
- Lead AI-specific adversarial testing for prompt injection, jailbreaks, data poisoning, retrieval manipulation, insecure tool use, agent-loop failures, secrets exposure, model extraction, and vector-store corruption.
- Ensure human-in-the-loop controls, documented override and escalation mechanisms, audit logging, and clear distinction between AI-generated and human-verified outputs.
- Produce or direct production-ready architecture artifacts, technical documentation, model cards, test evidence, SBOM inputs, operational runbooks, and release packages.
- Coordinate directly with Federal Agency CIO/OCIO organizations, AI governance teams, enterprise platform teams, cloud organizations, architecture review boards, security authorization teams, privacy, accessibility, legal, records, and other Government stakeholders.
- Support RAIA, PIA/PTA, ATO/RMF, SSP, SAP, SAR, POA&M, and related compliance documentation.
- Ensure custom code, prompts, configurations, models, datasets, embeddings, and deployment assets are developed and maintained in Government-controlled repositories.
- Serve as the technical primary point of contact for assigned Federal Agency stakeholders and provide planned/unplanned coverage for other key personnel.
- Act as, or work directly with, the designated Security Lead to certify security evidence for each applicable deliverable.
Minimum Qualifications
- Bachelor’s degree in computer science, data science, software engineering, information systems, statistics, engineering, or comparable field; Master’s degree preferred.
- Seven or more years of hands-on experience in AI/ML engineering, software engineering, cloud engineering, enterprise architecture, or data engineering, including delivery of AI-enabled systems.
- At least two years of hands-on production experience in one or more of generative AI, LLM integration, RAG, model evaluation, AI agents, or production AI system operations.
- Demonstrated ability to deliver across AI, application development, data engineering, UX, integration, and operations in ambiguous environments.
- Demonstrated experience designing secure enterprise or Federal AI architectures, including cloud AI platforms, APIs, data pipelines, vector databases, model orchestration, MCP, evaluation systems, and MLOps.
- Working knowledge of Federal or comparably regulated security, privacy, accessibility, Responsible AI, ATO/RMF, FISMA, FedRAMP, and PII/CUI protection practices.
- Demonstrated experience advising Government or executive stakeholders and leading technical delivery across multiple teams.
Required Certification Evidence
The candidate must hold and provide verifiable evidence of at least one relevant AI/ML credential, such as:
- Azure AI / Generative AI certification
- AWS Generative AI or Machine Learning certification
- Google Cloud Generative AI / ML credential
- Databricks Generative AI Engineer credential
- NVIDIA Generative AI credential
- Certified Artificial Intelligence Professional (CAIP)
- Equivalent recognized AI/ML certification
Certification evidence must be supplied at onboarding and annually thereafter. Self-attestation is not sufficient.
Preferred Qualifications
- Direct experience supporting Federal Agencies, especially systems involving PII, CUI, public-facing digital services, grants, enforcement, benefits, claims, or case-management operations.
- Experience with AWS, Azure AI Foundry, Google Vertex AI, AWS Bedrock, GitHub Copilot/VS Code, GitLab, JIRA, and Government-controlled CI/CD pipelines.
- Experience with accessibility-by-design and delivery of systems me
Pay
$90.00 - $110.00 per hour
Benefits
- 401(k) matching
- Dental insurance
- Employee assistance program
- Flexible schedule
- Health insurance
- Life insurance
- Paid time off
- Parental leave
- Professional development assistance
- Referral program
- Vision insurance
Ability to Commute
- Washington, DC 20591 (Required)
Work Location
In person
Prep for this interview
5 questions likely to come up at Bravent LLC, each matched to a line in this posting.
- 1Cloud, deployment and security
They wrote: “Experience with AWS, Azure AI Foundry, Google Vertex AI, AWS Bedrock, GitHub Copilot/VS Code, GitLab, JIRA, and Government-controlled CI/CD pipelines.”
How would you debug a deployment that works in staging but fails at the customer's site?
How to answer
Why they ask: Customer environments differ in ways you can't see. This tests method under pressure.
- Compare versions, config and environment variables
- Check network rules, proxies and certificates (common in enterprise)
- Look at resource limits and permissions
- Add the missing check to your install steps afterwards
Tip: List the usual suspects and how you'd check each without full access.
- 2Discovery and scoping
They wrote: “Lead agency-specific AI solution discovery, feasibility analysis, architecture, engineering design, integration, deployment, and operational support.”
A customer says “we want AI in our claims process.” What do you ask in the first meeting?
How to answer
Why they ask: Most FDE work starts this vague. They want to hear you find the real problem before you pick a tool.
- Map the current process: who touches a claim, in what order, with which systems
- Find the slowest or most error-prone step and ask what it costs them
- Agree on one measurable goal (for example time per claim, or error rate)
- Ask what data you can access, and who signs off on security
Tip: Ask about the work, not the technology. Who does it today, how long it takes, what a mistake costs, and how they'd know it got better.
- 3Integrations and APIs
They wrote: “At least two years of hands-on production experience in one or more of generative AI, LLM integration, RAG, model evaluation, AI agents, or production AI system operations.”
Walk me through connecting our product to a customer's internal system that has no documentation.
How to answer
Why they ask: This happens weekly in the job. They want a calm, step-by-step method.
- Find who owns the system and get read access to a test environment
- Watch real traffic or logs to learn the data shapes
- Build a thin adapter with logging around every call
- Agree on how failures get reported, and who fixes what
Tip: Talk about finding the owner of the system first. The code is the easy part.
- 4Data and SQL
They wrote: “Demonstrated experience designing secure enterprise or Federal AI architectures, including cloud AI platforms, APIs, data pipelines, vector databases, model orchestration, MCP, evaluation systems, and MLOps.”
How would you design a daily pipeline that feeds customer data into our product?
How to answer
Why they ask: Tests basic data engineering: scheduling, failures, re-runs and monitoring.
- Extract with timestamps or change tracking, not full dumps, when possible
- Transform in steps you can re-run without creating duplicates
- Validate row counts and key fields before loading
- Alert a person on failure, with enough detail to fix it
Tip: Make re-running safe (same input, same result) and say how you'd know it failed.
- 5LLMs, RAG and agents
They wrote: “Lead AI-specific adversarial testing for prompt injection, jailbreaks, data poisoning, retrieval manipulation, insecure tool use, agent-loop failures, secrets exposure, model extraction, and vector-store corruption.”
A customer wants a chatbot over their internal documents. How do you build it?
How to answer
Why they ask: RAG is the most common first project. They want the whole path, from messy files to answers people trust.
- Collect and clean the documents; handle permissions per user
- Split them into chunks, embed, and store with source links
- Retrieve, then answer with citations; say “I don't know” when retrieval is weak
- Build a test set of real questions with expected answers before launch
Tip: Spend more time on the data and on testing than on the model.
Not the one? Get the next one by email
New roles land here every day. Pick what you want to hear about.
Want the job, not just the posting? Learn how FDEs work.
FDE Info is our free weekly newsletter for forward deployed engineers: tools, field notes from real deployments, and how FDE teams hire and pay.
Free. One email a week. Unsubscribe in one click.
