Senior Software Engineer (Applied AI / Forward Deployed)
Posted 3 days ago
The role in short
Lead forward-deployed AI engineering engagements by embedding with intelligence operators to design, build, and validate production agentic systems and guardrails.
Belay Technologies provides leading-edge technology and engineering services to the DoD, as well as state-of-the-art commercial products. We are a certified Service-Disabled Veteran-Owned Small Business (SDVOSB) in the Baltimore-Washington Area. We take the smartest approach, delivering innovative solutions to our customers. Put your vision into action with Team Belay!
Belay Technologies has been voted Baltimore Business Journal's (BBJ) Best Places to Work 2026!
Belay Technologies is seeking a Senior Software Engineer to join our intel team. AIMASSIST brings AI into the operations organization, embedding engineers with the operators, analysts, and developers who carry out the CNO mission. Our engineers learn how those teams work, find the tasks that consume their time, and build AI agents and tools that let them move faster without giving up accuracy or control. We build on both frontier models and on-premises open-weight models, served through the customer's enterprise AI platform, and turn them into mission capabilities such as agents that triage incoming data, automate repetitive tradecraft, or research across disparate sources on an analyst's behalf. We are now introducing autonomous agents the mission can trust: agents that take real actions within guardrails, are measured against evaluations built with the people who use them, and log every step so a reviewer can follow what they did. As a senior Applied AI Engineer, you will lead forward-deployed work with a mission customer from the first conversation through production and set the technical direction for the agents your team builds. You will mentor engineers who are newer to applied AI and help define how our team develops software with AI, including the review standards and validation layers that let us put autonomous coding agents to work on mission code. This role suits an experienced engineer who wants to shape how AI is fielded across operations and grow into project leadership.
Responsibilities
- Lead forward-deployed engagements with operations customers: elicit requirements from operators and analysts, identify the workflows where AI saves the most time, and deliver solutions from prototype to production.
- Architect and build agentic systems in Python that connect LLMs, MCP servers, Skills, and subagents to mission data sources and existing operational tools.
- Design guardrails for autonomous agents, including scoped permissions, human approval gates, and audit logging, so agents can act on operational systems safely.
- Build evaluation suites with mission users that measure the accuracy, reliability, and failure modes of AI capabilities, and use the results to decide when a capability is ready to field.
- Evaluate new models, frameworks, and AI design patterns, both frontier and open-weight, for fit against mission problems, and brief findings and tradeoffs to technical and government leadership.
- Use AI coding agents (e.g., Claude Code) in daily development, and help define team standards for reviewing, testing, and validating AI-generated code.
- Mentor engineers through design reviews, code reviews, and pairing, and lead technical design discussions for your project area.
- Communicate progress, risks, and tradeoffs to project leads and government stakeholders.
Candidates should have the following qualifications
- TS/SCI Clearance with polygraph
- 12 yrs., B.S. in a technical discipline or 4 additional yrs. in place of B.S.
Skills Requirements
- Extensive professional software engineering experience with advanced Python proficiency, including production services, APIs, and data pipelines that users depend on.
- Experience building LLM applications or agentic systems that use tool/function calling, structured outputs, context management, and multi-step workflows.
- Experience working directly with end users or mission customers to turn ambiguous operational problems into delivered software.
- Experience designing tests, evaluations, or validation layers for AI or other non-deterministic systems.
- Experience using AI coding assistants or agents (e.g., Claude Code, Codex, Copilot) for code generation, refactoring, and debugging, with the review discipline to own and explain everything they produce.
- Technical leadership experience, such as serving as tech lead, leading design decisions, or mentoring engineers.
- Experience with Git, CI/CD pipelines (GitLab preferred), Docker/Kubernetes, and Atlassian tools (Jira, Confluence).
- Strong written and verbal communication with technical and non-technical audiences.
Candidates are desired to have the following skills
- Experience with Model Context Protocol (MCP), Skills, subagents, or multi-agent orchestration frameworks such as PydanticAI or LangGraph.
- Experience securing LLM applications against prompt injection, data leakage, and over-permissioned tool access.
- Experience with retrieval systems (RAG, vector databases, Elasticsearch).
- Experience building spec-driven or autonomous development pipelines with AI coding agents.
- Prior forward-deployed or embedded customer-facing engineering experience.
- Experience with production CNO capabilities and operations, or knowledge of end-to-end SIGINT collection and analysis systems.
Perks and Benefits
- 8 weeks paid leave - 4 weeks of personal leave, 3 Yay! days, take off on your birthday,11 paid holidays and optional leave up to 6 days through Belay's volunteer program
- 10% matching in 401(k) contributions vested on day one
- $5,000 annual training/tuition
- Student Loan Repayment Program
- 100% company-funded HSA
- Rich medical coverage (100% coinsurance)
- Dental coverage including orthodontia
- Up to $420,000 in life insurance, premiums 100% company funded
- Amazon Prime, gym reimbursement, monthly lunches, games and prizes
- Pet adoption program, generous referral bonus program, fun events, and more!
At Belay, we are committed to creating an environment where you can thrive both professionally and personally. We believe in transparency and fairness in compensation. Several factors influence final salary including, but not limited to, geographic location, Federal Government contract labor categories and contract wage rates, relevant prior work experience, education, specific skills and competencies and certifications. Belay offers a competitive total compensation package that includes benefits and perks, a flexible work-life balance, and career development.
Salary Range: $190,000 to $240,000
Belay Technologies is a certified Service-Disabled Veteran-Owned Small Business located in Columbia, Maryland (Baltimore/Washington area). Belay Technologies specializes in systems automation and full stack development. Belay Technologies provides leading technology and engineering solutions to the DoD, as well as state-of-the-art commercial products. We hire software engineers, web designers, test engineers, systems engineers, systems administrators, database engineers and other tech services. We are an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, national origin, gender, sexual orientation, gender identity, age, physical or mental disability, genetic factors, military/veteran status or other characteristics protected by law.
Prep for this interview
5 questions likely to come up at Belay Technologies, each matched to a line in this posting.
- 1LLMs, RAG and agents
They wrote: “Architect and build agentic systems in Python that connect LLMs, MCP servers, Skills, and subagents to mission data sources and existing operational tools.”
When would you use an agent that takes actions, and how do you keep it safe?
How to answer
Why they ask: Agent work is growing fast in FDE postings. Senior candidates are expected to know where agents fail.
- Use agents for multi-step tasks with clear success checks; avoid them where one wrong action is costly
- Give the agent narrow tools and permissions, not broad access
- Require human approval for risky actions
- Log every step so the customer can audit it
Tip: Start from the cost of a wrong action. Limit what the agent can do, and log everything.
- 2Cloud, deployment and security
They wrote: “Experience with Git, CI/CD pipelines (GitLab preferred), Docker/Kubernetes, and Atlassian tools (Jira, Confluence).”
How would you debug a deployment that works in staging but fails at the customer's site?
How to answer
Why they ask: Customer environments differ in ways you can't see. This tests method under pressure.
- Compare versions, config and environment variables
- Check network rules, proxies and certificates (common in enterprise)
- Look at resource limits and permissions
- Add the missing check to your install steps afterwards
Tip: List the usual suspects and how you'd check each without full access.
- 3Discovery and scoping
They wrote: “Lead forward-deployed engagements with operations customers: elicit requirements from operators and analysts, identify the workflows where AI saves the most time, and deliver solutions from prototype to production.”
A customer says “we want AI in our claims process.” What do you ask in the first meeting?
How to answer
Why they ask: Most FDE work starts this vague. They want to hear you find the real problem before you pick a tool.
- Map the current process: who touches a claim, in what order, with which systems
- Find the slowest or most error-prone step and ask what it costs them
- Agree on one measurable goal (for example time per claim, or error rate)
- Ask what data you can access, and who signs off on security
Tip: Ask about the work, not the technology. Who does it today, how long it takes, what a mistake costs, and how they'd know it got better.
- 4Integrations and APIs
They wrote: “Extensive professional software engineering experience with advanced Python proficiency, including production services, APIs, and data pipelines that users depend on.”
How would you set up single sign-on for a large customer?
How to answer
Why they ask: SSO (SAML or OAuth/OIDC) is a common first task in enterprise deals and a common source of delays.
- Confirm their identity provider (Okta, Entra ID, Google) and protocol
- Exchange metadata and test in a staging tenant first
- Map user groups to roles in your product
- Plan for user removal: what happens when someone leaves the company
Tip: Mention the people involved, and the protocol second. The customer's identity team usually controls the timeline.
- 5Ownership and working with customers
They wrote: “Experience working directly with end users or mission customers to turn ambiguous operational problems into delivered software.”
Tell me about a project where the requirements kept changing. What did you do?
How to answer
Why they ask: Ambiguity is the norm. They want proof you can keep moving without a clear spec.
- The project and why requirements kept moving
- What you did to pin them down (short demos, written decisions)
- The trade-offs you made to keep shipping
- The result, with a number if you have one
Tip: Show how you reduced the change, beyond how you coped with it.
Not the one? Get the next one by email
New roles land here every day. Pick what you want to hear about.
Want the job, not just the posting? Learn how FDEs work.
FDE Info is our free weekly newsletter for forward deployed engineers: tools, field notes from real deployments, and how FDE teams hire and pay.
Free. One email a week. Unsubscribe in one click.
