Agentic AI & Observability Engineer

Role: Agentic AI & Observability Engineer

Location: Plano, TX or Mc Lean VA Prefer Local

Role Summary

We are looking for an experienced Agentic AI & Observability Lead or an Architect with strong hands-on expertise in AI agent development, MCP (Model Context Protocol) server development, observability, data analytics, and cloud-native engineering. The candidate will design and build intelligent agents, MCP-based integrations, telemetry and analytics solutions, and automated provisioning capabilities across modern distributed application environments.

The ideal candidate should have strong development experience in Python and/or JavaScript, combined with practical knowledge of OpenTelemetry, Kibana, distributed tracing, log analytics, Kubernetes, AWS, data streaming, CI/CD, and security & compliance.

Key Responsibilities

• Design, develop, and deploy AI/Agentic AI applications and autonomous agents that interact with enterprise systems, APIs, data platforms, and operational tools.
• Design and build MCP Servers to securely expose enterprise tools, APIs, data sources, and operational capabilities to AI agents.
• Develop reusable MCP tools, resources, and integrations, including authentication, authorization, error handling, logging, and monitoring.
• Build backend services, APIs, integrations, and automation using Python and/or JavaScript.
• Implement application and platform observability using the OpenTelemetry framework, including logs, metrics, and traces.
• Implement and troubleshoot distributed tracing across microservices and distributed application environments.
• Build advanced log analytics, dashboards, visualizations, and operational insights using Kibana.
• Analyze telemetry and operational data to identify anomalies, application failures, performance bottlenecks, and recurring operational patterns.
• Design and integrate real-time and near-real-time data streams for telemetry, analytics, and AI-agent use cases.
  
• Develop and deploy solutions in Kubernetes-based containerized environments.
• Design and integrate solutions with AWS services and cloud-native architectures.
• Build and maintain CI/CD pipelines for automated build, testing, security validation, and deployment.
• Automate application, infrastructure, and service provisioning using Python or JavaScript.
• Apply enterprise security, compliance, identity, access-control, secrets-management, and data-protection requirements across agent and observability solutions.
• Work with application development, SRE, DevOps, platform, security, and architecture teams to onboard applications and implement observability and AI-driven operational capabilities.
• Develop production-quality solutions with appropriate scalability, resiliency, performance, security, and monitoring.

Required Technical Skills

Agentic AI & Development

• Strong experience developing AI agents / Agentic AI solutions.
• Hands-on experience designing and building MCP Servers and integrating MCP clients, tools, resources, APIs, and enterprise data sources.
• Strong programming experience in Python and/or JavaScript.
• Experience developing REST APIs, backend services, integrations, and automation frameworks.

Observability & Analytics

• Strong hands-on experience with OpenTelemetry (OTel).
• Experience implementing and analyzing distributed traces, logs, and metrics.
• Strong experience with Kibana dashboards and log analytics.
• Ability to correlate telemetry across distributed systems for troubleshooting and root-cause analysis.
• Experience working with high-volume telemetry and data streams.

Cloud & Platform Engineering

• Strong experience with Kubernetes, containers, and microservices architectures.
• Hands-on experience with AWS and cloud-native services.
• Experience developing and maintaining CI/CD pipelines.
• Experience with infrastructure/application provisioning and automation using Python or JavaScript.

Security & Compliance

• Understanding of secure application and API development.
• Experience implementing authentication, authorization, RBAC, secrets management, encryption, and secure communication.
• Ability to develop solutions aligned with enterprise security, governance, audit, and compliance requirements.

Preferred Qualifications

• Experience integrating AI agents with observability, incident management, DevOps, or enterprise operational platforms.
• Experience building agent-based solutions for incident triage, root-cause analysis, log analysis, remediation, and operational automation.
• Knowledge of SRE, AIOps, SLIs/SLOs, reliability engineering, and automated remediation.
• Experience working with large-scale distributed and microservices-based enterprise applications.
• Understanding of secure AI-agent architectures, including tool permissions, guardrails, identity propagation, auditability, and controlled access to enterprise systems.