AI Operations Engineer

Job Description: AI Operations Engineer

Position Overview

We are seeking an AI Operations Engineer to operate and maintain a GPU-accelerated AI inference platform. This is a hands-on technical role responsible for the day-to-day operations of LLM serving infrastructure — from bare-metal OS provisioning through production model deployment, monitoring, and capacity planning.

The platform serves large language models to internal applications. You will be the dedicated engineer ensuring this infrastructure is reliable, performant, and ready to absorb new models and workloads as requirements evolve.

Key Responsibilities

Infrastructure Provisioning and Automation

Take a vanilla Linux installation from zero to production-ready: OS hardening, patching, GPU driver and toolkit installation, Python environment setup, and inference engine deployment.

Maintain and extend infrastructure-as-code automation with proper version control practices to ensure repeatable, auditable deployments across the fleet.

Manage offline and restricted-network deployment workflows including local package repositories and model distribution with no internet dependency on production nodes.

Provision and configure containerized workloads with proper service management, including database backends, reverse proxies, and supporting services.

LLM Inference Engine Operations

Deploy, configure, and maintain LLM inference serving engines across a fleet of enterprise GPUs.

Configure multi-GPU model sharding based on model size, GPU topology, and workload requirements. Understand when and why to distribute a model across GPUs versus running independent instances.

Manage model quantization strategies and understand the trade-offs between model quality, memory footprint, and throughput for each approach.

Perform GPU memory and cache sizing calculations to determine maximum concurrency, context length limits, and optimal resource utilization per model and hardware configuration.

Configure and troubleshoot model-specific features including tool/function calling, reasoning modes, and multimodal model serving.

Evaluate and deploy new model releases as they are published — including testing multilingual quality, tool calling accuracy, and throughput characteristics.

API Gateway and Traffic Management

Operate and configure an API gateway layer providing load balancing, model routing, API key management, token counting, and per-application usage tracking.

Manage the supporting services around the gateway: database backends, caching layers, and reverse proxies with appropriate access controls.

Understand HTTP semantics, streaming responses (SSE), OpenAI-compatible API conventions, and how to diagnose issues across the full request path from client application through proxy to inference backend.

Monitoring, Reporting, and Cost Analysis

Own the observability stack: metrics collection, dashboarding, and GPU-level telemetry (utilization, memory, temperature, power draw).

Instrument and report on key operational metrics: request throughput, latency percentiles (p50/p95/p99), queue depth, concurrent users, tokens per second, and GPU utilization.

Deliver cost analysis and usage reporting down to per-user token consumption and per-application cost attribution.

Define and monitor alerting thresholds for capacity, performance degradation, and hardware health.

Capacity Planning and Performance Engineering

Conduct throughput analysis across different GPU configurations and model sizes to inform procurement and scaling decisions.

Plan fleet composition based on workload mix and future model requirements.

Benchmark new models and quantization methods to validate performance before production rollout.

Identify and resolve performance bottlenecks across the stack — from GPU driver and library compatibility issues to runtime behavior and network throughput.

Developer Collaboration and Prompt Engineering

Work closely with application development teams to onboard new workloads onto the inference platform.

Review and provide recommendations on system prompts used by applications to improve response quality, reduce token waste, and optimize for the specific models being served.

Help developers understand model capabilities and limitations, appropriate parameter tuning, and best practices for structured output and function calling.

Assist in troubleshooting model behavior issues reported by application teams — distinguishing between model limitations, prompt issues, and infrastructure problems.

Fine-Tuning (Foundational)

Understand the fundamentals of model fine-tuning: adapter-based methods (LoRA/QLoRA), dataset preparation, training configuration, and evaluation.

Be able to deploy fine-tuned models into the serving infrastructure.

Collaborate with data science or ML teams on fine-tuning workflows and understand how fine-tuned models differ in serving requirements from base models.

Required Qualifications

Experience

Minimum 4 years of hands-on experience in Linux systems engineering, with at least 2 years involving GPU infrastructure or ML/AI workloads.

Demonstrated experience deploying and operating LLM inference engines in production environments.

Strong working knowledge of the NVIDIA GPU software stack: drivers, toolkits, runtime libraries, and common failure modes.

Experience with infrastructure-as-code and configuration management tools.

Solid understanding of Linux containerization technologies, including rootless operation and service management integration.

Technical Skills

Operating Systems: Linux administration (RHEL family preferred), including kernel tuning, service management, storage, and network configuration.

GPU and AI Stack: NVIDIA drivers and toolkits, LLM inference engines, model quantization techniques, multi-GPU parallelism concepts.

Networking: HTTP/HTTPS, reverse proxy configuration, TLS/SSL, firewall management, network file systems, and load balancing concepts.

Monitoring: Metrics collection, visualization, and alerting using industry-standard observability tools.

Automation: Configuration management (Ansible preferred), shell scripting, Python for operational tooling.

Version Control: Git with disciplined branching, tagging, and commit practices.

Python: Virtual environments, dependency management, and debugging Python-based services.

Soft Skills

Ability to work independently and take ownership of infrastructure decisions with minimal supervision.

Clear technical communication — able to document procedures, write runbooks, and explain infrastructure constraints to development teams.

Methodical troubleshooting approach: isolate variables, reproduce issues, document root causes and fixes.

Comfort operating in environments with data sovereignty requirements and restricted internet access.

Preferred Qualifications

Experience with LLM API gateway or proxy solutions for multi-model routing and management.

Familiarity with LLM fine-tuning workflows (LoRA, QLoRA, dataset preparation, evaluation).

Experience with air-gapped or restricted-network deployments.

Knowledge of Arabic NLP or multilingual model evaluation.

Experience scaling GPU infrastructure across growing fleet sizes.

Familiarity with cloud GPU providers and bare-metal GPU hosting environments.

Understanding of Mixture of Experts (MoE) model architectures and their serving implications.