OmniOps is a Riyadh-based technology solutions provider, serving organizations across Saudi Arabia and beyond. With offices in Jordan, Egypt, and Morocco, we specialize in empowering businesses to migrate and scale their AI technology infrastructure confidently, achieving a high level of maturity in their digital landscapes. Our comprehensive services and products guarantee seamless cloud migration, scalability, and management, ensuring optimal performance and consistent reliability for our valued partners.
Role Summary
You set the technical direction for how we deliver observability to every client, across Grafana Alloy, the LGTM stack (Loki, Grafana, Tempo, Mimir) and OpenTelemetry. At Intermediate level the job is to deploy and run the stack well. At Senior/Lead level the job is to design the reference architecture, define the standards others deploy to, mentor the engineers doing the work, and be the senior technical voice in front of clients.
You stay hands-on: you will still write Alloy pipelines, debug cardinality blow-ups and lead incident response on the platform. But your main output is a platform and a team that scale across many tenants without you in every change.
Architecture and platform strategy
- Own the reference architecture for client observability across all deployment models: on-prem LGTM, Grafana Cloud, and hybrid setups that mix the two.
- Decide which model fits each client, based on data volume, retention, compliance, data residency, cost and the client's in-house skills.
- Design for scale and multi-tenancy: Mimir and Loki tenant isolation, sharding,replication, object storage backends, high availability and disaster recovery.
- Set the OpenTelemetry strategy: when to use OTLP end to end, when to use Alloy's
native components, and how semantic conventions and resource attributes are applied across clients.
- Track the Grafana and OpenTelemetry roadmaps, run proofs of concept (for example Grafana Beyla/eBPF, Pyroscope profiling, Faro frontend observability) and decide what we adopt.
Standards, automation and pipelines
- Define and maintain the standard Alloy configurations for Kubernetes, Linux and Windows, as versioned, reusable modules rather than per-client copies.
- Own the label and attribute taxonomy and the cardinality budget; set guardrails (relabelling, drop rules, limits) that stop one client or service from degrading a shared stack.
- Drive infrastructure-as-code and GitOps for the whole platform: stacks, agents, dashboards, alert rules and Grafana Cloud resources.
- Build the release and upgrade process for agents and backends, including staged rollouts, canaries and rollback across client estates.
Reliability, cost and operations
- Set SLOs for the observability platform itself (ingest availability, query latency, data freshness) and report against them.
- Own capacity planning and cost: storage growth, retention tiers, Grafana Cloud usage and plan limits, and the levers to reduce spend (adaptive metrics, log sampling, trace tail sampling).
- Lead or act as escalation point for platform incidents, run blameless postmortems and make sure the fixes land.
- Mature self-monitoring into a meta-monitoring layer that detects silent data loss, agent drift and backend saturation before clients notice.
- Embed security in the platform: authentication and SSO, RBAC and team/folder permissions, secrets handling, encryption in transit and at rest, and PII scrubbing/redaction in log and trace pipelines.
Client engagement and advisory
- Act as the senior technical lead on client engagements: discovery, solution design, effort estimates and technical sign-off.
- Run observability maturity assessments and produce roadmaps that move clients from basic monitoring to SLO-driven operations.
- Lead migrations onto the platform from tools such as Prometheus/Thanos, ELK/OpenSearch, Datadog, New Relic, Splunk or legacy Grafana Agent/Promtail.
- Coach client app teams on OpenTelemetry instrumentation and SLO design. The boundary from the Intermediate role still holds: clients own what they emit and their target numbers; we own the standards, the pipelines and the alerting machinery. Atthis level you set and negotiate that boundary, not just follow it.
Leadership and mentoring
- Mentor Intermediate and Junior engineers through pairing, design reviews and code reviews.
- Write and own the runbooks, architecture decision records and onboarding material that let the team deliver consistently without you.
- Set the team's engineering quality bar: review standards, testing of configurations, and definition of done for a client rollout.
- Work with delivery managers, sales/pre-sales and account teams to shape scope, pricing inputs and service offerings.
Required Skills And Experience
- 6+ years in DevOps, SRE or platform engineering, with at least 3 years focused on observability in production.
- Deep, production-scale experience with the Grafana stack: Grafana, Loki, Mimir (or Prometheus/Thanos/Cortex), Tempo and Alertmanager, including tuning, scaling and troubleshooting their internals (ingesters, distributors, compactors, query frontends, object storage).
- Expert with Grafana Alloy and/or the OpenTelemetry Collector: designing pipelines, writing reusable modules, processors and sampling strategies, and running them across Kubernetes, Linux and Windows.
- Strong OpenTelemetry knowledge: OTLP, SDKs and auto-instrumentation in at least two languages, semantic conventions, context propagation and the logs/metrics/traces correlation model.
- Advanced Kubernetes: operators, Helm chart authoring, RBAC, resource tuning and multi-cluster setups.
- Advanced query and data-modelling skills: PromQL, LogQL and TraceQL; recording rules; cardinality analysis and control.
- Proven experience designing SLOs, error budgets and burn-rate alerting, and turning them into working Grafana or Alertmanager rules.
- Infrastructure-as-code and GitOps in practice (Terraform, Helm, Argo CD or Flux, Ansible) and CI/CD for configuration.
- Grafana Cloud at an administrative level: stacks and tenants, access policies, usage and cost management, private connectivity.
- A track record of technical leadership: owning architecture decisions, mentoring engineers and leading incidents.
- Clear client-facing communication: you can explain trade-offs to engineers and to non technical stakeholders, and hold scope boundaries politely and firmly.
Nice To Have
- Experience leading observability delivery in an MSP or multi-tenant managed-service setting, across many clients at once.
- Hands-on with AWS, Azure and GCP native telemetry (CloudWatch, Azure Monitor, Cloud Logging) and integrating it into a central backend.
- Windows telemetry at depth: Windows Exporter, Event Log collection, ETW and IIS logs via Alloy.
- eBPF-based observability (Grafana Beyla, Cilium/Hubble), continuous profiling (Pyroscope) or frontend observability (Grafana Faro).
- Grafana k6, synthetic monitoring, Grafana OnCall/IRM or incident tooling such as PagerDuty or Opsgenie.
- Contributions to Grafana, OpenTelemetry or related open-source projects, or public talks and writing on observability.
- Certifications such as Grafana Certified Associate, CKA/CKAD, or a cloud architect certification.
