--- title: A Beginner's Guide to Using the Prometheus Operator description: We explain what operators are in general, how the Prometheus operator works and how to configure it to best use Prometheus and Alertmanager. image: https://blog.container-solutions.com/hubfs/prometheus.png --- Check out our **[Cloud Native Services](https://www.container-solutions.com/services) **and** **book a call with one of our experts today!  [![Container Solutions](https://blog.container-solutions.com/hubfs/CS_blog/logo-main.svg "Container Solutions")](http://container-solutions.com) - share: - [**](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fblog.container-solutions.com%2Fprometheus-operator-beginners-guide) - [**](https://www.twitter.com/share?url=https%3A%2F%2Fblog.container-solutions.com%2Fprometheus-operator-beginners-guide) - [**](http://www.linkedin.com/shareArticle?mini=true&url=https://blog.container-solutions.com/prometheus-operator-beginners-guide) - [**](https://blog.container-solutions.com/prometheus-operator-beginners-guide#) [![A Beginners Guide to Using the Prometheus Operator](https://blog.container-solutions.com/hubfs/prometheus.png) ](https://blog.container-solutions.com/prometheus-operator-beginners-guide) [Kubernetes](https://blog.container-solutions.com/tag/kubernetes), [SRE](https://blog.container-solutions.com/tag/sre), [CNO](https://blog.container-solutions.com/tag/cno) # A Beginner's Guide to Using the Prometheus Operator [Rodrigo Martinez](https://blog.container-solutions.com/author/rodrigo-martinez) ![Rodrigo Martinez](https://blog.container-solutions.com/hubfs/Author%20Images/T02P1FKF3-U01BD762FT9-66844689eb6f-512.jpeg) January 25, 2022 16 minutes Read Prometheus is a simple and effective open-source monitoring system. In the years after we published the article [Monitoring Microservices with Prometheus](https://blog.container-solutions.com/microservice-monitoring-with-prometheus), the system has graduated from the Cloud Native Computing Foundation (CNCF) and became the preferred monitoring tool for distributed systems. One of the reasons for this, as mentioned in our previous article, is its intuitive simplicity. It doesn’t try to do anything fancy. It provides a data store, data scrapers, an alerting mechanism and a very simple user interface. Deploying Prometheus and the associated Alertmanger tool can be a complicated task, but there is tooling available to simplify and automate the process, such as the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) project. In this blog post, we explain what operators are in general, how the Prometheus operator works and how to configure it to best use Prometheus and Alertmanager. ## Operators As stated in our article [Kubernetes Operators Explained](https://blog.container-solutions.com/kubernetes-operators-explained), operators are a kind of software extension to Kubernetes. They provide a consistent approach to handle all the application operational processes automatically, without any human intervention, which they achieve through close cooperation with the Kubernetes API. Operators are built on two key principles of Kubernetes: Custom Resources (CRs), implemented here by way of Custom Resource Definitions (CRDs) and custom controllers. A CR is an extension of the Kubernetes API that provides a place where you can store and retrieve structured data—the desired state of your application. Custom controllers are used to observe this CR and with the received information take actions to adjust the Kubernetes cluster to the desired state. ## The Prometheus Operator The main purpose of this operator is to simplify and automate the configuration and management of the Prometheus monitoring stack running on a Kubernetes cluster. Essentially it is a custom controller that monitors the new object types introduced through the following CRDs: - **Prometheus**: defines the desired Prometheus deployments as a StatefulSet - **Alertmanager**: defines a desired Alertmanager deployment - **ServiceMonitor**: declaratively specifies how groups of Kubernetes services should be monitored - **PodMonitor**: declaratively specifies how groups of pods should be monitored - **Probe**: declaratively specifies how groups of ingresses or static targets should be monitored - **PrometheusRule**: defines a desired set of Prometheus alerting and/or recording rules - **AlertmanagerConfig**: declaratively specifies subsections of the Alertmanager configuration ## Why the Prometheus Operator As expressed before, using the operator can drastically reduce the effort to configure, implement, and manage all the components of a Prometheus monitoring stack. It also provides dynamic updates of resources, like alerting and/or Prometheus rules, with no downtime. Using the introduced CRDs is relatively straight forward, and a turn key solution for adopting operational best practices for this stack. Furthermore, this approach makes it possible to run multiple instances, even with different versions of Prometheus. ## Using the Prometheus Operator ## Prerequisites To follow the examples shown in this post, it is necessary to meet the following requirements: - **Kubernetes cluster**: For testing purposes, we recommend using [Kind](https://kind.sigs.k8s.io) to run a local cluster using Docker containers, [Minikube](https://minikube.sigs.k8s.io/docs/) can be used as an alternative - `kubectl` **command-line tool**: installed and configured to connect to the cluster - **A web application exposing Prometheus metrics**: We’re using the [microservices-demo](https://github.com/microservices-demo), which simulates the user-facing part of an e-commerce website and exposes a `/metrics` endpoint for each service. Follow the [documentation](https://microservices-demo.github.io/deployment/kubernetes-start.html) to deploy it on the cluster ## Deploy the Operator We start by deploying the Prometheus Operator into the cluster. We have to create all the CRDs that define the Prometheus, Alertmanager, and ServiceMonitor abstractions used to configure the monitoring stack—as well as the Prometheus Operator controller and Service. This can be done using the `bundle.yaml` file from the Prometheus Operator GitHub repository: ``` kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml ``` First we verify that all the CRDs were created: ``` kubectl get crds ``` The output should be similar to this: ``` NAME CREATED AT alertmanagerconfigs.monitoring.coreos.com 2021-04-20T19:34:44Z alertmanagers.monitoring.coreos.com 2021-04-20T19:34:57Z podmonitors.monitoring.coreos.com 2021-04-20T19:35:00Z probes.monitoring.coreos.com 2021-04-20T19:35:01Z prometheuses.monitoring.coreos.com 2021-04-20T19:35:06Z prometheusrules.monitoring.coreos.com 2021-04-20T19:35:11Z servicemonitors.monitoring.coreos.com 2021-04-20T19:35:12Z thanosrulers.monitoring.coreos.com 2021-04-20T19:35:14Z ``` Then we check that the operator was created in the current namespace (default) and the pod is in the Running state: ``` kubectl get deploy ``` ``` NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 5m59s ``` ``` kubectl get pods ``` ``` NAME READY STATUS RESTARTS AGE prometheus-operator-5b5887c64b-w7gqj 1/1 Running 0 6m1s ``` Finally, we confirm that the operator service has also been created: ``` kubectl get service ``` ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-operator ClusterIP None 8080/TCP 8m25s ``` ## RBAC Permissions The Prometheus server needs access to the Kubernetes API to scrape targets and reach the Alertmanager clusters. Therefore, a `ServiceAccount` is required to provide access to those resources, which must be created and bound to a `ClusterRole` accordingly: ``` apiVersion: v1 kind: ServiceAccount metadata: name: prometheus --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/metrics - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: [""] resources: - configmaps verbs: ["get"] - apiGroups: - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: default ``` Add the above to a manifest file `rbac.yaml`, then apply: ``` kubectl apply -f rbac.yaml ``` Check that the role was created and bound to the `ServiceAccount`: ``` kubectl describe clusterrolebinding prometheus ``` ``` Name: prometheus Labels: Annotations: Role: Kind: ClusterRole Name: prometheus Subjects: Kind Name Namespace ---- ---- --------- ServiceAccount prometheus default ``` ## Deploy and Configure ## Prometheus After creating the Prometheus `ServiceAccount` and giving it access to the Kubernetes API, we can deploy the Prometheus instance. Create a file `prometheus.yaml` with this content: ``` apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: serviceAccountName: prometheus serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} podMonitorSelector: {} resources: requests: memory: 400Mi ```   This manifest defines the `serviceMonitorNamespaceSelector`, `serviceMonitorSelector` and `podMonitorSelector` fields to specify which CRs to include. In this example, the `{}` value is used to match all the existing CRs. If we want for instance to match only the `serviceMonitors` in the `sock-shop` namespace we could use the following `matchLabels` value: ``` serviceMonitorNamespaceSelector: matchLabels: name: sock-shop ``` Apply the file: ``` kubectl apply -f prometheus.yaml ``` Check that the instance is in the Running state: ``` kubectl get prometheus ``` ``` NAME VERSION REPLICAS AGE prometheus 10s ``` ``` kubectl get pods ``` ``` NAME READY STATUS RESTARTS AGE prometheus-prometheus-0 2/2 Running 5 10s ``` A `prometheus-operated` service should also have been created: ``` kubectl get services ``` ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-operated ClusterIP None 9090/TCP 17s ``` Access the server by forwarding a local port to the service: ``` kubectl port-forward svc/prometheus-operated 9090:9090 ``` ![Image-1](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-1.png?width=1920&name=Image-1.png) ## ServiceMonitor The operator uses `ServiceMonitors` to define a set of targets to be monitored by Prometheus. It uses label selectors to define which Services to monitor, the namespaces to look for, and the port on which the metrics are exposed. Create a file `service-monitor.yaml` with the following content to add a `ServiceMonitor` so that the Prometheus server scrapes only its own metrics endpoints: ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: prometheus labels: name: prometheus spec: selector: matchLabels: operated-prometheus: "true" namespaceSelector: any: true endpoints: - port: web ``` The `ServiceMonitor` only matches services containing the `operated-prometheus: "true"` label which is added automatically to all the Prometheus instances and scrapes the port named `web` on all the underlying endpoints. As the `namespaceSelector` is set to `any:true` all the services in any namespace matching the selected labels are included. After applying the manifest, the Prometheus endpoints that were picked up as scrape targets should be shown on the Prometheus UI targets page: ![Image-2](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-2.png?width=1920&name=Image-2.png) ## PodMonitor There could be use cases that require scraping Pods directly, without direct association with services (for instance scraping sidecars). The operator also includes a `PodMonitor` CR, which is used to declaratively specify groups of pods that should be monitored. As an example, we’re using the `front-end` app from the [microservices-demo](https://github.com/microservices-demo) project, which, as we mentioned before, simulates the user-facing part of an e-commerce website that exposes a `/metrics` endpoint. Define a `PodMonitor` in a manifest file `podmonitor.yaml` to select only this deployment pod from the sock-shop namespace. Even though it could be selected using a `ServiceMonitor`, we’ve used a `targetPort` field instead. This is because the pod exposes metrics on port `8079` and doesn’t include a port name: ``` apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: front-end labels: name: front-end spec: namespaceSelector: matchNames: - sock-shop selector: matchLabels: name: front-end podMetricsEndpoints: - targetPort: 8079 ``` The front-end endpoint should have been added as a Prometheus target: ![Image-3](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-3.png?width=1920&name=Image-3.png) ## Additional Scrape Configuration It’s possible to append additional scrape configurations to the Prometheus instance via secret files. These files must follow the[ Prometheus configuration scheme](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) and the user is responsible to make sure that they are valid. An example of which you can see below, where an additional job is added to Prometheus to scrape the catalogue service endpoints. First generate the `prometheus-additional-job.yaml` file that declares the job to scrape the catalogue service: ``` \- job\_name: "catalogue" static\_configs: - targets: \["catalogue.sock-shop"\] ``` ```   ``` Then create the `additional-scrape-configs.yaml` secret file with the `prometheus-additional-job.yaml` content: ``` kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional-job.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml ``` Check the `additional-scrape-configs.yaml` content and apply: ``` apiVersion: v1 data: prometheus-additional-job.yaml: LSBqb2JfbmFtZTogImNhdGFsb2d1ZSIKICBzdGF0aWNfY29uZmlnczoKICAgIC0gdGFyZ2V0czogWyJjYXRhbG9ndWUuc29jay1zaG9wIl0K kind: Secret metadata: creationTimestamp: null name: additional-scrape-configs ``` ``` kubectl apply -f additional-scrape-configs.yaml ``` Finally the Prometheus instance must be edited to reference the additional configuration using the `additionalScrapeConfigs` field: ``` apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: serviceAccountName: prometheus serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} podMonitorSelector: {} additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional-job.yaml resources: requests: memory: 400Mi enableAdminAPI: false ``` ``` kubectl apply -f prometheus.yaml ``` Now the catalogue endpoint should also be listed as a Prometheus target: ![Image-4](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-4.png?width=770&name=Image-4.png) ## Alertmanager The Prometheus Operator also introduces an Alertmanager resource, which allows users to declaratively describe an Alertmanager cluster. It also adds an `AlertmanagerConfig` CR, which allows users to declaratively describe Alertmanager configurations. First, create an `alertmanager-config.yaml` file to define an `AlertmanagerConfig` resource that sends notifications to a non-existent `wechat` receiver and its corresponding Secret file: ``` apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: config-alertmanager labels: alertmanagerConfig: socks-shop spec: route: groupBy: ['job'] groupWait: 30s groupInterval: 5m repeatInterval: 12h receiver: 'wechat-socks-shop' receivers: - name: 'wechat-socks-shop' wechatConfigs: - apiURL: 'http://wechatserver:8080/' corpID: 'wechat-corpid' apiSecret: name: 'wechat-config' key: 'apiSecret' --- apiVersion: v1 kind: Secret type: Opaque metadata: name: wechat-config data: apiSecret: cGFzc3dvcmQK ``` Apply the above manifest: ``` kubectl apply -f alertmanager-config.yaml ``` Then create the `alertmanager.yaml` file to define the Alertmanager cluster: ``` apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: name: socks-shop spec: replicas: 1 alertmanagerConfigSelector: matchLabels: alertmanagerConfig: example ``` The `alertmanagerConfigSelector` field is used to select the correct `AlertmanagerConfig`. Apply and check that it was created: ``` kubectl apply -f alertmanager.yaml ``` ``` alertmanager.monitoring.coreos.com/socks-shop created ``` An `alertmanager-operated` service should have been created automatically, use it to access the Alertmanager web UI: ``` kubectl port-forward svc/alertmanager-operated 9093:9093 ``` ![Image-6](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-6.png?width=1920&name=Image-6.png) Now we have a fully functional Alertmanager cluster, but without any alerts fired against it. Alert rules can be added using the `PrometheusRule` custom resource to define the rules that are evaluated. ## PrometheusRules The `PrometheusRule` CR supports defining one or more `RuleGroups`. These groups consist of a set of rule objects that can represent either of the two types of rules supported by Prometheus, recording or alerting. As an example, create the `prometheus-rule.yaml` file with the following `PrometheusRule` that will always trigger an alert: ``` apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: socks-shop role: alert-rules name: prometheus-example-rules spec: groups: - name: ./example.rules rules: - alert: ExampleAlert expr: vector(1) ``` Now that the Alertmanager cluster is running and an alert rule was created, we need to connect it to Prometheus. To do this, edit the Prometheus instance to specify the Alertmanager cluster to use and the alert rules to be mounted into it: ``` apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: serviceAccountName: prometheus serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} podMonitorSelector: {} additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional-job.yaml resources: requests: memory: 400Mi enableAdminAPI: false alerting: alertmanagers: - namespace: default name: alertmanager-operated port: web ruleSelector: matchLabels: role: alert-rules prometheus: socks-shop ``` We should be able to see the alert firing in the Alertmanager UI and also in the Prometheus instance alerts section: ![Image-6](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-6.png?width=1920&name=Image-6.png) ![Image-7](https://blog.container-solutions.com/hs-fs/hubfs/Blog%20Images/Image-7.png?width=1920&name=Image-7.png) ## Final thoughts As we mentioned at the beginning of this post, and saw in the earlier examples, using the Prometheus Operator can help to reduce the overhead of managing the components of Prometheus and Alertmanager in a fast, automated, and reliable way. Despite all the advantages an operator provides, we have to remember that we are adding another layer of abstraction. With this comes a small increase in complexity, which can lead to unnoticed misconfigurations that could be harder to debug than a traditional static configuration. A final thought, although the operator may be seen as introducing an extra layer of complexity, it is our opinion that this is a justified trade-off, and that the benefits far outweigh any potential downsides. [![Download the Cloud Native Attitude book for free](https://no-cache.hubspot.com/cta/default/2252258/ac93f2fd-838a-41d9-baa1-49b957ca2622.png)](https://cta-redirect.hubspot.com/cta/redirect/2252258/ac93f2fd-838a-41d9-baa1-49b957ca2622) [### Entitlement in Open Source ](https://blog.container-solutions.com/entitlement-in-open-source) [![prev-arrow](https://cdn2.hubspot.net/hubfs/3842749/Wow%202019/Images/left-arrow.png) Previous article](https://blog.container-solutions.com/entitlement-in-open-source) [ Next article ![next-arrow](https://cdn2.hubspot.net/hubfs/3842749/Wow%202019/Images/right-arrow.png) ](https://blog.container-solutions.com/tutorialexternal-secrets-with-hashicorp-vault) [ ### Tutorial: How to Set External-Secrets with Hashicorp Vault ](https://blog.container-solutions.com/tutorialexternal-secrets-with-hashicorp-vault) Comments Leave your Comment ![cs-logo-white](https://blog.container-solutions.com/hs-fs/hubfs/cs-logo-white.png?width=150&height=61&name=cs-logo-white.png "cs-logo-white") #### **Talk to sales** [info@container-solutions.com](mailto:info@container-Solutions.com) #### Stay In Touch - [**](https://www.linkedin.com/company/container-solutions/) - [**](https://twitter.com/containersoluti) © 2024 Container Solutions