Pelorus Development Guide

We appreciate your interest in contributing to Pelorus! Use this guide to help you get up and running.

There are three main tracks of Pelorus development to consider.

Deployment Automation

This track mostly involves testing, fixing, and updating our Helm chart(s) to streamline the installation and configuration experience. Knowledge of Helm, OpenShift, Operators and Prometheus configuration is assumed for this work. 1. Dashboard Development

This is where we take the raw data we've collected and turn it into actionable visual representations that will help IT organizations make important decisions. Knowledge of Grafana and PromQL is required for contribution here. 1. Exporter Development

This track is focused around the development of custom Prometheus exporters to gather the information we need in order to calculate our core metrics. Python development experience is assumed.

Contributing to Deployment Automation

We use Helm to provide an automated deployment and configuration experience for Pelorus. We are always doing work to cover more and more complex use cases with our helm charts. In order to be able to effectively contribute to these charts, you'll need a cluster that satisfies all of the installation prerequisites for Pelorus.

See the Install guide for more details on that.

Currently we have two charts:

The operators chart installs the community operators on which Pelorus depends.
- Prometheus Operator
- Grafana Operator
The pelorus chart manages the Pelorus stack, which includes:
- Prometheus
- Thanos
- Grafana
- A set of Grafana Dashboards and Datasources
- The Pelorus exporters, managed in an exporter subchart.

We use Helm's chart-testing tool to ensure quality and consistency in the chart. When making updates to one of the charts, ensure that the chart still passes lint testing using ct lint. The most common linting failure is forgetting to bump the version field in the Chart.yaml. See below for instructions on updating the version.

Updating the chart versions

When any of our Helm charts are updated, we need to bump the version number. This allows for a seemless upgrade experience. In order to maintain versioning with our repository, our version numbers are derived from our tagged releases. Git provides a command which will give us that version, via git describe. However, the git generated version number is not fully semver compatible. To bridge that gap we use a tool called Version Tester (vert). We have also written a pre-commit hook script that will automatically detect when a version bump is needed and will make the required change. Here are the steps to get the hook set up.

Install Helm's chart-testing tool.
Install the latest release of vert
1. Run vert -g ^1 $(git describe) to test that its working.
Copy the pre-commit hook into your git hooks directory. cp _test/pre-commit .git/hooks/pre-commit

This script will use Helm's built-in linter to check whether a version bump is necessary, and if it is, it will take the current version from Chart.yaml and increment it one patch version. It will also keep appVersion fo the Pelorus chart up to date with the repo version using git describe.

Dashboard Development

We are continually doing work to enhance and bugfix the Pelorus dashboards. Doing so requires a complete Pelorus stack, including all exporters required to populate a given dashboard. See the Dashboards user guide for that information.

To effectively do dashboard development, you'll likely need at least two browser windows open, one with Grafana, and another with Prometheus for testing queries. Since our dashboards are imported to Grafana via the Grafana Operator, they get imported in read-only mode. Because of this, you'll need to make a copy of it for development purposes.

The following outlines a workflow for working on a dashboard:

Sign in to Grafana via the Grafana route.
Once signed in, sign as an administrator
Click the signin button in the bottom right corner
The admin credentials can be pulled from the following commands: oc get secrets -n pelorus grafana-admin-credentials -o jsonpath='{.data.GF_SECURITY_ADMIN_USER}' | base64 -d oc get secrets -n pelorus grafana-admin-credentials -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
Export the dashboard JSON.
Open the dashboard, and select the Share... button.
Select the Export tab.
Click View JSON.
Click Copy to Clipboard.
Import as a new dashboard
Click Create -> Import.
Paste your JSON code in the box and click Load.
Change the Name and Unique Identifier fields, and click Import.
Make changes to the live dashboard. You can do this by clicking the dropdown by the panel names, and selecting Edit.
Once you are happy with your changes, export your updated dashboard, and replace the existing content in the GrafanaDashbaord CR.
Open the dashboard, and select the Share... button.
Select the Export tab.
Click View JSON.
Click Copy to Clipboard.

Open the appropriate GrafanaDashboard CR file, and paste the new dashboard JSON over the existing. >:mag: NOTE
>Be sure to match the indentation of the previous dashboard JSON. Your git diffs should still show only the lines changed like the example below

    $ git diff charts/deploy/templates/metrics-dashboard.yaml
    diff --git a/charts/deploy/templates/metrics-dashboard.yaml b/charts/deploy/templates/metrics-dashboard.yaml
    index 73151ad..c470afc 100644
    --- a/charts/deploy/templates/metrics-dashboard.yaml
    +++ b/charts/deploy/templates/metrics-dashboard.yaml
    @@ -25,7 +25,7 @@ spec:
                "editable": true,
                "gnetId": null,
                "graphTooltip": 0,
    -            "id": 2,
    +            "id": 3,
                "links": [],
                "panels": [
                    {
    @@ -323,7 +323,7 @@ spec:
                    "tableColumn": "",
                    "targets": [
                        {
    -                    "expr": "count (deploy_timestamp)",
    +                    "expr": "count (count_over_time (deploy_timestamp [$__range]) )",
                        "format": "time_series",
                        "instant": true,
                        "intervalFactor": 1,
    @@ -410,7 +410,7 @@ spec:

You're done! Commit your changes and open a PR!

Exporter Development

A Pelorus exporter is simply a Prometheus exporter. While they can technically be written in many languages, we've written ours in Python using the Prometheus python client library. We chose Python because it seems to be the most popular programming language for operations teams.

Exporter directory layout

The following is a minimal directory structure for a Pelorus exporter.

exporters/exporter/
├── app.py
├── __init__.py
├── pelorus -> ../pelorus
├── README.md
└── values.yaml

Running locally

Running an exporter on your local machine should follow this process:

Install dependencies

pip install -r exporters/requirements.txt
pip install -r exporters/requirements-dev.txt

Set any environment variables required (or desired) for the given exporter (see Configuring Exporters to see supported variables).
```
export GIT_TOKEN=xxxx
export GIT_USER=xxxx
```

Log in to your OpenShift cluster

oc login --token=<token> --server=https://api.cluster-my.fun.domain.com:6443

(Optional) To avoid certificate warnings and some possible errors, you need to set up your local machine to trust your cluster certificate
1. Download your cluster ca.crt file
2. Add cert to system trust bundle
3. Pass cert bundle with your login command
```
oc login --token=<token> --server=https://api.cluster-my.fun.domain.com:6443  --certificate-authority=/etc/pki/tls/certs/ca-bundle.crt
```
Start the exporter
```
python exporters/committime/app.py
```

At this point, your exporter should be available at http://localhost:8080

curl http://localhost:8080

IDE Setup (VSCode)

Most of us use Visual Studio Code to do our python development. The following extensions for VSCode are useful. Each can be installed by hitting Ctrl+P and pasting the commands below.

Markdown Preview Github Styling
```
ext install bierner.markdown-preview-github-styles
```
- Python
  
  ext install ms-python.python

Code also comes with a nice debugger feature. Here is a starter configuration to use with our exporters. Just create a file called .vscode/launch.json in your pelorus/ project directory with the following content.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Commit Time Exporter",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/exporters/committime/app.py",
            "console": "integratedTerminal",
            "env": {
                "GITHUB_USER": "<github username here>", 
                "GITHUB_TOKEN": "<personal access token here>",
                "LOG_LEVEL": "INFO",
                "APP_LABEL": "app.kubernetes.io/name"
            }
        },
        {
            "name": "Deploy Time Exporter",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/exporters/deploytime/app.py",
            "console": "integratedTerminal",
            "env": {
                "LOG_LEVEL": "INFO",
                "APP_LABEL": "app.kubernetes.io/name"
            }        },
        {
            "name": "Deploy Time Exporter",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/exporters/failure/app.py",
            "console": "integratedTerminal",
            "env": {
                "SERVER": "<Jira server url>",
                "PROJECT": "<Jira project ID>",
                "USER": "<Jira username>",
                "TOKEN": "<Jira personal access token>",
                "LOG_LEVEL": "INFO",
                "APP_LABEL": "app.kubernetes.io/name"
            }
        }
    ]
}

For more information, see the Debugging doc in VS Code.

Testing Pull Requests

The following are notes and general steps for testing Pull Requests for specific types of changes.

Dashboard Changes

Clone/checkout the PR fork/branch git remote add themoosman git@github.com:themoosman/pelorus.git git fetch themoosman git checkout themoosman/feature-branch
Install Pelorus from checked out fork/branch. >:mag: Note
>:mag: In most cases you can deploy changes to an existing deployment to retain existing data.
Log into Grafana via the grafana route. oc get route grafana-route -n pelorus
Click on the dashboard containing changes, and visually validate the behavior change described in the PR >:mag: Note
>Eventually we'd like to have some Selenium tests in place to validate dashboards. If you have skills in this area let us know!

Exporter Changes

Most exporter changes can be tested locally.

Clone/checkout the PR fork/branch git remote add themoosman git@github.com:themoosman/pelorus.git git fetch themoosman git checkout themoosman/feature-branch
Install both the runtime and development dependencies. pip install -r exporters/requirements.txt pip install -r exporters/requirements-dev.txt
Run unit tests using python -m pytest.
You can also run coverage reports with the following: coverage run -m pytest coverage report
Gather necessary configuration information.
Run exporter localy. You can do this either via the command line, or use the provided VSCode debug confuration to run it in your IDE Debugger.
Once exporter is running, you can test it via a simple curl localhost:8080. You should be validating that:
You get a valid response with metrics.
Confirm the format of expected metrics.

Helm Install changes

For testing changes to the helm chart, you should just follow the standard install process, then verify that:

All expected pods are running and healthy
Any expected behavior changes mentioned in the PR can be observed.

We are in the process of refactoring our helm charts such that they can be tested more automatically using helm chart-testing. Some general guidelines are outlined in the CoP Helm testing strategy. More to come soon.

Release Management Process

The following is a walkthrough of the process we follow to create and manage versioned releases of Pelorus.

Create a lightweight release candidate tag from the master branch. The tag name should be the next sequential Semantic Version with the suffix -rc. Then push the new tag to the main repository.
```
git tag <version>-rc
git push -u upstream <version>-rc
```
1. Generate git release notes from the git log.
  
  git log .. --pretty=format:"- %h %s by %an" --no-merges 3. On the Pelorus releases page, click Draft a new release. * Select the tag that was pushed in the first step * The release title should be Release candidate for <version>. * In the main text area, create a # Release Notes heading, and then paste in the git log output. * Check the box that says This is a pre-release. * Click Publish Release. 4. Test. Test. Test. * Ensure that all github actions in the repo are passing. * Ensure that you can install Pelorus from the tagged version of the code, including all exporters and optional configurations. (Ensure you update the values.yaml file you are using to refer to the tagged version of the code in all builds) * Follow the above guidance for testing Pull requests. 5. If any bugs are found, open PRs to fix them, then delete the release and tag that was created, and start again from step 1. 6. Create an annotated tag for the final release with the -rc suffix removed.
  
  git checkout -rc git tag -a git push -u upstream 8. Generate git release notes from the git log.
  
  git log .. --pretty=format:"- %h %s by %an" --no-merges 7. On the Pelorus releases page, click Draft a new release. * Select the tag that was pushed in the previous step. * The release title should be Release <version>. * In the main text area, create a # Release Notes heading, and then paste in the git log output. * Click Publish Release.