A brief history Link to heading
The Gogs project has been proudly sponsored by DigitalOcean since 2015 via its Open Source Credits program. Benefited from that, the official website (https://gogs.io) and demo site (https://try.gogs.io) infrastructure has been hosted on DigitalOcean ever since.
For a very long time, we have been using a beefy machine (a DigitalOcean Droplet) to host everything we use, including the service itself, the data storage, the database, and almost everything that’s related.
Over the years, DigitalOcean has added a lot more features than we ever anticipated. Therefore, in the beginning of this year, we decided to get things to be state-of-the-art by revamping the project infrastructure setup, fully utilizing the amazing platform provided by DigitalOcean, as well as adopting the cool kid Infrastructure as Code (IaC) to replace hand-rolled scripts and playbooks.
In this post, I will walk you through how the new infrastructure of the Gogs project is set up with some details.
The grand scheme Link to heading
First of all, let’s present to you the often the most exciting and attracting part of an infrastructure setup, the architecture diagram!
As you can see, the actual infrastructure is fully managed by the IaC provider (in this case, the Pulumi, will get to that in a bit), all the way from the DNS, to the IP address of the droplet, the firewall, the container registry, the monitoring, the data storage, finally to the database backends. It’s truly amazing to see how many features we are able to get out of the DigitalOcean!
Continuous deployment Link to heading
On every commit to the main branch of the gogs/gogs repository, it builds and pushes a new Docker image to the DigitalOcean Container Registry:
...
- name: Login to DigitalOcean Container registry
uses: docker/login-action@v3
with:
registry: registry.digitalocean.com
username: ${{ secrets.DIGITALOCEAN_USERNAME }}
password: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
- name: Build and push images
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64,linux/arm/v7
push: true
tags: |
...
registry.digitalocean.com/gogs/gogs:latest
...
Then, we have a cron job setup on the droplet gogs-do-nyc3-01 to pull down the latest Docker image and restart the demo deployment. This makes sure we dogfood our commits in the fastest way possible to address any obvious show-stoppers if they ever occur.
Database connections Link to heading
The demo site takes full advantages of the DigitalOcean Managed Databases by using its managed PostgreSQL and Caching (which is actually Redis). The Gogs service is configured to connect to them via private connections through the VPC network, and TLS is always required for both of them:
[database]
TYPE = postgres
HOST = private-gogs-postgresql-nyc3-01-....ondigitalocean.com:...
NAME = gogs
SCHEMA = public
USER = ...
PASSWORD = ...
SSL_MODE = require
[session]
PROVIDER = redis
PROVIDER_CONFIG = network=tcp,addr=private-gogs-redis-nyc3-01-....ondigitalocean.com:...,password=...,tls=true
Infrastructure as Code Link to heading
Based on our research, Terraform and Pulumi are two-most popular providers and competitors in the IaC space. Because we have used Terraform extensively at work (and of course, encountered some dissatisfactions), and one of our former colleague strongly recommended Pulumi (because he was the founding engineer of Pulumi 😂), we decided to give Pulumi a try see how it compares to Terraform (this will be a separate blog post for another time).
Below is a quick peak of how the Pulumi stack look like for the Gogs project infrastructure:
$ pulumi stack -s default
...
Current stack resources:
TYPE NAME
pulumi:pulumi:Stack gogs-default
├─ digitalocean:index/uptimeCheck:UptimeCheck try.gogs.io-uptime-check
├─ cloudflare:index/record:Record gogs.io
├─ cloudflare:index/record:Record try.gogs.io-TXT-SPF
├─ cloudflare:index/record:Record gogs.io-AAAA
├─ cloudflare:index/record:Record dl.gogs.io
├─ cloudflare:index/record:Record dl.gogs.io-AAAA
├─ cloudflare:index/record:Record www.gogs.io
├─ cloudflare:index/record:Record crowdin.gogs.io
├─ cloudflare:index/record:Record email.mg.gogs.io
├─ cloudflare:index/record:Record email.try.gogs.io
├─ cloudflare:index/record:Record gogs.io-MX-1
├─ cloudflare:index/record:Record try.gogs.io-MX-1
├─ cloudflare:index/record:Record gogs.io-MX-2
├─ cloudflare:index/record:Record try.gogs.io-MX-2
├─ digitalocean:index/containerRegistry:ContainerRegistry gogs
├─ cloudflare:index/record:Record gogs.io-MX-3
├─ digitalocean:index/volume:Volume gogs-volume-nyc3-01
├─ cloudflare:index/record:Record admin._domainkey.gogs.io
├─ digitalocean:index/databaseCluster:DatabaseCluster gogs-postgresql-nyc3-01
├─ cloudflare:index/record:Record gogs.io-TXT-SPF
├─ digitalocean:index/databaseCluster:DatabaseCluster gogs-redis-nyc3-01
├─ cloudflare:index/record:Record krs._domainkey.try.gogs.io
├─ cloudflare:index/record:Record mg.gogs.io-TXT-SPF
├─ digitalocean:index/uptimeCheck:UptimeCheck gogs.io-uptime-check
├─ cloudflare:index/record:Record pic._domainkey.mg.gogs.io
├─ digitalocean:index/uptimeAlert:UptimeAlert try.gogs.io-uptime-alert
├─ digitalocean:index/droplet:Droplet gogs-do-nyc3-01
├─ digitalocean:index/databaseDb:DatabaseDb gogs-postgresql-nyc3-01-db-gogs
├─ digitalocean:index/uptimeAlert:UptimeAlert gogs.io-uptime-alert
├─ cloudflare:index/record:Record try.gogs.io-AAAA
├─ digitalocean:index/project:Project gogs
├─ digitalocean:index/reservedIp:ReservedIp gogs-demo-ip
├─ digitalocean:index/firewall:Firewall gogs-demo-firewall
├─ digitalocean:index/databaseFirewall:DatabaseFirewall gogs-redis-nyc3-01-firewall
├─ digitalocean:index/monitorAlert:MonitorAlert gogs-demo-monitor-cpu
├─ digitalocean:index/databaseFirewall:DatabaseFirewall gogs-postgresql-nyc3-01-firewall
├─ cloudflare:index/record:Record try.gogs.io
├─ digitalocean:index/monitorAlert:MonitorAlert gogs-demo-monitor-memory
├─ digitalocean:index/monitorAlert:MonitorAlert gogs-demo-monitor-disk
├─ pulumi:providers:digitalocean default_4_34_0
└─ pulumi:providers:cloudflare default_5_42_0
...
Because we do not use the Pulumi Cloud, we need to find a self-managed state backend for the Pulumi to work. Luckily, we can use DigitalOcean Spaces because it is S3-compatible. All we need to do is to initialize Pulumi using a “S3” backend:
aws --profile digitalocean configure set region "us-east-1" # Can be any region
aws --profile digitalocean configure set aws_access_key_id "{ACCESS KEY}"
aws --profile digitalocean configure set aws_secret_access_key "{SECRET KEY}"
pulumi login "s3://{BUCKET NAME}?endpoint=nyc3.digitaloceanspaces.com&profile=digitalocean"
We commit all the code (written in Go) into a GitHub repository, and implemented CI/CD for the Pulumi setup via GitHub Actions.
On every pull request, a pulumi preview will be ran to preview the diff:
...
jobs:
preview:
if: ${{ github.event_name == 'pull_request'}}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
name: Preview
permissions:
contents: read
pull-requests: write
strategy:
matrix:
project: [ "gogs" ]
runs-on: ubuntu-latest
steps:
...
- name: Set up AWS CLI
run: |
aws --profile digitalocean configure set region "us-east-1"
aws --profile digitalocean configure set aws_access_key_id "${{ secrets.DIGITALOCEAN_ACCESS_KEY }}"
aws --profile digitalocean configure set aws_secret_access_key "${{ secrets.DIGITALOCEAN_SECRET_KEY }}"
- name: Install Pulumi
uses: pulumi/actions@v6
- name: Pulumi preview
id: pulumi-preview
run: |
cd ${{ matrix.project }}
pulumi login --cloud-url "s3://{BUCKET NAME}?endpoint=nyc3.digitaloceanspaces.com&profile=digitalocean"
pulumi preview --stack default --diff --non-interactive |& tee preview.txt
echo 'PREVIEW<<EOF' >> "$GITHUB_OUTPUT"
cat preview.txt >> "$GITHUB_OUTPUT"
echo 'EOF' >> "$GITHUB_OUTPUT"
- name: Comment on PR
uses: thollander/actions-comment-pull-request@v3
with:
message: |
#### :tropical_drink: `preview` on ${{ matrix.project }}/default
<details>
<summary>Pulumi report</summary>
<pre>
${{ steps.pulumi-preview.outputs.PREVIEW }}
</pre>
</details>
comment-tag: execution
The diff output will also be posted to the pull request as a comment:
Upon merging the commit into the main branch, the pulumi up is ran to actually apply the changes:
jobs:
up:
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
name: Up
strategy:
matrix:
project: [ "gogs" ]
runs-on: ubuntu-latest
steps:
...
- name: Pulumi up
run: |
cd ${{ matrix.project }}
pulumi login --cloud-url "s3://{BUCKET NAME}?endpoint=nyc3.digitaloceanspaces.com&profile=digitalocean"
pulumi up --stack default --diff --non-interactive --skip-preview
Monitoring Link to heading
The last but not the least puzzle of an infrastructure setup is of course, the monitoring!
We have two types of monitoring checks and their alerts set up using DigitalOcean Monitoring:
- Resource monitoring and alerts
- Uptime monitoring and alerts
By having these alerts set up, we will be able to get timely notifications regarding any possible outage of the services we manage.
What’s next? Link to heading
While we have got the most our way to the grand scheme, there are still things left due to reasons, including:
- Set up Prometheus and Grafana Cloud for application monitoring
- Deploy Bytebase via DigitalOcean App Platform for managing and inspecting database backends with proper access control
- Implement the Secrets Storage to stop persisting API and access tokens all over the places
Stay tuned!