Forge
Self-hosted analytics that I actually own. A Go backend writing events into ClickHouse, a React dashboard on top, and everything deployed to my own Kubernetes cluster with Helm and Terraform.
Content
How it started
This started as Press, a long time ago: a CMS that collects the content from different sources like WordPress and serves it from one place. That project decayed and was never finished, but the idea is still around. It had different legs, and two of them were media and statistics.
Statistics was the leg I actually needed, so that's the one I started. The repository is still called press.io because of that. It ended up being more ambitious than the original idea, so at some point I rebranded everything: Press stayed as the content part and Forge is the whole thing.
The reason I needed it: I was using Plausible for the statistics of my sites and I liked it, but 500k events was around 49€/month. That was for I-Ching, a free app I built for fun, so paying that every month was never going to happen.
The trick that makes it cheap is that I don't keep the events. Saving every single one of them is madness, so I aggregate when the row is written and then the event is gone. That has caveats and it means I have to decide upfront what I want to measure, but it's the reason the same volume costs me a few euros instead of fifty.
Getting there was SLOW. The first version of the stats was built on refreshable materialized views and that was the wrong idea, so a good part of the early commits are migrations and rollbacks around that. I wrote about that in the ClickHouse page. Once the stats became insert-triggered, a benchmark inserting millions of events stayed stable in CPU and memory, and the queries came back immediately.
How it works
Events come in through a small public API, get pushed into a queue, and workers turn them into rows in ClickHouse. The dashboard never queries raw events: everything it shows comes from materialized views that are kept up to date as the data lands.
The backend is a handful of tiny binaries that share the same codebase, one cmd per process:
- api-public - the endpoint the outside world talks to.
- api-events - event ingestion, the only part that needs to be fast.
- api-internal - what the dashboard uses.
- worker-event-writer, worker-event-processor - the ingestion pipeline.
- worker-journeys, worker-insights, worker-report - the aggregations.
Each one is its own container with its own Helm chart, so I can give the ingestion path more memory than the report generator without touching anything else.
Why Go
The first version of the ingestion path was going to be NestJS, like everything else I write. Then I measured both, and the numbers are in the GoLang page. I'm paying for the machine myself, so that decided it.
Data
- ClickHouse for everything time-series: events, sessions, stats, journeys. Materialized views do the aggregation work at write time.
- MongoDB for the boring stuff: projects, users, configuration.
- (JetStream) as the queue between the API and the workers.
Valkey(a fork of Redis) for caching.
Infrastructure
Terraform drives Kubernetes and Helm, so every chart is a resource in the same plan. Two modules: the shared stateful services, and the app - instantiated once for staging and once for production.
Backups go to S3 on a cron, with a restore drill so I know they are real, and a Grafana dashboard ships as part of the deployment.
Status
Stats is the part that works and the one I use every day. The rest of Forge is still on the way:
- Studio - generate marketing assets from the content. Not available yet.
- Content - the
CMSfrom Press. Mostly a dream for now. - Text - translations for a codebase.
The current work is datasources: pulling metrics from Prometheus, Loki and Google Play so the same dashboard can show them next to the web analytics.
