Under the Hood: Designing Rabetly’s Scalable Architecture

As traffic grows and usage patterns become more complex, a URL-shortening service must be rock-solid, lightning-fast, and easy to maintain. This article dissects the key building blocks that keep Rabetly efficient at scale.

1. Nginx at the Edge

Unified entry point: HTTP for bots (robots.txt, sitemap.xml) and HTTPS (443) for users.
Domain-based routing: Static files ↔ Ember dashboard ↔ redirect micro-service.
Security: Let’s Encrypt certificates auto-renewed via ACME DNS-01.

2. Gunicorn + Django

3 worker processes per instance, bound to a UNIX socket.
--capture-output sends all Python errors to /var/log/gunicorn/error.log.
Systemd ensures autorestart and uniform journaling.

3. Celery + RabbitMQ

Tasks for geo-IP lookup, click enrichment, email alerts.
Redis result backend for idempotent retries.

4. High-Volume Analytics with Kafka & TimescaleDB

Raw click events → Kafka topics → partitioned PostgreSQL.
Aggregated metrics live in TimescaleDB with continuous aggregates (sub-100 ms queries).
Cold partitions archived to S3.

5. Custom Domains & Hot-Reload SSL

New customer domain? A Cloudflare API call triggers DNS-01 validation; certificates drop in and Nginx hot-reloads without downtime.

6. Observability

Prometheus scrapes Nginx, Gunicorn, and Celery exporters.
Grafana dashboards; Alertmanager pages on 5xx spikes.
ELK stack unifies logs for full request tracing.

In Part 2 we’ll tackle schema optimisations and advanced hardening.