Exo Mercado — Writing

Goodbye PEM files: moving to RDS IAM database authentication

Mon, 13 Jul 2026 00:00:00 GMT

For years, connecting to our production database looked like this: find the shared .pem file (usually in someone's Slack DMs), SSH into a bastion host, open a tunnel to the private RDS instance, and paste in a database password that hadn't changed since the instance was created. It worked, which is exactly why nobody fixed it. This is how I finally replaced it with RDS IAM database authentication, and what I'd tell past me before starting.

The old setup, and why it aged badly

The database lived in a private subnet, as it should. The only way in was an SSH tunnel through a bastion:

ssh -i team-bastion.pem -N -L 5432:mydb.abc123.ap-southeast-1.rds.amazonaws.com:5432 ec2-user@bastion.example.com

Humans were only half of it. Our Go services run on Lambda, and the functions needed the same path in, so the key was baked straight into the deployment artifact and the tunnel was opened in code:

//go:embed team-bastion.pem
var bastionKey []byte

func connect(ctx context.Context) (*pgx.Conn, error) {
	signer, err := ssh.ParsePrivateKey(bastionKey)
	if err != nil {
		return nil, err
	}

	bastion, err := ssh.Dial("tcp", "bastion.example.com:22", &ssh.ClientConfig{
		User:            "ec2-user",
		Auth:            []ssh.AuthMethod{ssh.PublicKeys(signer)},
		HostKeyCallback: ssh.InsecureIgnoreHostKey(), // yes, really
	})
	if err != nil {
		return nil, err
	}

	cfg, err := pgx.ParseConfig(os.Getenv("DATABASE_URL")) // static password inside
	if err != nil {
		return nil, err
	}
	cfg.DialFunc = func(ctx context.Context, network, addr string) (net.Conn, error) {
		return bastion.Dial(network, addr)
	}
	return pgx.ConnectConfig(ctx, cfg)
}

Every invocation paid for an SSH handshake before it could run a query, and every function in the account carried a copy of the key to production.

The problems weren't hypothetical. We hit all of them:

One key, many people. The .pem file was shared, so rotating it meant coordinating with everyone at once. So we didn't rotate it.
The key shipped with every build. Because the Lambdas embedded it, rotating the key also meant rebuilding and redeploying every function that carried a copy. One secret, dozens of deployment artifacts.
Offboarding was a lie. When someone left, they kept a working copy of the key unless we rotated it (see above).
No audit trail. Every connection came from ec2-user through the same tunnel. CloudTrail and the database logs both said the same thing: someone connected. Great, thanks.
The bastion itself. One more EC2 instance to patch, monitor, and pay for, whose entire job was being a hole in the network.
Static database credentials. The app password and the human password were both long-lived strings sitting in password managers and, let's be honest, a few shell histories.

How RDS IAM auth works

The idea is simple: instead of a password, the database accepts a short-lived token signed by AWS. Your IAM identity becomes your database identity. There are four pieces.

1. Enable IAM auth on the instance. A checkbox in the console, or:

aws rds modify-db-instance \
  --db-instance-identifier mydb \
  --enable-iam-database-authentication \
  --apply-immediately

2. Create a database user mapped to IAM. For Postgres, you grant the rds_iam role. For MySQL you'd create the user with AWSAuthenticationPlugin instead.

CREATE USER exo_mercado WITH LOGIN;
GRANT rds_iam TO exo_mercado;
GRANT readonly TO exo_mercado; -- normal grants still apply

3. Attach an IAM policy allowing rds-db:connect for that user. Note the resource ARN uses the DbiResourceId (starts with db-), not the instance name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds-db:connect",
      "Resource": "arn:aws:rds-db:ap-southeast-1:123456789012:dbuser:db-ABCDEFGHIJKL01234/exo_mercado"
    }
  ]
}

4. Generate a token and use it as the password. The token is a presigned request, valid for 15 minutes:

TOKEN=$(aws rds generate-db-auth-token \
  --hostname mydb.abc123.ap-southeast-1.rds.amazonaws.com \
  --port 5432 \
  --username exo_mercado)

PGPASSWORD=$TOKEN psql \
  "host=mydb.abc123.ap-southeast-1.rds.amazonaws.com port=5432 dbname=app user=exo_mercado sslmode=verify-full sslrootcert=global-bundle.pem"

Yes, there's still a .pem file in that command. It's the public RDS CA bundle, not a secret, and I find that funny.

The Go side: Lambda through RDS Proxy

For the Lambdas we didn't point pgx at the instance directly. Lambda churns connections by design, so we put RDS Proxy in front and enabled IAM auth on the proxy. The proxy holds the warm pool and speaks to the database using credentials it keeps in Secrets Manager; our code only ever authenticates to the proxy with a token. Two details matter here:

The rds-db:connect policy for a proxy uses the proxy id (starts with prx-) in the resource ARN, not the instance's db- id, and it goes on the Lambda's execution role. The function itself holds no credentials at all.
Tokens expire in 15 minutes, so you generate one per new connection, not one per deploy. pgx has the perfect hook for this: BeforeConnect.

const proxyEndpoint = "app-proxy.proxy-abc123.ap-southeast-1.rds.amazonaws.com:5432"

func newPool(ctx context.Context) (*pgxpool.Pool, error) {
	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		return nil, err
	}

	cfg, err := pgxpool.ParseConfig(
		"postgres://app_service@" + proxyEndpoint + "/app?sslmode=verify-full")
	if err != nil {
		return nil, err
	}
	cfg.BeforeConnect = func(ctx context.Context, cc *pgx.ConnConfig) error {
		token, err := auth.BuildAuthToken(
			ctx, proxyEndpoint, awsCfg.Region, cc.User, awsCfg.Credentials)
		if err != nil {
			return err
		}
		cc.Password = token
		return nil
	}
	return pgxpool.NewWithConfig(ctx, cfg)
}

auth.BuildAuthToken comes from github.com/aws/aws-sdk-go-v2/feature/rds/auth and does the same thing as generate-db-auth-token, using whatever credentials the Lambda role provides. Every fresh connection in the pool gets a fresh token, so expiry never bites. Compare this to the SSH version above: no embedded key, no tunnel, no static password, and about half the code.

One pleasant surprise: the proxy endpoint presents a certificate that public roots already trust, so verify-full works without shipping the RDS CA bundle in the artifact.

The gotchas

Tokens expire in 15 minutes. That's the point, but it means anything that reconnects needs to generate a fresh token each time, not cache one in an env var. In Go, BeforeConnect handles it (see above). For humans, a small wrapper script around psql does the same job.

SSL is required. Connections without TLS are rejected outright. Set sslmode=verify-full and ship the CA bundle; don't settle for require.

Auth rate limits are real. IAM auth adds work per connection, and AWS documents a connection-rate ceiling. For humans running psql this is irrelevant. For Lambda it absolutely is not, which is the other reason the functions go through RDS Proxy: the database sees a stable pool instead of a stampede of token validations. If a proxy is overkill for your case, a pooler like PgBouncer covers the same ground.

You still need a network path. IAM auth replaces the password, not the VPC. The database is still in a private subnet, so you need to reach it somehow: client VPN, SSM port forwarding to any instance in the VPC, or just running from workloads already inside the VPC. We went with SSM port forwarding for humans, which killed the bastion too, since SSM needs no open inbound ports and no SSH keys at all.

What actually improved

No shared secret. There's no PEM file for the database and no password to pass around. Access is your IAM identity, full stop.
Per-user audit trail. exo_mercado connects as exo_mercado. Database logs and CloudTrail finally agree on who did what.
Credentials expire on their own. A leaked token is worthless in 15 minutes. Rotation stopped being a project and became a property.
Offboarding is one action. Disable the IAM user and every database they could reach is closed to them, immediately, without touching the database.
Deploys stopped carrying secrets. The Lambda artifact is code and nothing else. Rotating anything no longer touches a build pipeline.
One less server. The bastion is gone, along with its patch schedule.

The migration itself took an afternoon per database, most of it spent standing up the proxy, swapping the tunnel code for BeforeConnect, and updating runbooks. The hardest part was believing it would be that small. If you're still passing a .pem file around to reach RDS, this is one of the rare security upgrades that makes daily life easier instead of worse.

Building RBAC from scratch

Tue, 30 Jun 2026 00:00:00 GMT

A few years ago I built the access-control layer for an internal enterprise platform from the ground up — no off-the-shelf authorization library, just Go, DynamoDB, and a model we could actually reason about. This is what I learned.

The platform served at least 500 users, and access control had to stay flexible: roles assigned to users on the fly, with the permissions on each role adjustable per module.

Why build it instead of reaching for a library

There are good authorization libraries out there, and for most projects I'd use one. We didn't, for two reasons.

First, our permissions were tied to systems we were integrating — Azure AD for identity, plus JIRA and Bitbucket for the actual work. A generic library would have meant bending our domain to fit its model anyway, so the "free" part wasn't really free.

Second, authorization is the one place I want zero magic. When someone asks "why can this user do that?", I need to answer it by reading code, not by tracing through a framework's policy engine. Building it ourselves kept the whole decision path in plain sight.

The model

I kept it deliberately boring. Three nouns:

Permissions are the atoms — a verb on a resource, like repo:read or user:invite. Nothing checks a role directly; everything checks a permission.
Roles are named bundles of permissions (admin, maintainer, viewer).
Assignments bind a user to a role, optionally scoped to a team or project.

The single most important decision was that code only ever checks permissions, never roles. Roles are a convenience for humans assigning access; the enforcement layer doesn't know they exist. That one rule is what kept the system from rotting: when product asked for "let maintainers do X too," it was a one-line change to a role's permission set, not a hunt through the codebase for every if role == "maintainer".

Storing it in DynamoDB

DynamoDB rewards you for knowing your access patterns up front, so I started there. The hot path is always the same question: given this user, what can they do? So role assignments are keyed by user, and resolving a user's effective permissions is a single query plus a couple of cheap role lookups that are easy to cache.

The trap I almost fell into was modelling permissions as something you compute with a scan. You never want authorization to depend on a table scan — it's the one query that runs on every request, and it has to stay fast and predictable. Keep the "what can this user do" lookup down to a key-based read and you're fine.

Enforcing it in Go

Enforcement lived in one place: middleware. The handler declares the permission it needs, and the middleware resolves the caller's effective permissions and either continues or returns a 403.

func RequirePermission(p Permission) Middleware {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            user := UserFromContext(r.Context())
            if !user.Permissions.Has(p) {
                http.Error(w, "forbidden", http.StatusForbidden)
                return
            }
            next.ServeHTTP(w, r)
        })
    }
}

The win here isn't the code — it's that there's exactly one code path that grants or denies. No handler does its own ad-hoc check. If authorization has a bug, I know the one file to open.

The gotchas nobody warns you about

Caching effective permissions is a footgun. You'll want to cache the resolved permission set because it's read constantly — but the moment you do, revoking access stops being instant. I cached with a short TTL and an explicit invalidation when a user's assignments change. Decide your staleness budget on purpose; don't let it be an accident.

The frontend hides, the backend enforces. It's tempting to treat a hidden button as security. It isn't. The UI consumes the same permission set purely to decide what to show — every real decision still happens on the server. Getting this boundary right is most of the battle.

Roles multiply if you let them. The first time a request doesn't fit an existing role, the lazy fix is a new role. Do that enough and you've got forty near-identical roles nobody understands. Because we checked permissions and not roles, we could usually adjust an existing role instead of minting a new one.

Deny-by-default, always. The absence of a permission is a "no." New resources and new endpoints start locked, and access is something you add deliberately. The inverse — open by default, lock things down later — is how you end up with a breach.

{/* TODO(exo): drop in a real story here — a specific bug, a tricky permission edge case, or a moment the deny-by-default rule saved you. This is the part readers remember. */}

Would I do it again?

For that platform, yes. The model was small enough to hold in my head, the enforcement path was a single file, and "why can this user do that?" was always answerable. If your authorization rules are genuinely complex — hierarchies, attribute-based conditions, relationship graphs — reach for something purpose- built instead. But for plain role-based access, hand-rolling it bought us clarity that was worth far more than the few days it took to write.

The whole thing comes down to one habit: check permissions, not roles, and deny by default. Get that right and the rest is just plumbing.

Colophon — how this site is built

Mon, 29 Jun 2026 00:00:00 GMT

I like reading the colophon on other people's sites, so here's mine. The design has a name, Signal Path, and the idea behind it is a page that behaves like a circuit diagram: signals routed along clean paths, nothing on screen that isn't also doing a job.

The stack

Still static Astro, still zero JavaScript shipped by default. Pages are plain .astro; this writing lives in MDX through typed content collections, so a post with broken frontmatter fails the build instead of shipping.

Color and type

Styling is Tailwind CSS v4, configured CSS-first. There's no config file; the tokens live in an @theme block in global.css.

The palette is called Copper & Carbon. Carbon (#0C1014) is the ground. Copper (#DE8B52) is the one accent that means "active" or "interactive," and verdigris (#5FB8A3) is the second signal, reserved for "steady" status. Dark is the canonical mode. There's a light "datasheet" mode too, but the whole site was designed dark-first, and the light theme reads like a printed spec sheet.

Type does real work here. Archivo (variable) is the display face, and its width axis actually animates. IBM Plex Sans carries the body copy. JetBrains Mono is the machine voice: labels, timestamps, anything the site says in its own robotic register.

The hero

The hero is a Canvas 2D service mesh. Nodes and packets drift around, pulled toward your cursor like it has a bit of gravity. As you scroll, the mesh collapses along a line I call the Seam, which folds down into the fixed rail on the left. A scattered network becoming one clean path is the whole Signal Path idea in a single gesture.

The name

My name doesn't just fade in, it decodes. The glyphs flicker through random characters, then settle, while Archivo's width axis widens them into place. There's a subtle shimmer over the top, drawn in WebGL with OGL, just enough to catch the light.

The command palette

Hit ⌘K (or Ctrl+K) for a terminal-style command palette: jump around, flip the theme, copy my email, find the easter egg. It's the fastest way through the site if you'd rather keep your hands on the keyboard.

Moving between pages uses Astro's View Transitions, so navigation animates instead of blinking white.

The parts you don't see

Every animation checks prefers-reduced-motion first. If you've asked your OS to calm things down, or JavaScript never runs, you still get the whole page, finished, with nothing hidden. And the color tokens run through a build-time contrast check: if any text-and-background pair drops below WCAG AA, the build fails. Accessibility that only holds when someone remembers to check it isn't accessibility.

That's the tour. Thanks for reading the colophon.