ATProto Browser
Experimental browser for the Atmosphere
Experimental browser for the Atmosphere
{
"uri": "at://did:plc:5rms2ebhdngu24hgsu3s2hqd/com.whtwnd.blog.entry/3ldhukkf2il24",
"cid": "bafyreihtwb4mijhdefvmdah3b7m324csbo6yc7spyqvxclzkgrnv33lrxm",
"value": {
"$type": "com.whtwnd.blog.entry",
"theme": "github-light",
"title": "Nomadic cloudy adventures in PDS land",
"content": "Or how I managed to get the PDS to run as a Nomad job. With a few detours. And some cursing. Actually a lot of cursing. \n\n## A warning: don't do what I did, kids!\n\nTo preface this post, I messed up and managed to delete my *actual running PDS*'s data directory by way of fat-fingering the delete button on the cloud volume I was using for it. This means if you have no *rotationKeys* set on your did:plc you are straight out of luck, since the keys required to sign the operation to point your old identity to a new PDS are... as you guessed it, in that data directory. \n\nSo set up an additional *rotationKey* that you save in a safe place!\n\n## Assumptions!\n\nI'm assuming you can figure out what [Nomad](https://nomadproject.io), [Consul](https://consul.io), and [Vault](https://vaultproject.io) are. In case you have no idea what I'm talking about, feel free to read this post anyway but you may have to do a little on-a-tangent reading :) \n\n## Nomad. and why the hell am I doing this?\n\nOkay, good question! The reason I wanted to do it is because on the one hand, why not, and on the other hand, it'd be convenient for a project I've been tinkering on (slowly) for a while now. The end goal is to be able to offer EU based hosting for people to spin up their own PDS with a few clicks and a few questions. Also to run an EU based PDS where people can migrate to if they don't want to be on Bluesky's PDS's anymore. \n\nAnyway. Managing all that by hand is not a viable solution, and since at $orkplace we use Nomad in a professional capacity, I figured what the heck, I'll just do that for my happy fun time hobby projects too because at least I can re-use a lot of the tooling I've built over the years to benefit from it. \n\nI'll add that it's not just Nomad, but Consul and Vault are also very much present. \n\n## The Cluster(tm)\n\nThe current cluster runs the Nomad servers, Consul servers, and Vault servers. Nomad is for container orchestration, Consul for service discovery and configuration key/value store, and Vault is for secrets management (i.e. things stored in there are encrypted, and it allows my Nomad jobs to \"not know a goddamn thing\"). \n\nFor now there are 2 additional nodes that act as Nomad agents, where workloads are scheduled. One ingress node, and one node that runs workloads. This is supported by a bunch of S3 compatible object storage, cloud volumes, and a load balancer that does all the TLS termination before forwarding all traffic to the ingress node. \n\nI use [Hetzner Cloud](https://www.hetzner.com/cloud) for all hosting, because really, AWS is just not something my wallet will support. And it's in the EU. And it's not an American company. And for many reasons that makes me happy. \n\n## Ingress\n\nFor ingress, I use [Traefik](https://traefik.io/traefik/) - I know the PDS distribution as we get it from Bluesky uses Caddy, but Traefik is what I've been using for a while now and I know how to get it to do what I want. It also lets me do some fun shenanigans behind the scenes that I'm experimenting with (think \"walled garden\"). \n\n## The PDS docker image\n\nWhen you download the PDS installer from Bluesky, it actually sets up 3 containers, and does some wonky things with systemd. In Nomad, i don't need 3 containers (pds, caddy, watchtower), I just need the pds container. Fixed easily enough, just look at the *compose.yml* and pull out the definition. One thing it lacks is a port mapping, so for those of you wondering, the PDS image listens on port 3000. \n\n## Making it work \n\nNomad operates with *jobs* - a job is comprised of some meta information, and 1..n task groups. Each task group can have 1..n tasks in it. All tasks in a task group are allocated to the same agent (physical server). Sounds complicated, but isn't. And in this case fully unnecessary to dive into that deeper since the PDS job is a 1-group-1-task type thing, so simple. For sheets and geegles I've included the job spec (with comments) below:\n\n```\njob \"pds-peedee\" {\n # there can be multiple datacenters in a region, this indicates I don't care where it ends up\n datacenters = [\"*\"]\n namespace = \"default\"\n type = \"service\"\n\n # this will ensure this job only runs on a Nomad agent of the specified class. If I had 324 of these, it'd pick the one that would satisfy\n # the schedulers' preference of where to put it.\n constraint {\n attribute = \"${node.class}\"\n value = \"pds_4g_2c\"\n }\n \n group \"pds\" {\n # this isn't really used when you have only 1 instance of a task group running, but\n # it does mean if I fat-finger the job spec into not working, it will revert back to a previous one. It also won't consider\n # the tasks in this group healthy unless they've been reported as such for at least 10 seconds, and they have 1 minute\n # to actually get there.\n update {\n max_parallel = 1\n canary = 1\n min_healthy_time = \"10s\"\n healthy_deadline = \"1m\"\n auto_revert = true\n auto_promote = true\n } \n # this is a Container Storage Interface sourced volume; this is basically a Hetzner cloud volume that I'm pulling in\n volume \"storage\" {\n type = \"csi\"\n source = \"pds-peedee\"\n attachment_mode = \"file-system\"\n access_mode = \"single-node-writer\"\n read_only = false\n } \n # network settings; bridge mode creates a little tiny slice of private network on the nomad agent. the DNS servers are pointing\n # to Docker's bridge IP so that I can use Consul's DNS (which I bound there).\n network {\n mode = \"bridge\"\n port \"pds\" {\n to = 3000\n } \n dns {\n servers = [ \"172.17.0.1\" ]\n } \n } \n # here's where the fun begins; this is a service definition that gets\n # registered in Consul with a health check, and various tags. The tags are used\n # by Traefik to decide what requests to route where.\n service {\n port = \"pds\"\n check {\n name = \"PDS port listening\"\n type = \"http\"\n path = \"/xrpc/_health\"\n interval = \"60s\"\n timeout = \"10s\"\n }\n tags = [\n \"traefik.enable=true\",\n \"traefik.http.routers.pds-peedee.entrypoints=https\",\n # the almighty routing rule: this lets me host another app or site on https://peedee.es - only pass \n # to the PDS the things that concern the PDS basically. Or render unto... you know the rest. Something\n # with Caesar. \n \"traefik.http.routers.pds-peedee.rule=(Host(`peedee.es`) || HostRegexp(`^.+\\\\.peedee\\\\.es$`)) && (PathPrefix(`/xrpc/`) || PathPrefix(`/.well-known/`) || PathPrefix(`/oauth/`) || PathPrefix(`/@`))\",\n ] \n } \n task \"pds\" {\n driver = \"docker\"\n\n # here we say we need to use the PDS port (which above was specified that the container listens on port 3000, and Nomad\n # will allocate a port on a chosen (private) network interface dynamically. \n # force_pull will ensure that any time we restart this we get the latest and greatest 0.4 version \n config {\n ports = [ \"pds\" ]\n image = \"ghcr.io/bluesky-social/pds:0.4\"\n force_pull = true\n }\n \n # this sets up a Workload Identity JWT that allows us to access our Vault \n vault {}\n\n # this actually maps the above included CSI volume into the container filesystem\n volume_mount {\n volume = \"storage\"\n destination = \"/pds\"\n read_only = false\n }\n\n # well, instead of an env file we get an env block. Some values maybe redacted. \n env {\n PDS_HOSTNAME=\"peedee.es\"\n PDS_DATA_DIRECTORY=\"/pds\"\n PDS_DID_PLC_URL=\"https://plc.directory\"\n PDS_BSKY_APP_VIEW_URL=\"https://api.bsky.app\"\n PDS_BSKY_APP_VIEW_DID=\"did:web:api.bsky.app\"\n PDS_REPORT_SERVICE_URL=\"https://mod.bsky.app\"\n PDS_REPORT_SERVICE_DID=\"did:plc:ar7c4by46qjdydhdevvrndac\"\n PDS_CRAWLERS=\"https://bsky.network\"\n LOG_ENABLED=\"true\"\n PDS_CONTACT_EMAIL_ADDRESS=\"ping@peedee.es\"\n PDS_EMAIL_FROM_ADDRESS=\"noreply@peedee.es\"\n PDS_INVITE_REQUIRED=\"true\"\n PDS_BLOB_UPLOAD_LIMIT=\"52428800\"\n PDS_BLOBSTORE_S3_BUCKET=\"supersecretbucketname\"\n PDS_BLOBSTORE_S3_REGION=\"fsn1\"\n PDS_BLOBSTORE_S3_ENDPOINT=\"https://fsn1.your-objectstorage.com\"\n PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=\"true\"\n PDS_PRIVACY_POLICY_URL=\"https://peedee.es/d/privacy/\"\n PDS_TERMS_OF_SERVICE_URL=\"https://peedee.es/d/terms/\"\n PDS_HOME_URL=\"https://peedee.es/\"\n }\n # this is where all those juicy, juicy secrets come in from Vault. A template is rendered using values taken from\n # Vault's secure key/value store, and the \"env=true\" bit at the end means the rendered result is merged\n # with the env block above. \n template {\n data = <<EOH\n{{ with secret \"secret/data/application/pds-peedee\" }}{{with .Data}}\nPDS_JWT_SECRET={{ .data.jwt_secret }}\nPDS_ADMIN_PASSWORD={{ .data.admin_password }}\nPDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX={{ .data.rotation_key_hex }}\nPDS_EMAIL_SMTP_URL={{ .data.smtp_url }}\nPDS_BLOBSTORE_S3_ACCESS_KEY_ID={{ .data.s3_access_key_id }}\nPDS_BLOBSTORE_S3_SECRET_ACCESS_KEY={{ .data.s3_secret_access_key }}\n{{ end }}{{ end }}\nEOH\n destination = \"${NOMAD_SECRETS_DIR}/pds-secrets.env\"\n env = true\n }\n # and here I tell it how much memory and CPU it can use. Memory is actually an enforced limit,\n # whereas CPU isn't.\n resources {\n cpu = 1024\n memory = 1024\n }\n }\n }\n}\n```\n\n## Caveats and other such things\n\nThis is of course hilariously over-complicated if you just want to run your own PDS for yourself. I also have the routing rule from hell in this job because I need to be able to share the same domain for another website, *and* the PDS. \n\nIf it's dedicated to just your PDS the routing rule becomes much simpler. \n\n## Closing remarks\n\nSince I wrote this at 4:30am I'll apologise if it doesn't seem to make much sense. But I wanted to write this anyway because hey why not words words words! Anyway. Mostly just put into a blog post to demonstrate how to run a PDS in a Nomad job. Might help if you ever (god have mercy on your soul) want to do it with Kubernetes. \n\nThe configuration of the PDS is also still subject to various tweaks, so don't just copy/paste what's there. Actually don't do that anyway because you aren't me. I think. ",
"createdAt": "2024-12-17T03:52:21.967Z",
"visibility": "public"
}
}