Migrate to the Pixelfed + Docker Compose v2

There are a lot of changes in how Pixelfed Docker/Docker Compose images work - it's a complete rewrite - with a couple of breaking/significant changes.

But don't worry! This document and the Migration Guide further down covers all of the required changes and steps you need to safely migrate.

We don't take breaking changes or lengthy migration guides lightly, and future iterations and improvements will be either seamless or much smaller in scope and complexity.

We do however believe that the many improvements (listed below) are worth it, and sets us up for a brighter future for self-hosted and maintainable Pixelfed servers

Breaking Changes

These changes are breaking, removing, or changing existing behavior from Docker Compose v1 and requires your attention and possibly some steps to fix.

The Migration Guide has step by step guide for most of these!

New `Dockerfile` Breaking

All runtime variations of Pixelfed on Docker are now build from a single Dockerfile, rather than the previously three.

Please see the Docker Runtimes and Docker Customization documentation for more information.

New Docker Compose file Breaking

The docker-compose.yml file has been rewritten from the ground up to be provide better defaults and more flexible out of the box.

This mean that most things you would likely want to tweak can be controlled from your .env config file, like installing additional APT Packages, PECL/PHP extensions, changing PHP version, modifying PHP settings (such as memory_limit), disabling specific services, changing where data and config files are stored, and so on.

New `.env` / `.env.docker` file Breaking

The starter .env.docker file has been significantly expanded with most of the Pixelfed settings available, along with many Docker Compose specific ones.

Your Pixelfed specific settings are (of course) unchanged, but you need to configure them again after copying the new .env.docker file.

Using `bind` volumes Breaking

The old docker-compose.yml configuration file declared four anonymous Docker volumes for storing Pixelfed related data within.

These are no longer used, instead favoring a Docker bind volume approach where content is stored directly on the server disk, outside of a Docker volume.

The consequence of this change is that all data stored in the - now unsupported - Docker volumes will no longer be accessible by Pixelfed.

The db-data volume definitely contains important data - it's your database, after all!
The app-storage volume definitely contains important data - it's files uploaded to - or seen by - your server!
The redis-data volume might contain important data (depending on your configuration)
The app-bootstrap volume does not contain any important data - all of it will be generated automatically in the new setup on startup. We will not be migrating this!

Please see the Migration steps section for information on moving your data.

No Docker networks Breaking change

The docker-compose.yml no longer include any custom networks, instead favoring the simplicity of the default bridge network.

The published ports can be configured via your .env file, look for the DOCKER_*_HOST_PORT keys.

Changes needing attention

These changes are mostly informational and unlikely to cause any issues during upgrade, but included for visibility and in case your specific set up require them to be changed.

New Redis version Attention needed

Redis has been upgraded from using the last version 5 release to using the latest 7.2 version instead - favoring the debian variant instead of alpine.

The Redis version can be controlled in your .env file via DOCKER_REDIS_VERSION.

If you want to keep using the Redis version set DOCKER_REDIS_VERSION="5-alpine" in your .env file.

If you want to use 7.2 but the alpine variant, simply append -alpine to the DOCKER_REDIS_VERSION (e.g. DOCKER_REDIS_VERSION=7.2-alpine).

Pinned MariaDB version Attention needed

The included db service is now pinned to MariaDB 11.2 instead of latest.

You can revert to the previous setting by setting DB_VERSION="latest" in your .env file.

Automatic "One-time setup tasks" Attention needed

This is also covered in the migration guide but existing Pixelfed instances need to disable the automatic run of One-time setup tasks by setting DOCKER_APP_RUN_ONE_TIME_SETUP_TASKS=0 in your .env file.

Your container will fail to start up if these are already configured for your site.

New or improved features

All the new and exciting features and capabilities. 🚀

This is where we hope we can convince you that the breaking changes and migration work was worth it ❤️

Nginx Proxy service New

The new docker-compose.yml includes an optional (but enabled by default) Nginx Proxy for SSL/TLS termination.

Please see How do I use my own Proxy server? and DOCKER_PROXY_* keys in the .env file for more information.

LetsEncrypt/ACME service New

The new docker-compose.yml includes an optional (but enabled by default) ACME/LetsEncrypt service that when combined with the new Nginx Proxy automatically creates and maintains your SSL/TLS certificates.

Please see How do I use my own SSL certificate? and DOCKER_PROXY_* + LETSENCRYPT_* keys in the .env file for more information.

Automatic run of "One-time setup tasks" New

When you set up a new Pixelfed server, there is handful of commands you need to run once (and only once) - these are called One-time setup tasks.

These steps are now automatically run for you when creating a new Pixelfed server.

Extensive documentation New

How to run and use the new Docker setup has been documented extensively. Not only on this page, but also in all the new scripts, Dockerfile, and .env.docker file.

We hope this will make it much easier to confidently and comfortably run your Pixelfed server with Docker - and when things do go wrong, the debugging and fixing of the issue much easier and quicker.

Automatic publishing of Docker images Improved

Going forward we will automatically build and push Docker images for Pixelfed to both Docker Hub and GitHub Container Registry.

We will be automatically pushing all combinations of our supported Docker runtimes.

Further more, testing Pull Requests has never been easier, as we will also build and push Docker images for all Pull Requests with tag prefix {RUNTIME}-pr-{ID} (e.x. gcr.io/pixelfed-glitch/pixelfed:apache-pr-4844).

You can control which Pixelfed release you use via the DOCKER_APP_RELEASE key/value pair in your .env file.

Customizable `Dockerfile` New

It's now possible to tweak many Docker related settings without copying or forking Pixelfed Dockerfile!

The new Dockerfile has many Build Arguments (e.g., --build-arg) allowing you to easily

All of these (and more) settings are controlled from your .env file, and you simply need to run docker compose build to build your own bespoke version of Pixelfed on your server!

Customizable `ENTRYPOINT` New

Do you want to run a script on container start up? Or perhaps to disable a specific init script? Or even disable all of the init scripts?

Now you can! And like with all the other settings it's just a setting in your .env file!

Templating files New

The new ENTRYPOINT system offers an extensible and flexible way for you to template configuration files such as your php.ini during container start up.

The templating system has access to all settings from your .env file, and you can easily add your own templates!

Automatic PHP/Web server configuration New

Thanks to the new templating system we now automatically calculate and configure required PHP / Web server settings for you - using your .env settings - such as

(php.ini) upload_max_filesize with this formula (MAX_PHOTO_SIZE * MAX_ALBUM_LENGTH) + BUFFER
(php.ini) post_max_size with this formula (MAX_PHOTO_SIZE * MAX_ALBUM_LENGTH) + BUFFER
(php.ini) max_file_uploads using MAX_ALBUM_LENGTH
(php.ini) memory_limit using PHP_MEMORY_LIMIT
(php.ini) [Date]date.timezone using APP_TIMEZONE

Permission auto-fixing New

If your installation for some reason has issues with permissions to files/directories, you can now automatically fix ownership and permissions on start up.

This is an opt-in feature.

Faster Docker image building Improved

We now utilize BuildKit, layer caching, multi-stage, and multi-platform to really speed up the building of Docker images.

Under ideal conditions, a docker compose build can now complete in less than a minute for most PHP changes. This of course also improve the speed of building Docker images in GitHub Actions and CI!

Fork friendly Docker releasing New

Forks of Pixelfed Glitch that have enabled GitHub Actions should automatically have the same Docker build + push experience as pixelfed-glitch/pixelfed have.

The docker workflow will by default build + push to the GitHub Container Registry for the project the commit was made to, meaning any downstream projects will not have to fork or modify their docker workflow to have a reliable release process.

Forks can further more set GitHub Actions Project variables DOCKER_HUB_USERNAME, DOCKER_HUB_ORGANISATION, DOCKER_HUB_REPO and DOCKER_HUB_TOKEN for automatic pushing of images to Docker Hub as well. Please see the .github/workflows/docker.yml file for more information.

Migration steps

INFO

This is a best-effort guide to help migrate off the old system. The operation is potentially complicated (and risky), so please be careful!

DANGER

PLEASE MAKE SURE TO BACKUP YOUR SERVER AND DATA BEFORE ATTEMPTING A MIGRATION

YOUR INSTANCE WILL BE DOWN WHILE DOING THE MIGRATION; PLEASE PLAN ACCORDINGLY; DEPENDING ON DATA SIZE IT COULD TAKE ANYWHERE FROM 5 MINUTES TO 5 HOURS

1) Backup

Make sure to back up your server (ideally after step 1 below has been completed, but before is better than not at all!)
Capture the current Git version / Pixelfed release you are on (e.g., git --no-pager log -1 outputs the commit reference as the 2nd word in the first line)
Backup your .env file (we will do this in step 3 as well)
Backup your docker-compose.yml file (cp docker-compose.yml docker-compose.yml.old)
Read through the entire document before starting

2) Migrate `.env` file

The new .env file for Docker is a bit different from the old one (many new settings!) so the easiest is to grab the new .env.docker file and modify it from scratch again.

bash

cp .env .env.old
wget -O .env.new https://raw.githubusercontent.com/pixelfed-glitch/pixelfed/main/.env.docker

Then open your old .env.old configuration file, and for each of the key/value pairs within it, find and update the key in the new .env.new configuration file.

Don't worry, though; the file might look different (and significantly larger), but it behaves exactly the way the old file did; it just has way more options!

TIP

Don't worry if a key is missing in .env.new; you can add those key/value pairs back to the new file - ideally in the Other configuration section near the end of the file - but anywhere should be fine.

This is a great time to review your settings and familiarize yourself with all the new ones.

INFO

In particular the following sections

PHP configuration section (near the end of the file) where
- The DOCKER_APP_PHP_VERSION settings control your PHP version
- The PHP_MEMORY_LIMIT settings control your PHP memory limit
Docker Specific configuration section (near the end of the file) where
- The DOCKER_ALL_HOST_DATA_ROOT_PATH setting dictates where the newly migrated data will live.
- The DOCKER_APP_RUN_ONE_TIME_SETUP_TASKS controls whether the One-time setup tasks should run. We do not want this since your Pixelfed instance is already set up!
Frequently Asked Question / FAQ
- How do I use my own Proxy server?
- How do I use my own SSL certificate?

3) Stop containers

WARNING

This will take your Pixelfed instance offline

Stop all running containers (web, worker, redis, db)

bash

docker compose down

4) Update source code

Update your project to the latest release of Pixelfed by running:

bash

git pull origin $release

INFO

The $release can be any valid git reference like develop, staging, latest, or a tagged release such as v0.12.0-glitch.1.4.0.

5) Migrate data

The migration guide temporarily branches into two paths here:

Path A if you used Docker Anonymous Volumes.
Path B if you used host/bind volumes.

Which Docker volume type am I using?

If your old docker-compose.yml had a volumes configuration section like below, you used should follow Path A.

This is also what the v1 docker-compose.yml file for pixelfed/pixelfed used.

yaml

volumes:
  db-data:
  redis-data:
  app-storage:
  app-bootstrap:

A) Anonymous volumes

WARNING

It's important to note that this is a copy operation - so disk usage will (temporarily) double while you migrate

We provide a "migration container" for your convenience that can access both the new and old volumes, allowing you to copy the data into the setup.

You can use mv (move files) instead of rsync in the guide below - but this is a be a destructive action, so you can't quickly roll back to your old setup, as the data no longer exists in the anonymous Docker volumes.

A1) Run migration container

You can access the Docker container with both old and new volumes by running the following command:

bash

docker compose -f docker-compose.migrate.yml run migrate bash

This will put you in the /migrate directory within the container, containing 9 directories like shown here:

plain

|-- app-storage
|   |-- new
|   `-- old
|-- db-data
|   |-- new
|   `-- old
`-- redis-data
    |-- new
    `-- old

A2) Check old folders

First thing we want to do is to check if the data inside the container looks correct!

The following commands should all return SOME files and data - if they do not - then there might be an issue with the anonymous volume binding.

INFO

The content of the old folders may not be exactly whats show in the examples below - if a couple of the files or folders match, it's extremely likely it's correct!

bash

$ ls app-storage/old
app  debugbar  docker  framework  logs
oauth-private.key  oauth-public.key  purify

# Redis data might also be entirely empty, thats *okay*
$ ls redis-data/old
appendonlydir server.pid

$ ls db-data/old
aria_log_control  ddl_recovery-backup.log  ib_buffer_pool  ib_logfile0
ibdata1  mariadb_upgrade_info  multi-master.info  mysql
performance_schema  pixelfed_prod  sys  undo001  undo002  undo003

A3) Check new folders

The following commands should all return NO files and data - if they contain data - you need to delete it (backup first!) or skip that migration step.

If you haven't run docker compose up since you updated your project in step (2) - they should be empty and good to go.

bash

ls app-storage/new
ls db-data/new
ls redis-data/new

A4) Copy data

WARNING

This is where we potentially will double your disk usage (temporarily)

Now we will copy the data from the old volumes to the new ones.

The migration container has rsync installed - which is perfect for that kind of work!

NOTE It's important that the "source" (first path in the rsync command) has a trailing / - otherwise, the directory layout will turn out wrong!

NOTE Depending on your server, these commands might take some time to finish; each command should provide a progress bar with a rough time estimation.

NOTE rsync should preserve ownership, permissions, and symlinks correctly for you and all the files copied.

Let's copy the data by running the following commands:

bash

rsync -avP app-storage/old/ app-storage/new
rsync -avP db-data/old/ db-data/new
rsync -avP redis-data/old/ redis-data/new

A5) Sanity checking

Let's make sure everything is copied over successfully!

Each new directory should contain something like (but not always exactly) the following - NO directory should have a single folder called old; if they do, the rsync commands above didn't work correctly - and you need to move the content of the old folder into the "root" of the new folder like shown a bit in the following sections.

The redis-data/new directory might also contain a server.pid

bash

$ ls redis-data/new
appendonlydir

The app-storage/new directory should look something like this

bash

$ ls app-storage/new
app  debugbar  docker  framework  logs  oauth-private.key  oauth-public.key  purify

The db-data/new directory should look something like this. There might be a lot of files or very few files, but there must be a mysql, performance_schema, and ${DB_DATABASE} (e.g., pixelfed_prod directory)

bash

$ ls db-data/new
aria_log_control  ddl_recovery-backup.log  ib_buffer_pool  ib_logfile0  ibdata1  mariadb_upgrade_info  multi-master.info  mysql  performance_schema  pixelfed_prod  sys  undo001  undo002  undo003

If everything looks good, type exit to leave the migration container.

B) Bind/Host volumes

If you used Bind/Host volumes, then this guide can't offer any hard step-by-step guide to move your data, but instead will let you know how things would look if you started from scratch, so you can mirror or change this behavior.

In your .env file, the following KEY/VALUE pairs controls where your data and config files will go - please review each and make sure your files are in these paths, or adjust them to fit your current layout.

You can see in the docker-compose.yml file for each service volume section how they are used.

DOCKER_ALL_HOST_DATA_ROOT_PATH - The "root" path for all service data.
Default: ./docker-compose-state/data
DOCKER_ALL_HOST_CONFIG_ROOT_PATH - The "root" path for all service configs.
Default: ./docker-compose-state/config
DOCKER_APP_HOST_STORAGE_PATH (/var/www/storage) - Path for Pixelfed storage, e.x., uploads by users, emojis, and such
Default: ${DOCKER_ALL_HOST_DATA_ROOT_PATH}/pixelfed/storage
DOCKER_APP_HOST_CACHE_PATH (/var/www/bootstrap/cache)
Path for Pixelfed bootstrap/cache data.
Default: ${DOCKER_ALL_HOST_DATA_ROOT_PATH}/pixelfed/cache
DOCKER_REDIS_HOST_DATA_PATH
Path where Redis will store it's data.
Default: ${DOCKER_ALL_HOST_DATA_ROOT_PATH}/redis
DOCKER_DB_HOST_DATA_PATH
Path where your database (MariaDB) store it's data.
Default: ${DOCKER_ALL_HOST_DATA_ROOT_PATH}/db

6) Start containers

With all an updated Pixelfed (step 2), updated .env file (step 3), and migrated data (steps 4, 5, 6, and 7), we're ready to start things back up again.

But before we start your Pixelfed server, let's put the new .env file we made in step 1 in its proper place.

bash

cp .env.new .env

The Database

The first thing we want to try is to start up the database by running the following command and checking the logs:

bash

docker compose up -d db
docker compose logs --tail 250 --follow db

If there are no errors and the server isn't crashing, great! If you have an easy way of connecting to the database via a GUI or CLI client, do that as well and verify the database and tables are all there.

Redis

The next thing we want to try is to start up the Redis server by running the following command and checking the logs:

bash

docker compose up -d redis
docker compose logs --tail 250 --follow redis

if there are no errors and the server isn't crashing, great!

Worker

The next thing we want to try is to start up the Worker server by running the following command and checking the logs:

bash

docker compose up -d worker
docker compose logs --tail 250 --follow worker

The container should output a lot of logs from the docker-entrypoint system, but eventually you should see these messages

Configuration complete; ready for start up
Horizon started successfully.

If you see one or both of those messages, the worker seems to be running.

If the worker is crash looping, inspect the logs and try to resolve the issues.

You can consider the following additional steps:

Enabling DOCKER_APP_ENTRYPOINT_DEBUG, which will show even more log output to help understand what is going on
Enabling DOCKER_APP_ENSURE_OWNERSHIP_PATHS against the path(s) that might have permission issues
Fixing permission issues directly on the host since your data should all be in the ${DOCKER_ALL_HOST_DATA_ROOT_PATH} folder (./docker-compose-state/data by default)

Web

The final service, web, will bring your site back online! What a journey it has been.

Let's get to it: run these commands to start the web service and inspect the logs.

bash

docker compose up -d web
docker compose logs --tail 250 --follow web

The output should be identical to that of the worker, so please see that section for debugging tips if the container is crash looping.

If the web service comes online without issues, start the rest of the (optional) services, such as the proxy, if enabled, by running:

bash

docker compose up -d
docker compose logs --tail 250 --follow

If you changed anything in the .env file while debugging, some containers might restart now; that's perfectly fine.

7) Verify

With all services online, it's time to go to your browser and check everything is working.

Upload and post a picture
Comment on a post
Like a post
Check Horizon (https://${APP_DOMAIN}/horizon) for any errors
Check the Docker compose logs via docker compose logs --follow

If everything looks fine, yay, you made it to the end! Let us do some cleanup

8) Cleanup

With everything working, please take a new snapshot/backup of your server before we do any cleanup. A post-migration snapshot is handy since it contains both the old and new configuration + data, making any recovery much easier in a rollback scenario later.

Now, with all the data in the new folders, you can delete the old Docker Container volumes (if you want, completely optional)

List all volumes and give them a look:

bash

docker volume ls

The volumes we want to delete end with the volume name (db-data, app-storage, redis-data, and app-bootstrap.) but have some prefixes in front of them.

Once you have found the volumes in in the list, delete each of them by running:

bash

docker volume rm $volume_name_in_column_two_of_the_output

You can also delete the docker-compose.yml.old and .env.old files since they are no longer needed:

bash

rm docker-compose.yml.old
rm .env.old

Rollback

Oh no, something went wrong? No worries; you have backups and a quick way back!

Move `docker-compose.yml` back

bash

cp docker-compose.yml docker-compose.yml.new
cp docker-compose.yml.old docker-compose.yml

Move `.env` file back

bash

cp env.old .env

Go back to the old source code version

bash

git checkout $commit_id_from_step_0

Start things back up

bash

docker compose up -d

Verify it worked

See Step 7 for recommended steps to verify everythin is working

Migrate to the Pixelfed + Docker Compose v2 ​

Breaking Changes ​

New Dockerfile Breaking ​

New Docker Compose file Breaking ​

New .env / .env.docker file Breaking ​

Using bind volumes Breaking ​

No Docker networks Breaking change ​

Changes needing attention ​

New Redis version Attention needed ​

Pinned MariaDB version Attention needed ​

Automatic "One-time setup tasks" Attention needed ​

New or improved features ​

Nginx Proxy service New ​

LetsEncrypt/ACME service New ​

Automatic run of "One-time setup tasks" New ​

Extensive documentation New ​

Automatic publishing of Docker images Improved ​

Customizable Dockerfile New ​

Customizable ENTRYPOINT New ​

Templating files New ​

Automatic PHP/Web server configuration New ​

Permission auto-fixing New ​

Faster Docker image building Improved ​

Fork friendly Docker releasing New ​

Migration steps ​

1) Backup ​

2) Migrate .env file ​

3) Stop containers ​

4) Update source code ​

5) Migrate data ​

A) Anonymous volumes ​

A1) Run migration container ​

A2) Check old folders ​

A3) Check new folders ​

A4) Copy data ​

A5) Sanity checking ​

B) Bind/Host volumes ​

6) Start containers ​

The Database ​

Redis ​

Worker ​

Web ​

7) Verify ​

8) Cleanup ​

Rollback ​

Move docker-compose.yml back ​

Move .env file back ​

Go back to the old source code version ​

Start things back up ​

Verify it worked ​

Migrate to the Pixelfed + Docker Compose v2

Breaking Changes

New `Dockerfile` Breaking

New Docker Compose file Breaking

New `.env` / `.env.docker` file Breaking

Using `bind` volumes Breaking

No Docker networks Breaking change

Changes needing attention

New Redis version Attention needed

Pinned MariaDB version Attention needed

Automatic "One-time setup tasks" Attention needed

New or improved features

Nginx Proxy service New

LetsEncrypt/ACME service New

Automatic run of "One-time setup tasks" New

Extensive documentation New

Automatic publishing of Docker images Improved

Customizable `Dockerfile` New

Customizable `ENTRYPOINT` New

Templating files New

Automatic PHP/Web server configuration New

Permission auto-fixing New

Faster Docker image building Improved

Fork friendly Docker releasing New

Migration steps

1) Backup

2) Migrate `.env` file

3) Stop containers

4) Update source code

5) Migrate data

A) Anonymous volumes

A1) Run migration container

A2) Check old folders

A3) Check new folders

A4) Copy data

A5) Sanity checking

B) Bind/Host volumes

6) Start containers

The Database

Redis

Worker

Web

7) Verify

8) Cleanup

Rollback

Move `docker-compose.yml` back

Move `.env` file back

Go back to the old source code version

Start things back up

Verify it worked