2024 Server Migration

This page is to collect information about our current configuration and options for switching to a new server going forward, since CentOS 7 goes end-of-life at 2024-06-30. Please feel free to edit to add missing information or correct errors.

Migration Progress

Servers

There are 3 current servers:

• Linode - The legacy server running CentOS7. Mediawiki was migrated off due to bot hits driving up the load. This was also done to reduce surface attack area of an EOL CentOS.
• Penguin - The current production server running *only* mediawiki.
• Nestling - The current development/staging server.

Penguin

This server is now serving Mediawiki. It's a virtual server:

• Name: penguin
• OS: Debian 12
• Region: Atlanta
• Plan: Nanode 1GB
• IPv4: 139.144.29.132
• IPv6: 2600:3c02::f03c:94ff:fe10:c62f

Nestling

Currently doing dev on it. It'll become the staging server once migration is complete.

• Name: nestling
• OS: Debian 11
• Region: Chicago
• Plan: Nanode 1GB Linode 2 GB due to high memory use of Mailman 3.
• IPv4: 172.234.26.233
• IPv6: 2600:3c06::f03c:95ff:fed6:b745

This server was selected to run this OS version and location because it was the only image and location that featured cloud-init which was necessary for automated builds. Once Debian 12 gets that feature or we find something better, we'll migrate again which should be much simpler.

Ansible

From https://www.ansible.com Ansible is an open source IT automation engine that automates provisioning, configuration management, application deployment, orchestration, and many other IT processes. I am using ansible to automate most of the setup and migration. Right now it does 99% of the work with some web interface clicking required here and there.

Setup Steps

Completed

Here are the steps completed:

• Create or rebuild new linode, with cloud-init that adds admin users and ssh public keys.
• Set hostname, /etc/hosts, disable root ssh login
• Add board group and create board home directory
• apt update
• Install apache, mariadb, php
• Install and run certbot
• Install mediawiki and restore images and sql data
• Mail - mua, postfix, opendkim, SPF, DKIM, DMARC
• Backup mediawiki data and config for future rebuild and migration
• Restore mediawiki data and config
• Add ufw management and set up firewall
• fail2ban complete and added to penguin also
• mail aliases
• established staging.wplug.org subdomain
• mailman3 installation finally successful
• limit outbound mail in postfix so we don't spam people while testing
• ran out of memory on the 1GB plan so bumped it up to 2GB RAM
• added 512MB /swapfile on top of the 512MB swap partition
• mailman 2.1 to 3 migration
• mailman 3 backup and restore
• move django /admin page to a different url
• apache http and www redirect to staging.wplug.org

Working On

• Cleaning up Legacy data

To Do

• Create a new index page to point to wiki and mailman
• Migrate users and data
• Backup and restore using ansible

Proper Planning

This section is the preplanning done prior to the migration

Server selection

Linode has been acquired by Akamai and raised their prices. Could potentially downgrade from the $12/mo (2GB RAM) plan to the $5 (1GB RAM "Nanode") plan depending on distro system requirements. Could also consider dropping paid backup service or using something different.

Other VPS providers can be considered, not sure if they are any cheaper/better.

Cloud Plans

Looking at 1GB and 2GB plans.

Vendor	Plan	$/Mo	Memory	CPU	Storage	Transfer
Linode	Nanode 1GB	$5	1GB	1	25GB	1TB
Linode	Linode 2GB	$12	2GB	1	50GB	2TB
DigitalOcean	1GiB	$6	1GiB	1	25GiB	1TB
DigitalOcean	2GiB	$12	2GiB	1	50GiB	2TB
Hostwinds		$4.99	1GB	1	30GB	1TB
Hostwinds		$9.90	2GB	1	50GB	2TB
Vultr		$5	1GB	1	25GB	1TB
Vultr		$10	2GB	1	55GB	2TB

All services seem to be similar, with price differences only by a dollar or two.

OS selection

CentOS Linux is dead. CentOS Stream is only viable as long as Red Hat/IBM has a business case for it. Can consider other options like pure Debian or Ubuntu LTS.

Service enumeration

Services that are currently running on the WPLUG Linode server. These should include things that are user-facing only, not infrastructure.

• E-mail (Postfix)
• Mailing lists (Mailman)
• Wiki (MediaWiki)
• Monkeybot IRC bot (infobot)
  • While one of the oldest WPLUG services, since freenode is essentially dead, there may not be good reason to keep it alive.
• Ticketing system (Trac)
  • This is virtually unused, likely not worth deploying.

Infrastructure software which supports the services above.

• Web server (Apache) - it would be possible to use Nginx instead, but I (Vance) am not familiar with setting it up
• PHP (Apache mod_php) for MediaWiki - could be replaced with PHP-FPM
• Let's Encrypt for TLS certificate (certbot)
• Python for Mailman, Let's Encrypt, and Trac
• Perl for Monkeybot
• MariaDB for MediaWiki - may be possible to use SQLite or PostgreSQL instead
• Greylisting daemon (Postgrey)
• Fail2ban

Support lifetime

• CentOS Stream 9 - 2027-05-31 (limited to RHEL Full Support lifetime)
• Rocky Linux 9 - 2032-05-31
• AlmaLinux 9 - 2032-05-31
• Debian 12 "bookworm" - 2028-06-10 or later
• Ubuntu 22.04 LTS - 2027-04

System requirements

	CPU	RAM	Disk
CentOS / Rocky / Alma 9	amd64, arm64, ppc64el	1.5 GB	10 GB
Debian 12	amd64, arm64, i386 (686 or higher), ppc64el, mipsel, mips64el	512 MB	4 GB
Ubuntu 22.04	1 GHz amd64, arm64, ppc64el	1 GB	2.5 GB

Currently we are using 22 GB of disk, as follows:

594M    /boot
35M     /etc
11G     /home    - monkeybot.log is 7.3GB
25M     /opt
2.2G    /root
4.0K    /srv
36K     /tmp
2.1G    /usr
6.3G    /var

Software availability

This table is to track, for the different distros under consideration, whether the software we need is available within its repositories. We want to minimize the number of applications which have to be maintained manually.

Key:

• B: in distro's base repository
• A: in an additional repository provided by the distro
• T: in a third-party repository
• ~: not available in any known repository
• ?: availability unknown

	CentOS / Rocky / Alma 9	Debian 12	Ubuntu 22.04
postfix	B 3.5.9	B 3.7.5	B 3.6.4
mailman	~	B 3.3.8	A (3) 20200530
postgrey	T 1.37EPEL	B 1.37	A 1.36
mediawiki	~	B 1.39	A 1.35
infobot	~	~	~
trac	~	A 1.6 (backports)	A 1.5.3
apache	B 2.4.53	B 2.4.57	B 2.4.52
nginx	B 1.20.1 / 1.22.1	B 1.22.1	B 1.18.0
php	B 8.0 / 8.1	B 8.2	B 8.1
php-fpm	B 8.0 / 8.1	B 8.2	A 8.1
python2	~	~	A 2.7.18
python3	B 3.9.16	B 3.11.2	B 3.10.4
perl5	B 5.32	B 5.36	B 5.34
mariadb	B 10.5	B 10.11	A 10.6
mysql	B 8.0	~	B 8.0
sqlite	B 3.34	B 3.40	B 3.37
fail2ban	T 1.0.2EPEL	B 1.0.2	A 0.11.2
certbot	T 2.6.0EPEL	B 2.1.0	A 1.21.0

Third-party repositories:

• EPEL - Extra Packages for Enterprise Linux

Nice-to-haves

• We have an archive of static web pages from the pre-2007 server "penguin" - it would be nice to make this history available somehow.