2024 Server Migration: Difference between revisions

From WPLUG
Jump to navigation Jump to search
No edit summary
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page is to collect information about our current configuration and options for switching to a new server going forward, since CentOS 7 goes end-of-life at 2024-06-30. Please feel free to edit to add missing information or correct errors.
This page is to collect information about our current configuration and options for switching to a new server going forward, since CentOS 7 goes end-of-life at 2024-06-30. Please feel free to edit to add missing information or correct errors.


= Migration Progress =
== Servers ==
There are 3 current servers:
* Linode - The legacy server running CentOS7. Mediawiki was migrated off due to bot hits driving up the load. This was also done to reduce surface attack area of an EOL CentOS.
* Penguin - The current production server running *only* mediawiki.
* Nestling - The current development/staging server.
=== Penguin ===
This server is now serving Mediawiki. It's a virtual server:
* Name: penguin
* OS: Debian 12
* Region: Atlanta
* Plan: Nanode 1GB
* IPv4: 139.144.29.132
* IPv6: 2600:3c02::f03c:94ff:fe10:c62f
=== Nestling ===
Currently doing dev on it. It'll become the staging server once migration is complete.
* Name: nestling
* OS: Debian 11
* Region: Chicago
* Plan: <s>Nanode 1GB</s> Linode 2 GB due to high memory use of Mailman 3.
* IPv4: 172.234.26.233
* IPv6: 2600:3c06::f03c:95ff:fed6:b745
This server was selected to run this OS version and location because it was the only image and location that featured cloud-init which was necessary for automated builds. Once Debian 12 gets that feature or we find something better, we'll migrate again which should be much simpler.

== Ansible ==
From https://www.ansible.com
<q>Ansible is an open source IT automation engine that automates provisioning, configuration management, application deployment, orchestration, and many other IT processes.</q>
I am using ansible to automate most of the setup and migration. Right now it does 99% of the work with some web interface clicking required here and there.

== Setup Steps ==
=== Completed ===
Here are the steps completed:
* Create or rebuild new linode, with cloud-init that adds admin users and ssh public keys.
* Set hostname, /etc/hosts, disable root ssh login
* Add board group and create board home directory
* apt update
* Install apache, mariadb, php
* Install and run certbot
* Install mediawiki and restore images and sql data
* Mail - mua, postfix, opendkim, SPF, DKIM, DMARC
* Backup mediawiki data and config for future rebuild and migration
* Restore mediawiki data and config
* Add ufw management and set up firewall
* fail2ban complete and added to penguin also
* mail aliases
* established staging.wplug.org subdomain
* mailman3 installation finally successful
* limit outbound mail in postfix so we don't spam people while testing
* ran out of memory on the 1GB plan so bumped it up to 2GB RAM
* added 512MB /swapfile on top of the 512MB swap partition
* mailman 2.1 to 3 migration
* mailman 3 backup and restore
* move django /admin page to a different url
* apache http and www redirect to staging.wplug.org

=== Working On ===
* Cleaning up Legacy data

=== To Do ===
* Create a new index page to point to wiki and mailman
* Migrate users and data
* Backup and restore using ansible


= Proper Planning =
This section is the preplanning done prior to the migration
== Server selection ==
== Server selection ==


Line 6: Line 72:


Other VPS providers can be considered, not sure if they are any cheaper/better.
Other VPS providers can be considered, not sure if they are any cheaper/better.

=== Cloud Plans ===
Looking at 1GB and 2GB plans.
{| class="wikitable"
! Vendor !! Plan !! $/Mo !! Memory !! CPU !! Storage !! Transfer
|-
| Linode || Nanode 1GB || $5 || 1GB || 1 || 25GB || 1TB
|-
| Linode || Linode 2GB || $12 || 2GB || 1 || 50GB || 2TB
|-
| DigitalOcean || 1GiB || $6 || 1GiB || 1 || 25GiB || 1TB
|-
| DigitalOcean || 2GiB || $12 || 2GiB || 1 || 50GiB || 2TB
|-
| Hostwinds || || $4.99 || 1GB || 1 || 30GB || 1TB
|-
| Hostwinds || || $9.90 || 2GB || 1 || 50GB || 2TB
|-
| Vultr || || $5 || 1GB || 1 || 25GB || 1TB
|-
| Vultr || || $10 || 2GB || 1 || 55GB || 2TB
|}
All services seem to be similar, with price differences only by a dollar or two.


== OS selection ==
== OS selection ==
Line 27: Line 116:
* Web server (Apache) - it would be possible to use Nginx instead, but I (Vance) am not familiar with setting it up
* Web server (Apache) - it would be possible to use Nginx instead, but I (Vance) am not familiar with setting it up
* PHP (Apache mod_php) for MediaWiki - could be replaced with PHP-FPM
* PHP (Apache mod_php) for MediaWiki - could be replaced with PHP-FPM
* Let's Encrypt for TLS certificate
* Let's Encrypt for TLS certificate (certbot)
* Python for Mailman, Let's Encrypt, and Trac
* Python for Mailman, Let's Encrypt, and Trac
* Perl for Monkeybot
* Perl for Monkeybot
Line 65: Line 154:
|-
|-
|}
|}

Currently we are using 22 GB of disk, as follows:
594M /boot
35M /etc
11G /home - monkeybot.log is 7.3GB
25M /opt
2.2G /root
4.0K /srv
36K /tmp
2.1G /usr
6.3G /var


=== Software availability ===
=== Software availability ===
Line 157: Line 257:
|~
|~
|B 8.0
|B 8.0
|-
|sqlite
|B 3.34
|B 3.40
|B 3.37
|-
|-
|fail2ban
|fail2ban
Line 176: Line 281:


* We have an archive of static web pages from the pre-2007 server "penguin" - it would be nice to make this history available somehow.
* We have an archive of static web pages from the pre-2007 server "penguin" - it would be nice to make this history available somehow.






Latest revision as of 04:15, 30 October 2024

This page is to collect information about our current configuration and options for switching to a new server going forward, since CentOS 7 goes end-of-life at 2024-06-30. Please feel free to edit to add missing information or correct errors.

Migration Progress

Servers

There are 3 current servers:

  • Linode - The legacy server running CentOS7. Mediawiki was migrated off due to bot hits driving up the load. This was also done to reduce surface attack area of an EOL CentOS.
  • Penguin - The current production server running *only* mediawiki.
  • Nestling - The current development/staging server.

Penguin

This server is now serving Mediawiki. It's a virtual server:

  • Name: penguin
  • OS: Debian 12
  • Region: Atlanta
  • Plan: Nanode 1GB
  • IPv4: 139.144.29.132
  • IPv6: 2600:3c02::f03c:94ff:fe10:c62f

Nestling

Currently doing dev on it. It'll become the staging server once migration is complete.

  • Name: nestling
  • OS: Debian 11
  • Region: Chicago
  • Plan: Nanode 1GB Linode 2 GB due to high memory use of Mailman 3.
  • IPv4: 172.234.26.233
  • IPv6: 2600:3c06::f03c:95ff:fed6:b745

This server was selected to run this OS version and location because it was the only image and location that featured cloud-init which was necessary for automated builds. Once Debian 12 gets that feature or we find something better, we'll migrate again which should be much simpler.

Ansible

From https://www.ansible.com Ansible is an open source IT automation engine that automates provisioning, configuration management, application deployment, orchestration, and many other IT processes. I am using ansible to automate most of the setup and migration. Right now it does 99% of the work with some web interface clicking required here and there.

Setup Steps

Completed

Here are the steps completed:

  • Create or rebuild new linode, with cloud-init that adds admin users and ssh public keys.
  • Set hostname, /etc/hosts, disable root ssh login
  • Add board group and create board home directory
  • apt update
  • Install apache, mariadb, php
  • Install and run certbot
  • Install mediawiki and restore images and sql data
  • Mail - mua, postfix, opendkim, SPF, DKIM, DMARC
  • Backup mediawiki data and config for future rebuild and migration
  • Restore mediawiki data and config
  • Add ufw management and set up firewall
  • fail2ban complete and added to penguin also
  • mail aliases
  • established staging.wplug.org subdomain
  • mailman3 installation finally successful
  • limit outbound mail in postfix so we don't spam people while testing
  • ran out of memory on the 1GB plan so bumped it up to 2GB RAM
  • added 512MB /swapfile on top of the 512MB swap partition
  • mailman 2.1 to 3 migration
  • mailman 3 backup and restore
  • move django /admin page to a different url
  • apache http and www redirect to staging.wplug.org

Working On

  • Cleaning up Legacy data

To Do

  • Create a new index page to point to wiki and mailman
  • Migrate users and data
  • Backup and restore using ansible


Proper Planning

This section is the preplanning done prior to the migration

Server selection

Linode has been acquired by Akamai and raised their prices. Could potentially downgrade from the $12/mo (2GB RAM) plan to the $5 (1GB RAM "Nanode") plan depending on distro system requirements. Could also consider dropping paid backup service or using something different.

Other VPS providers can be considered, not sure if they are any cheaper/better.

Cloud Plans

Looking at 1GB and 2GB plans.

Vendor Plan $/Mo Memory CPU Storage Transfer
Linode Nanode 1GB $5 1GB 1 25GB 1TB
Linode Linode 2GB $12 2GB 1 50GB 2TB
DigitalOcean 1GiB $6 1GiB 1 25GiB 1TB
DigitalOcean 2GiB $12 2GiB 1 50GiB 2TB
Hostwinds $4.99 1GB 1 30GB 1TB
Hostwinds $9.90 2GB 1 50GB 2TB
Vultr $5 1GB 1 25GB 1TB
Vultr $10 2GB 1 55GB 2TB

All services seem to be similar, with price differences only by a dollar or two.

OS selection

CentOS Linux is dead. CentOS Stream is only viable as long as Red Hat/IBM has a business case for it. Can consider other options like pure Debian or Ubuntu LTS.

Service enumeration

Services that are currently running on the WPLUG Linode server. These should include things that are user-facing only, not infrastructure.

  • E-mail (Postfix)
  • Mailing lists (Mailman)
  • Wiki (MediaWiki)
  • Monkeybot IRC bot (infobot)
    • While one of the oldest WPLUG services, since freenode is essentially dead, there may not be good reason to keep it alive.
  • Ticketing system (Trac)

Infrastructure software which supports the services above.

  • Web server (Apache) - it would be possible to use Nginx instead, but I (Vance) am not familiar with setting it up
  • PHP (Apache mod_php) for MediaWiki - could be replaced with PHP-FPM
  • Let's Encrypt for TLS certificate (certbot)
  • Python for Mailman, Let's Encrypt, and Trac
  • Perl for Monkeybot
  • MariaDB for MediaWiki - may be possible to use SQLite or PostgreSQL instead
  • Greylisting daemon (Postgrey)
  • Fail2ban

Support lifetime

System requirements

CPU RAM Disk
CentOS / Rocky / Alma 9 amd64, arm64, ppc64el 1.5 GB 10 GB
Debian 12 amd64, arm64, i386 (686 or higher), ppc64el, mipsel, mips64el 512 MB 4 GB
Ubuntu 22.04 1 GHz amd64, arm64, ppc64el 1 GB 2.5 GB

Currently we are using 22 GB of disk, as follows: 

594M    /boot
35M     /etc
11G     /home    - monkeybot.log is 7.3GB
25M     /opt
2.2G    /root
4.0K    /srv
36K     /tmp
2.1G    /usr
6.3G    /var

Software availability

This table is to track, for the different distros under consideration, whether the software we need is available within its repositories. We want to minimize the number of applications which have to be maintained manually.

Key:

  • B: in distro's base repository
  • A: in an additional repository provided by the distro
  • T: in a third-party repository
  • ~: not available in any known repository
  • ?: availability unknown
CentOS / Rocky / Alma 9 Debian 12 Ubuntu 22.04
postfix B 3.5.9 B 3.7.5 B 3.6.4
mailman ~ B 3.3.8 A (3) 20200530
postgrey T 1.37EPEL B 1.37 A 1.36
mediawiki ~ B 1.39 A 1.35
infobot ~ ~ ~
trac ~ A 1.6 (backports) A 1.5.3
apache B 2.4.53 B 2.4.57 B 2.4.52
nginx B 1.20.1 / 1.22.1 B 1.22.1 B 1.18.0
php B 8.0 / 8.1 B 8.2 B 8.1
php-fpm B 8.0 / 8.1 B 8.2 A 8.1
python2 ~ ~ A 2.7.18
python3 B 3.9.16 B 3.11.2 B 3.10.4
perl5 B 5.32 B 5.36 B 5.34
mariadb B 10.5 B 10.11 A 10.6
mysql B 8.0 ~ B 8.0
sqlite B 3.34 B 3.40 B 3.37
fail2ban T 1.0.2EPEL B 1.0.2 A 0.11.2
certbot T 2.6.0EPEL B 2.1.0 A 1.21.0

Third-party repositories:

Nice-to-haves

  • We have an archive of static web pages from the pre-2007 server "penguin" - it would be nice to make this history available somehow.
Retrieved from "https://wplug.org/index.php?title=2024_Server_Migration&oldid=5061"