I remember the days of old; I meditate on all thy works; I muse on the work of thy hands.

Psalm 143:5 (KJV)

O sing unto the Lord a new song; for he hath done marvelous things: his right hand, and his holy arm, hath gotten him the victory.

Psalm 98:1 (KJV)

After upgrading from 10.2.10-MariaDB to 10.2.38-MariaDB, entries like the following appeared in the mariadb.log file:

2021-07-04  2:13:07 140428456904448 [Warning] InnoDB: Table mysql/innodb_table_stats has length mismatch in the column name table_name.  Please run mysql_upgrade

So I tried running sudo mysql_upgrade -u {user} -p{pwd} and got the following:

Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.column_stats                                 OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.gtid_slave_pos                               OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.index_stats                                  OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.roles_mapping                                OK
mysql.servers                                      OK
mysql.table_stats                                  OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Phase 2/7: Installing used storage engines... Skipped
Phase 3/7: Fixing views
mysqlcheck: Got error: 1102: Incorrect database name '#mysql50#.rocksdb' when selecting the database
FATAL ERROR: Upgrade failed

Turns out that this .rocksdb directory, possibly also a separate #rocksdb directory, both present in {wherever}/var/lib/mysql, interferes with mysql_upgrade.

The solution in my case was to stop the database service, move these folders out of {wherever}/var/lib/mysql, restart the service, and then run mysql_upgrade. Something like this:

sudo mkdir {wherever}/var/lib/mysql.kgs.20210703
sudo systemctl stop mariadb
sudo mv {wherever}/var/lib/mysql/.rocksdb {wherever}/var/lib/mysql.kgs.20210703/    
sudo mv {wherever}/var/lib/mysql/#rocksdb {wherever}/var/lib/mysql.kgs.20210703/    
sudo systemctl start mariadb

sudo mysql_upgrade -u {user} -p{pwd}

If you want, you might put the RocksDB directories back into {wherever}/var/lib/mysql at this point, but I didn’t. As far as I know, I’m not using RocksDB at all, so I’d rather not have anything RocksDB-related potentially interfering with mysql_upgrade, mysqldump, or anything else in the future. I notice, though, that a fresh #rocksdb folder–#rocksdb only, not .rocksdb also–materialized in {wherever}/var/lib/mysql after I restarted the database service.

When you create an Amazon Machine Image (AMI) of an EC2 instace via the AWS Console, you have no choice but to include the boot volume of your instance in the image, but you have the option of including or excluding any extra EBS volumes attached to the instance. That is, any extra EBS volumes will appear in the UI with an ‘X’ that you can click to exclude the volume.

It is my practice to exclude extra EBS volumes because I use extra volumes purely for data, and I want my AMIs to capture core instance state only, decoupled from the state of the data.

List php-fpm child processes:

ps -ylC php-fpm --sort:rss

Show average memory of php-fpm child processes:

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

MySQL query to show configured max connections:

show variables like 'max_connections';

MySQL query to show the max number of connections actually used since service startup:

show status like 'max_used_connections';

MySQL query to increase max connections (default is 151):

set global max_connections = 1000;

To increase MySQL max connections permanently, configure in /etc/my.cnf under [mysqld]:

[mysqld]
max_connections = 1000

To view the number of open files limit for a process:

ulimit -n

To increase the number of open files limit for any user/process, edit (for good measure) both /etc/security/limits.conf and /etc/security/limits.d/20-nproc.conf. Supply soft and hard limits.

*          soft    nofile    64000
*          hard    nofile    64000

Shell command to list the currently opened files:

lsof

MySQL query to show open files limit:

SHOW VARIABLES LIKE 'open_files_limit';

To increase the open files limit for MySQL, configure in /etc/my.cnf under [mysqld]:

[mysqld]
open_files_limit = 64000
max_connections = 400

Tune the Apache event MPM in httpd.conf (e.g., /etc/httpd/conf/httpd.conf) as follows. The settings shown are suitable for a server along the lines of an m5a.4xlarge.

...
</VirtualHost>

<ifmodule mpm_event_module>
    ServerLimit              32
    StartServers             4
    MinSpareThreads          75
    MaxSpareThreads          250
    ThreadLimit              64
    ThreadsPerChild          32
    MaxRequestWorkers        1024
    MaxConnectionsPerChild   100000
</ifmodule>

Tune the PHP FastCGI Process Manager (FPM) in www.conf (e.g., /etc/php-fpm.d/www.conf) as follows. The settings shown are suitable for a server along the lines of an m5a.4xlarge.

; Choose how the process manager will control the number of child processes.
; Possible Values:
;   static  - a fixed number (pm.max_children) of child processes;
;   dynamic - the number of child processes are set dynamically based on the
;             following directives. With this process management, there will be
;             always at least 1 children.
;             pm.max_children      - the maximum number of children that can
;                                    be alive at the same time.
;             pm.start_servers     - the number of children created on startup.
;             pm.min_spare_servers - the minimum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is less than this
;                                    number then some children will be created.
;             pm.max_spare_servers - the maximum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is greater than this
;                                    number then some children will be killed.
;  ondemand - no children are created at startup. Children will be forked when
;             new requests will connect. The following parameter are used:
;             pm.max_children           - the maximum number of children that
;                                         can be alive at the same time.
;             pm.process_idle_timeout   - The number of seconds after which
;                                         an idle process will be killed.
; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 1000

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 50

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 50

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 100

; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
;pm.process_idle_timeout = 10s;

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries.
; For endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
pm.max_requests = 100000

Scaling Considerations

Consider keeping the following in proportion to one another:

• Apache MaxRequestWorkers
• PHP FPM max_children
• MySQL MaxConnections

These represent, roughly, the number of requests that can be handled simultaneously.

The rationale for keeping these in proportion is that you wouldn’t want to configure a higher-level service to let in drastically more simultaneous requests than the lower-level services are configured to handle.

For example, if Apache is configured to accept 10,000 concurrent requests but PHP FPM is configured to handle only 50 concurrent requests, the PHP FPM service won’t be able to keep up.

So first settle on a value for the settings listed above (it doesn’t have to be exactly the same value for all three), then adjust the other Apache and PHP FPM settings accordingly. Many of the tuning-relevant Apache and PHP FPM settings are interrelated and must be consistent with one another.

Add the iOS Cordova platform:

cordova platform add ios

The preceding command will add platforms/ios with a slew of contents. Something like this:

+ platforms
  + ios
    + CordovaLib
    + MyApp
    + MyApp.xcodeproj
    + MyApp.xcworkspace
      ...
    - ios.json
      ...
    + platform_www
      ...
    + www

Note in particular MyApp.xcodeproj. This project file can be opened in XCode.

Build the iOS app:

cordova build ios