Offline docs (switch to live docs)                         UI-only         CLI-only

What is new with MAAS

Here you will find release notes for:

MAAS 3.3.3 has been released

We are happy to announce that MAAS 3.3.3 has been released. This is a maintenance release, with no new features, providing the following bug fixes:

MAAS 3.3.2 has been released

We are happy to announce that MAAS 3.3.2 has been release with the following bug fixes:

MAAS 3.3.1 has been released

We are happy to announce that MAAS 3.3.1 has been released with the following bug fixes:

MAAS 3.3 has been released

We are happy to announce that MAAS 3.3 has been released, with one additional bug fix. MAAS 3.3 is a concerted effort to improve MAAS on multiple fronts, including a large number of bug fixes.

Cumulative summary of MAAS 3.3 features

New features created for MAAS 3.3 include:

Improved capabilities include the following:

Greatly expanded documentation sections include:

Several forward-looking improvements are included as well:

These will be documented later in blog posts.

This release also includes well over one-hundred bug fixes. Read on to catch up with what we’ve done so far this cycle.

How to install MAAS 3.3

MAAS will run on just about any modern hardware configuration, even a development laptop. If you’re not sure whether your target server will handle MAAS, you can always double-check.

NOTE: NOTE that PostgreSQL 12 is deprecated with the release of MAAS 3.3, in favour of PostgreSQL 14. Support for PostgreSQL 12 will be discontinued in MAAS 3.4. Also note, though, that Postgres 14 does not run on Focal 20.04 LTS.

How to do a fresh snap install of MAAS 3.3

To install MAAS 3.3 from a snap, simply enter the following:

$ sudo snap install --channel=3.3 maas

After entering your password, the snap will download and install from the 3.3 channel.

How to upgrade from an earlier snap version to MAAS 3.3

Maybe instead of a fresh install, you want to upgrade from a earlier snap version to the 3.3 snap, and you are using a region+rack configuration, use this command:

$ sudo snap refresh --channel=3.3 maas

After entering your password, the snap will refresh from the 3.3 candidate channel. You will not need to re-initialise MAAS.

If you are using a multi-node maas deployment with separate regions and racks, you should first run the upgrade command above for rack nodes, then for region nodes.

How to initialise MAAS 3.3 snap for a test or POC environment

You can initialise MAAS as a compact version for testing. To achieve this, we provide a separate snap, called maas-test-db, which contains a PostgreSQL database for use in testing and evaluating MAAS. The following instructions will help you take advantage of this test configuration.

Once MAAS is installed, you can use the --help flag with maas init to get relevant instructions:

$ sudo maas init --help
usage: maas init [-h] {region+rack,region,rack} . . .

Initialise MAAS in the specified run mode.

optional arguments:
  -h, --help            show this help message and exit

run modes:
  {region+rack,region,rack}
    region+rack         Both region and rack controllers
    region              Region controller only
    rack                Rack controller only

When installing region or rack+region modes, MAAS needs a
PostgreSQL database to connect to.

If you want to set up PostgreSQL for a non-production deployment on
this machine, and configure it for use with MAAS, you can install
the maas-test-db snap before running 'maas init':
    sudo snap install maas-test-db
    sudo maas init region+rack --database-uri maas-test-db:///

We’ll quickly walk through these instructions to confirm your understanding. First, install the maas-test-db snap:

sudo snap install maas-test-db

Note that this step installs a a running PostgreSQL and a MAAS-ready database instantiation. When it’s done, you can double check with a built-in PostgreSQL shell:

$ sudo maas-test-db.psql
psql (12.4)
Type "help" for help.

postgres=# \l

This will produce a list of databases, one of which will be maasdb, owned by maas. Note that this database is still empty because MAAS is not yet initialised and, hence, is not yet using the database. Once this is done, you can run the maas init command:

sudo maas init region+rack --database-uri maas-test-db:///

After running for a moment, the command will prompt you for a MAAS URL; typically, you can use the default:

MAAS URL [default=http://10.45.222.159:5240/MAAS]:

When you’ve entered a suitable URL, or accepted the default, the following prompt will appear:

MAAS has been set up.

If you want to configure external authentication or use
MAAS with Canonical RBAC, please run

  sudo maas configauth

To create admins when not using external authentication, run

  sudo maas createadmin

Let’s assume you just want a local testing user named admin:

$ sudo maas createadmin
Username: admin
Password: ******
Again: ******
Email: admin@example.com
Import SSH keys [] (lp:user-id or gh:user-id): gh:yourusername

At this point, MAAS is basically set up and running. You can confirm this with sudo maas status. If you need an API key, you can obtain this with sudo maas apikey --username yourusername. Now you will be able to test and evaluate MAAS by going to the URL you entered or accepted above and entering your admin username and password.

Initialise MAAS for a production configuration

To install MAAS in a production configuration, you need to setup PostgreSQL, as described below.

Setting up PostgreSQL from scratch

To set up PostgreSQL, even if it’s running on a different machine, you can use the following procedure:

  1. You will need to install PostgreSQL on the machine where you want to keep the database. This can be the same machine as the MAAS region/rack controllers or a totally separate machine. If PostgreSQL (version 14) is already running on your target machine, you can skip this step. To install PostgreSQL, run these commands:

    sudo apt update
sudo apt install -y postgresql

  2. You want to make sure you have a suitable PostgreSQL user, which can be accomplished with the following command, where $MAAS_DBUSER is your desired database username, and $MAAS_DBPASS is the intended password for that username. Note that if you’re executing this step in a LXD container (as root, which is the default), you may get a minor error, but the operation will still complete correctly.

    sudo -u postgres psql -c "CREATE USER \"$MAAS_DBUSER\" WITH ENCRYPTED PASSWORD '$MAAS_DBPASS'"

  3. Create the MAAS database with the following command, where $MAAS_DBNAME is your desired name for the MAAS database (typically known as maas). Again, if you’re executing this step in a LXD container as root, you can ignore the minor error that results.

    sudo -u postgres createdb -O "$MAAS_DBUSER" "$MAAS_DBNAME"

  4. Edit /etc/postgresql/14/main/pg_hba.conf and add a line for the newly created database, replacing the variables with actual names. You can limit access to a specific network by using a different CIDR than 0/0.

    host    $MAAS_DBNAME    $MAAS_DBUSER    0/0     md5

  5. You can then initialise MAAS via the following command:

    sudo maas init region+rack --database-uri "postgres://$MAAS_DBUSER:$MAAS_DBPASS@$HOSTNAME/$MAAS_DBNAME"

NOTE: You should use localhost for $HOSTNAME if you’re running PostgreSQL on the same box as MAAS.

Don’t worry; if you leave out any of the database parameters, you’ll be prompted for those details.

How to do a fresh install of MAAS 3.3 from packages

MAAS 3.3 from packages runs on 22.04 LTS only. The recommended way to set up an initial MAAS environment is to put everything on one machine:

sudo apt-add-repository ppa:maas/3.3
sudo apt update
sudo apt-get -y install maas

Executing this command leads you to a list of dependent packages to be installed, and a summary prompt that lets you choose whether to continue with the install. Choosing “Y” proceeds with a standard apt package install.

Distributed environment

For a more distributed environment, you can place the region controller on one machine:

sudo apt install maas-region-controller

and the rack controller on another:

sudo apt install maas-rack-controller
sudo maas-rack register

These two steps will lead you through two similar apt install sequences.

How to upgrade from 3.2 or lower to MAAS 3.3

If you are running MAAS 3.2 or lower, you can upgrade directly to MAAS 3.3. You must first make sure that the target system is running Ubuntu 22.04 LTS by executing the following command:

lsb_release -a

The response should look something like this:

Distributor ID: Ubuntu
Description:    Ubuntu xx.yy
Release:    xx.yy
Codename:   $RELEASE_NAME

The required “xx.yy” required for MAAS 3.3 is “22.04,” code-named “jammy”.

If you are currently running Ubuntu focal 20.04 LTS, you can upgrade to jammy 22.04 LTS with the following procedure:

Upgrade the release:

sudo do-release-upgrade --allow-third-party

Accept the defaults for any questions asked by the upgrade script.

Reboot the machine when requested.

Check whether the upgrade was successful:

lsb_release -a

A successful upgrade should respond with output similar to the following:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04(.nn) LTS
Release:    20.04
Codename:   focal

If you’re upgrading from MAAS version 2.8 or lower to version 3.3: While the following procedures should work, note that they are untested. Use at your own risk. Start by making a verifiable backup; see step 1, below.

Back up your MAAS server completely; the tools and media are left entirely to your discretion. Just be sure that you can definitely restore your previous configuration, should this procedure fail to work correctly.

Add the MAAS 3.3 PPA to your repository list with the following command, ignoring any apparent error messages:

sudo apt-add-repository ppa:maas/3.3

Run the release upgrade like this, answering any questions with the given default values:

sudo do-release-upgrade --allow-third-party

Check whether your upgrade has been successful by entering:

lsb_release -a

If the ugprade was successful, this command should yield output similar to the following:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04(.nn) LTS
Release:    20.04
Codename:   focal

Check your running MAAS install (by looking at the information on the bottom of the machine list) to make sure you’re running the 3.3 release.

If this didn’t work, you will need to restore from the backup you made in step 1, and consider obtaining separate hardware to install MAAS 3.3.

Ansible playbooks for HA MAAS, PostgreSQL, and other MAAS configurations

Ansible↗ playbooks↗ are now available for MAAS. These extended YAML files automate various routine aspects of MAAS setup and configuration.

Ten words or less

Automate the drudgery of installing and configuring MAAS with Ansible.

Background explanations on Ansible

Ansible is more than scripting; it’s a fully-featured automation tool that allows you to group modules (binaries or snippets of code) into specific tasks. These tasks can be combined to build plays, and plays can be combined to build playbooks. The idea is to abstract each level so you’re not trying to keep track of which lines of Bash code you need to cut and past in between others.

With MAAS 3.3, playbooks are available to:

MAAS Playbooks are available from a repository↗. They will eventually be available through Ansible Galaxy.

There is also a set of groups that will automate setting up specific sections of MAAS. For example, there is a PostgreSQL group that sets up the primary and secondary PostgreSQL roles, bypassing the need to run both playbooks individually.

After installing ansible, running each of the playbooks on a blank machine will have a fresh install of MAAS ready to go. For example, running the region+rack will setup a region+rack on the host machine.

How you can use Ansible playbooks

In general terms, you can run any of the MAAS Ansible plays with a command of this form:

ansible-playbook -i hosts \
--extra_vars \
"maas_version=$MAAS_VERSION 
maas_postgres_password=$MAAS_PG_PASSWORD 
maas_postgres_replication_password=$MAAS_PG_REP_PASSWORD 
maas_installation_type=<deb|snap> 
maas_url=$MAAS_URL" \
./site.yaml

A command of this form will run all of the plays below (i.e., the entire playbook). If you want to run the tasks for one particular role (or roles), you can use the form –tags to limit which parts of the MAAS Ansible playbook run. Consult the Ansible documentation for more details on additional options and command structure.

NOTE: There’s some extra good news here: These Ansible playbooks were built and tested with MAAS 3.2, so you can use them with all variants of MAAS 3.2.

For example, suppose you want to install a MAAS region controller. As an operator, you want want to install a MAAS region controller onto a given host using Ansible. To accomplish this, you need only:

To set the maas_region_controller role, you add that role to your Inventory file in the form of either an INI or a YAML file, where each role is followed by the addresses of each host to attach the role to. The example below attaches the region controller role to a host running on 10.10.0.20 with the user ubuntu:

INI:

[maas_region_controller]
10.10.0.20 ansible_user=ubuntu
YAML:

all:
  maas_region_controller:
    hosts:
      10.10.0.20:
        ansible_user: ubuntu

When running the playbook for a host with the maas_region_controller role, the playbook installs the MAAS region controller. The documented ansible variable maas_installation_type provides the user with the ability to set whether it’s a deb installation or a snap installation, along with additional variables for MAAS version, snap channel and/or PPA.

The default installation is a snap. A successful run of the playbook should give the operator an accessible and ready MAAS instance. The playbook uses an Ansible variable to determine what version of MAAS to deploy. The playbook won’t execute (i.e “skipped” in the context of Ansible) if the Ubuntu version is incompatible with the version and install method. The Region Controller tasks should be able to execute on multiple hosts in a single execution if the target is an Ansible Group rather than a single host. The newly-installed region controller should be accessible at the specified host ip address, as though the controller had been installed manually.

Read the Ansible playbooks reference document to learn more about the feature and the additional playbooks that are available.

Improved machine list filtering

MAAS 3.3 dramatically reduces the latency associated with refreshing large machine lists.

Ten words or less

You can filter machines mere seconds after one page loads.

How list filtering is improved

NOTE: NOTE that this feature is still in development, so some of the feature-set described in this section may not be fully operational yet. As always, we reserve the right to change this feature-set until the final release of MAAS 3.3. These release notes will be updated as the feature develops.

MAAS 3.3 enhances the way you can filter the machine list, in two ways:

  1. You may begin filtering within a very short time after the first page of the machine list loads, even if you have more than 10,000 machines in the list.

  2. You have a wider range of filter choices, as described in the table below.

Note that with this version of MAAS, matching machine counts have been removed from the filter list for better performance.

More filter parameters have been added

The following table describes the expanded filter set for the MAAS machine list:

See How to search MAAS for more details on how to use these parameters.

Parameter (bold) w/example Shows nodes… Dyn Grp Man
arch:(=architecture) with “architecture” Grp
arch:(!=architecture) NOT with “architecture” Dyn
zone:(=zone-name) in “zone-name” Dyn Grp
zone:(!=zone-name) NOT in “zone-name” Dyn
pool:(=resource-pool) in “resource-pool” Dyn Grp
pool:(!=resource-pool) NOT in “resource-pool” Dyn
pod:(=pod-name) with “pod-name” Dyn Grp
pod:(!=pod-name) NOT with “pod-name” Dyn
pod_type:(=pod-type) with power type “pod-type” Dyn Grp Man
pod_type:(!=pod-type) NOT with power type “pod-type” Dyn Man
domain:(=domain-name) with “domain-name” Dyn Grp Man
domain:(!=domain-name) NOT with “domain-name” Dyn Man
status:(=op-status) having “op-status” Grp
status:(!=op-status) NOT having “op-status” Dyn
owner:(=user) owned by “user” Dyn Grp
owner:(!=user) NOT owned by “user” Dyn
power_state:(=power-state) having “power-state” Grp Man
power_state:(!=power-state) NOT having “power-state” Dyn Man
tags:(=tag-name) with tag “tag-name” Dyn
tags:(!=tag-name) NOT with tag “tag-name” Dyn
fabrics:(=fabric-name) in “fabric-name” Dyn
fabrics:(!=fabric-name) NOT in “fabric-name” Dyn
fabric_classes:(=fabric-class) in “fabric-class” Dyn Man
fabric_classes:(!=fabric-class) NOT in “fabric-class” Dyn Man
fabric_name:(=fabric-name) in “boot-interface-fabric” Dyn Man
fabric_name:(!=fabric-name) NOT in “boot-interface-fabric” Dyn Man
subnets:(=subnet-name) attached to “subnet-name” Dyn
subnets:(!=subnet-name) Not attached to “subnet-name” Dyn
link_speed:(link-speed) having “link-speed” Dyn Man
link_speed:(!link-speed) NOT having “link-speed” Dyn Man
vlans:(=vlan-name) attached to “vlan-name” Dyn
vlans:(!=vlan-name) NOT attached to “vlan-name” Dyn
storage:(storage-MB) having “storage-MB” Dyn Man
total_storage:(total-stg-MB) having “total-stg-MB” Dyn Man
total_storage:(!total-stg-MB) NOT having “total-stg-MB” Dyn Man
cpu_count:(cpu-count) having “cpu-count” Dyn Man
cpu_count:(!cpu-count) NOT having “cpu-count” Dyn Man
mem:(ram-in-MB) having “ram-in-MB” Dyn Man
mem:(!ram-in-MB) NOT having “ram-in-MB” Dyn Man
mac_address:(=MAC) having MAC address “MAC” Dyn Man
mac_address:(!=MAC) NOT having Dyn Man
agent_name:(=agent-name) Include nodes with agent-name Dyn Man
agent_name:(!=agent-name) Exclude nodes with agent-name Dyn Man
cpu_speed:(cpu-speed-GHz) CPU speed Dyn Man
cpu_speed:(!cpu-speed-GHz) CPU speed Dyn Man
osystem:(=os-name) The OS of the desired node Dyn Man
osystem:(!=os-name) OS to ignore Dyn Man
distro_series:(=distro-name) Include nodes using distro Dyn Man
distro_series:(!=distro-name) Exclude ndoes using distro Dyn Man
ip_addresses:(=ip-address) Node’s IP address Dyn Man
ip_addresses:(!=ip-address) IP address to ignore Dyn Man
spaces:(=space-name) Node’s spaces Dyn
spaces:(!=space-name) Node’s spaces Dyn
workloads:(=annotation-text) Node’s workload annotations Dyn
workloads:(!=annotation-text) Node’s workload annotations Dyn
physical_disk_count:(disk-count) Physical disk Count Dyn Man
physical_disk_count:(!disk-count) Physical disk Count Dyn Man
pxe_mac:(=PXE-MAC) Boot interface MAC address Dyn Man
pxe_mac:(!=PXE-MAC) Boot interface MAC address Dyn Man
fqdn:(=fqdn-value) Node FQDN Dyn Man
fqdn:(!=fqdn-value) Node FQDN Dyn Man
simple_status:(=status-val) Include nodes with simple-status Dyn Man
simple_status:(!=status-val) Exclude nodes with simple-status Dyn Man
devices:(=) Devices Dyn Man
interfaces:(=) Interfaces Dyn Man
parent:(=) Parent node Dyn Grp Man

Native support for 22.04 LTS and core22

MAAS can now be installed as a PPA, directly on Ubuntu 22.04, without the need to use snaps.

Ten words or less

MAAS packages now run on Ubuntu 22.04, aka Jammy Jellyfish.

Notes on 22.04 LTS MAAS packages

MAAS users want to install MAAS on a 22.04 LTS system via deb packages, as well as upgrade machines currently running MAAS on Ubuntu 20.04 LTS to 22.04 LTS. With the advent of MAAS 3.3, we have created an appropriate PPA with all required dependencies. This PPA can be directly installed on Ubuntu 22.04, Jammy Jellyfish, with no requirement to use snaps.

Note that the upgrade procedure will require a release upgrade from previous Ubuntu versions to Ubuntu 22.04. Also note that, with this version of MAAS, PostgreSQL 12 is deprecated and should be upgraded to PostgreSQL 14. The installation guide provides the necessary details.

UI performance improvements

We wanted to improve the performance of the machine list page for large (>10000 machines) MAASes, and allow users to search and filter machines as quickly as possible.

Ten words or less

We’re working on making large machine lists load in background.

Background performance work

In MAAS 3.2 and earlier, machine search and filter requires that all machines be fetched by the UI client before it becomes usable. For smaller MAASes this may not be an issue, but when considering MAASes with 1000 machines or more this can make the user wait an unacceptably long time before they can search and filter. With the release of MAAS 3.3, when a MAAS UI user wants to find a particular machine, they do not have to wait for all their machines data to load before they can start searching. The user can start searching for machines within a short time after the visible page of the machine list has fully loaded on the UI screen. See Improved machine list filtering, in these release notes, for details on the enhanced filtering capabilities that were included in this work.

Enhanced MIB support for Windows OS images

The procedure for creating custom Windows OS images has been thoroughly updated and verified.

Ten words or less

MAAS custom Windows images now support most releases and options.

What has been added to Windows custom images

Specifically, MIB now supports a much wider range of Windows images. Previously, only 2012 and 2106 Windows versions were supported with MIB. Now the list is much longer, bringing deployable MAAS versions up to date with the current Windows releases:

There are also special instructions for using both UEFI and BIOS bootloaders, as well as instructions for using LXD containers with custom-built Windows images.

Finally, MIB has been extended to accept a much wider range of options for windows builds. Some of the new Windows-specific options include:

Some news Windows-specific platform options include:

This update should make it much simpler to use custom-built Windows images with MAAS.

Shifting the MAAS API documentation to OpenAPI standards

MAAS API User want to experience the MAAS API in a more standard way, along the lines of the OpenAPI definition. MAAS 3.3 begins this process by providing most of the MAAS API functionality in a discover-able form. You should now be able to easily retrieve human-readable service documentation and API definitions using standard methods. Consult the API documentation↗ for details.

MAAS configuration settings reference

MAAS 3.3 documentation consolidates configuration settings in one article, in addition to their other mentions throughout the documentation set.

Ten words or less

“Settings” now has its own page, and some new options.

What is new about this update

MAAS configuration settings are scattered in various (generally relevant) places throughout the documentation, but there has never been one reference page that addresses all settings in one place. MAAS 3.3 remedies this by adding the Configuration settings reference.

A minor new feature added with MAAS 3.3 is MAAS site identity, which enables some new configuration parameters:

These enhancements were made available to assist users who have more than one instance (e.g., production and staging), and have issues with operations accidentally making changes to the wrong instance.

Improved MAAS event documentation

MAAS event documentation has been expanded to include much better explanations of MAAS events, including many examples.

Ten words or less

We’ve finally documented MAAS events, making them easier to decode.

Understanding MAAS events

Events are state changes that happen to MAAS elements, caused by MAAS itself, an external agent, or a users. Understanding events is an essential debugging skill. But events appear in three different places in MAAS, each presentation providing slightly different information. These screens are usually dense and hard to search.

In this major documentation update, we’ve standardised on the MAAS CLI events query command as the best way to review, filter, and summarise events. We’ve summarised the six main event types:

In addition, the new document explains how these event types tend to overlap when queried. We’ve also provide detailed instructions on how to use the most common filters:

Since the MAAS CLI returns JSON – which is hard to humans to parse – we’ve included some exemplary jq predicates of the form:

maas $PROFILE events query limit=20 \
| jq -r '(["USERNAME","NODE","HOSTNAME","LEVEL","DATE","TYPE","EVENT"] | 
(., map(length*"-"))),
(.events[] | [.username,.node,.hostname,.level,.created,.type,.description]) 
| @tsv' | column -t -s$'\t'

And finally, we provided some detailed usage examples. For instance, we walked a MAAS machine called fun-zebra through the following states:

We used this example command:

 maas $PROFILE events query level=INFO hostname=fun-zebra limit=1000 | jq -r '(["USERNAME","NODE","HOSTNAME","LEVEL","DATE","TYPE","EVENT"] | (., map(length*"-"))),(.events[] | [.username,.node,.hostname,.level,.created,.type,.description]) | @tsv' | column -t -s$'\t'

This gave us a reasonably thorough report of what happened to the machine:

USERNAME  NODE    HOSTNAME   LEVEL  DATE                        TYPE                   EVENT
--------  ----    --------   -----  ----                        ----                   -----
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:29:53  Exited rescue mode     
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:29:52  Powering off           
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:28:58  Rescue mode            
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:27:18  Loading ephemeral      
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:26:40  Performing PXE boot    
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:26:23  Power cycling          
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:26:23  Entering rescue mode   
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:26:14  Powering off           
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:26:14  Aborted testing        
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:24:08  Performing PXE boot    
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:23:51  Powering on            
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:23:51  Testing                
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:23:38  Released               
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:23:37  Powering off           
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:23:37  Releasing              
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:22:41  Deployed               
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:21:49  Rebooting              
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:18:42  Configuring OS         
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:17:42  Installing OS          
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:17:30  Configuring storage    
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:15:31  Loading ephemeral      
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:14:48  Performing PXE boot    
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:14:31  Powering on            
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 21:14:27  Deploying              
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:04:17  Ready                  
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:04:07  Running test           smartctl-validate on sda
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:01:27  Gathering information  
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:01:10  Loading ephemeral      
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:00:35  Performing PXE boot    
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:00:16  Powering on            
unknown   bk7mg8  fun-zebra  INFO   Thu, 29 Sep. 2022 20:00:16  Commissioning

Additional examples and techniques are provided as part of this new documentation.

Improved MAAS audit event documentation

MAAS audit event documentation has been greatly expanded to include much better explanations of MAAS audit events, including detailed examples of how to reconstruct machine life-cycles in the updated version of “How to work with audit event logs”.

Ten words or less

We’ve finally offered details about how you should audit MAAS.

Understanding how audit events explain MAAS internal operations

There’s probably no limit to what you can figure out if you use audit events properly. The problems are: (1) a lot goes on in MAAS, and (2) you need more than just the explicit audit events to get a clear picture of what’s happening. We’ve tried to address this by taking a deeper look at the auditing process (not just the events).

As you may know, an audit event is just a MAAS event tagged with AUDIT. It generally captures changes to the MAAS configuration and machine states. These events provide valuable oversight of user actions and automated updates – and their effects – especially when multiple users are interacting with multiple machines.

Viewing events

Audit events are examined using the MAAS CLI with the level=AUDIT parameter set:

$ maas $PROFILE events query level=AUDIT

You’ll probably get better results by appending a jq filter, to prettify the output:

$ maas $PROFILE events query level=AUDIT after=0 limit=20 \
| jq -r '(["USERNAME","HOSTNAME","DATE","EVENT"] | 
(., map(length*"-"))),
(.events[] | [.username,.hostname,.created,.description]) 
| @tsv' | column -t -s$'\t'

By itself, such a command might produce output similar to this:

USERNAME  HOSTNAME     DATE                        EVENT
--------  --------     ----                        -----
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 2 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 1 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  pci device 0 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  block device sda was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  interface enp5s0 was updated on node 8wmfx3
unknown   valued-moth  Thu, 21 Apr. 2022 19:45:14  0 bytes of memory was removed on node 8wmfx3
admin     valued-moth  Thu, 21 Apr. 2022 19:36:48  Started deploying 'valued-moth'.
admin     valued-moth  Thu, 21 Apr. 2022 19:36:21  Acquired 'valued-moth'.
admin     unknown      Thu, 21 Apr. 2022 19:21:46  Updated configuration setting 'completed_intro' to 'True'.
admin     unknown      Thu, 21 Apr. 2022 19:20:49  Updated configuration setting 'upstream_dns' to '8.8.8.8'.
admin     unknown      Thu, 21 Apr. 2022 19:20:49  Updated configuration setting 'maas_name' to 'neuromancer'.
admin     unknown      Thu, 21 Apr. 2022 19:20:47  Updated configuration setting 'http_proxy' to ''.
admin     unknown      Thu, 21 Apr. 2022 19:20:24  Logged in admin.

You can, of course, use the various event filters with level=AUDIT to further restrict your output.

The meaning of audit events

Later on in the documentation, we walk through a sample of audit events and demonstrate how to interpret and use them. This includes detailed examples of various audit event queries, walking through real-world examples to answer questions like:

  1. Who deployed comic-muskox?

  2. What happened to sweet-urchin?

  3. Why is fleet-calf in rescue mode?

  4. Where did these changes come from in setup.sh?

  5. What caused ruling-bobcat to be marked as broken?

  6. Who’s responsible for the DHCP snippet called foo?

Auditing with finesse

As part of the updates to our “How to work with audit event logs”, we’ve tried to offer you some finesse in reconstructing machine life-cycles. We’ve shown how to combine various levels of MAAS event queries with standard command line utilities to produce clear audit trails such as this one:

418606  ERROR    Marking node broken               Wed, 17 Nov. 2021 00:02:52  A Physical Interface requires a MAC address.
418607  DEBUG    Node changed status               Wed, 17 Nov. 2021 00:02:52  From 'New' to 'Broken'
418608  DEBUG    Marking node fixed                Wed, 17 Nov. 2021 00:04:24  
418609  DEBUG    Node changed status               Wed, 17 Nov. 2021 00:04:24  From 'Broken' to 'Ready'
418613  DEBUG    User acquiring node               Wed, 17 Nov. 2021 00:04:51  (admin)
418614  DEBUG    Node changed status               Wed, 17 Nov. 2021 00:04:51  From 'Ready' to 'Allocated' (to admin)
418615  DEBUG    User starting deployment          Wed, 17 Nov. 2021 00:04:51  (admin)
418616  DEBUG    Node changed status               Wed, 17 Nov. 2021 00:04:51  From 'Allocated' to 'Deploying'
418617  INFO     Deploying                         Wed, 17 Nov. 2021 00:04:51  
418618  AUDIT    Node                              Wed, 17 Nov. 2021 00:04:51  Started deploying 'ruling-bobcat'.
418619  INFO     Powering on                       Wed, 17 Nov. 2021 00:04:55  
418625  ERROR    Marking node failed               Wed, 17 Nov. 2021 00:05:32  Power on for the node failed: Failed talking to node's BMC: Failed to power pbpncx. BMC never transitioned from off to on.
418626  DEBUG    Node changed status               Wed, 17 Nov. 2021 00:05:32  From 'Deploying' to 'Failed deployment'
418627  ERROR    Failed to power on node           Wed, 17 Nov. 2021 00:05:32  Power on for the node failed: Failed talking to node's BMC: Failed to power pbpncx. BMC never transitioned from off to on.

In this case, we managed to recognise, rather quickly, that no physical interface had been defined for ruling-bobcat, hence deployment fails because MAAS can’t communicate with the node’s BMC. There are many other issues you can recognise with careful use of MAAS events to audit machine behaviours. We welcome your feedback on this new documentation endeavour.

MAAS 3.3 bug list

The following sections enumerate the bugs we’ve fixed in MAAS 3.3.

MAAS 3.3 Beta 1 bug list

So far in MAAS 3.3, we’ve fixed well over 100 bugs:

More bug-fixes are planned for later 3.3 releases.

MAAS 3.3 Beta 2 bug list

MAAS 3.3 Beta 3 bug list

MAAS 3.3 RC1 bug list

MAAS 3.3 RC3 bug list

MAAS 3.3.0 bug list

Release notes for other MAAS versions

Here are release notes for other relatively recent MAAS versions: