diff -Nru nova-26.1.0/debian/changelog nova-26.2.2/debian/changelog --- nova-26.1.0/debian/changelog 2023-05-12 08:15:11.000000000 +0000 +++ nova-26.2.2/debian/changelog 2024-07-10 21:49:31.000000000 +0000 @@ -1,3 +1,38 @@ +nova (2:26.2.2-1~deb12u3) bookworm-security; urgency=high + + * CVE-2024-40767: Regression VMDK/qcow arbitrary file access (CVE-2024-32498) + Applied upstream patches (Closes: #1076774): + - CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch + - CVE-2024-40767_2_Reproduce_iso_regression_with_deep_format_inspection_antelope.patch + - CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch + - CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch + * Add qemu-utils as build-depends to run new tests. + + -- Thomas Goirand Wed, 10 Jul 2024 23:49:31 +0200 + +nova (2:26.2.2-1~deb12u2) bookworm-security; urgency=high + + * CVE-2024-32498: Arbitrary file access through custom QCOW2 external data. + Added upstream patch (Closes: #1074762). + - libvirt_Add_encryption_support_to_qemu-img_create_command.patch + - CVE-2024-32498_nova-unified-patch-for-2023.1.patch + - CVE-2024-32498_4_late-nova-fix.patch + * Fix nova-compute-kvm depends to (= ${binary:Version}). + * New upstream point release. + + -- Thomas Goirand Wed, 06 Mar 2024 11:15:59 +0100 + +nova (2:26.2.0-1~deb12u1) bookworm; urgency=medium + + * Upgraded to latest point release with the below bufixes: + - Fixes CVE-2023-2088 (dropping patch). + - Enable use of service user token with admin context. + - Ironic: retry when node not available. + - Remove deleted projects from flavor access list. + - Revert "Debug Nova APIs call failures". + + -- Thomas Goirand Fri, 16 Jun 2023 06:39:29 +0200 + nova (2:26.1.0-4) unstable; urgency=high * Remove deprecated / removed options from nova-compute-ironic.conf. diff -Nru nova-26.1.0/debian/control nova-26.2.2/debian/control --- nova-26.1.0/debian/control 2023-05-12 08:15:11.000000000 +0000 +++ nova-26.2.2/debian/control 2024-07-10 21:49:31.000000000 +0000 @@ -113,6 +113,7 @@ python3-websockify, python3-wsgi-intercept, python3-yaml, + qemu-utils, sqlite3, subunit, Build-Conflicts: @@ -264,8 +265,8 @@ adduser, dpkg-dev, libvirt-daemon-system, - nova-common, - nova-compute, + nova-common (= ${binary:Version}), + nova-compute (= ${binary:Version}), python3-libvirt, qemu-block-extra, qemu-system-x86, diff -Nru nova-26.1.0/debian/patches/CVE-2023-2088_Use_force_True_for_os-brick_disconnect_during_delete.patch nova-26.2.2/debian/patches/CVE-2023-2088_Use_force_True_for_os-brick_disconnect_during_delete.patch --- nova-26.1.0/debian/patches/CVE-2023-2088_Use_force_True_for_os-brick_disconnect_during_delete.patch 2023-05-12 08:15:11.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2023-2088_Use_force_True_for_os-brick_disconnect_during_delete.patch 1970-01-01 00:00:00.000000000 +0000 @@ -1,1088 +0,0 @@ -Author: melanie witt -Date: Wed, 15 Feb 2023 22:37:40 +0000 -Subject: CVE-2023-2088 Use force=True for os-brick disconnect during delete - The 'force' parameter of os-brick's disconnect_volume() method allows - callers to ignore flushing errors and ensure that devices are being - removed from the host. - . - We should use force=True when we are going to delete an instance to - avoid leaving leftover devices connected to the compute host which - could then potentially be reused to map to volumes to an instance that - should not have access to those volumes. - . - We can use force=True even when disconnecting a volume that will not be - deleted on termination because os-brick will always attempt to flush - and disconnect gracefully before forcefully removing devices. -Bug: https://launchpad.net/bugs/2004555 -Bug-Debian: https://bugs.debian.org/1035981 -Origin: upstream, https://review.opendev.org/c/openstack/nova/+/882860 -Change-Id: I3629b84d3255a8fe9d8a7cea8c6131d7c40899e8 -Last-Update: 2023-05-12 - -diff --git a/doc/source/admin/configuration/cross-cell-resize.rst b/doc/source/admin/configuration/cross-cell-resize.rst -index e51e425..0c34fd1 100644 ---- a/doc/source/admin/configuration/cross-cell-resize.rst -+++ b/doc/source/admin/configuration/cross-cell-resize.rst -@@ -284,7 +284,7 @@ - Timeouts - ~~~~~~~~ - --Configure a :ref:`service user ` in case the user token -+Configure a :ref:`service user ` in case the user token - times out, e.g. during the snapshot and download of a large server image. - - If RPC calls are timing out with a ``MessagingTimeout`` error in the logs, -diff --git a/doc/source/admin/configuration/index.rst b/doc/source/admin/configuration/index.rst -index 233597b..f5b6fde 100644 ---- a/doc/source/admin/configuration/index.rst -+++ b/doc/source/admin/configuration/index.rst -@@ -19,6 +19,7 @@ - .. toctree:: - :maxdepth: 1 - -+ /admin/configuration/service-user-token - /admin/configuration/api - /admin/configuration/resize - /admin/configuration/cross-cell-resize -diff --git a/doc/source/admin/configuration/service-user-token.rst b/doc/source/admin/configuration/service-user-token.rst -new file mode 100644 -index 0000000..740730a ---- /dev/null -+++ b/doc/source/admin/configuration/service-user-token.rst -@@ -0,0 +1,59 @@ -+.. _service_user_token: -+ -+=================== -+Service User Tokens -+=================== -+ -+.. note:: -+ -+ Configuration of service user tokens is **required** for every Nova service -+ for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for -+ details. -+ -+Configure Nova to send service user tokens alongside regular user tokens when -+making REST API calls to other services. The identity service (Keystone) will -+authenticate a request using the service user token if the regular user token -+has expired. -+ -+This is important when long-running operations such as live migration or -+snapshot take long enough to exceed the expiry of the user token. Without the -+service token, if a long-running operation exceeds the expiry of the user -+token, post operations such as cleanup after a live migration could fail when -+Nova calls other service APIs like block-storage (Cinder) or networking -+(Neutron). -+ -+The service token is also used by services to validate whether the API caller -+is a service. Some service APIs are restricted to service users only. -+ -+To set up service tokens, create a ``nova`` service user and ``service`` role -+in the identity service (Keystone) and assign the ``service`` role to the -+``nova`` service user. -+ -+Then, configure the :oslo.config:group:`service_user` section of the Nova -+configuration file, for example: -+ -+.. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://104.130.216.102/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = secretservice -+ ... -+ -+And configure the other identity options as necessary for the service user, -+much like you would configure nova to work with the image service (Glance) or -+networking service (Neutron). -+ -+.. note:: -+ -+ Please note that the role assigned to the :oslo.config:group:`service_user` -+ needs to be in the configured -+ :oslo.config:option:`keystone_authtoken.service_token_roles` of other -+ services such as block-storage (Cinder), image (Glance), and networking -+ (Neutron). -diff --git a/doc/source/admin/live-migration-usage.rst b/doc/source/admin/live-migration-usage.rst -index 783ab5e..a1e7f18 100644 ---- a/doc/source/admin/live-migration-usage.rst -+++ b/doc/source/admin/live-migration-usage.rst -@@ -320,4 +320,4 @@ - - If live migrations routinely timeout or fail during cleanup operations due - to the user token timing out, consider configuring nova to use --:ref:`service user tokens `. -+:ref:`service user tokens `. -diff --git a/doc/source/admin/migrate-instance-with-snapshot.rst b/doc/source/admin/migrate-instance-with-snapshot.rst -index 6505967..2304310 100644 ---- a/doc/source/admin/migrate-instance-with-snapshot.rst -+++ b/doc/source/admin/migrate-instance-with-snapshot.rst -@@ -67,7 +67,7 @@ - - If snapshot operations routinely fail because the user token times out - while uploading a large disk image, consider configuring nova to use -- :ref:`service user tokens `. -+ :ref:`service user tokens `. - - #. Use the :command:`openstack image list` command to check the status - until the status is ``ACTIVE``: -diff --git a/doc/source/admin/support-compute.rst b/doc/source/admin/support-compute.rst -index 8522e51..31e32fd 100644 ---- a/doc/source/admin/support-compute.rst -+++ b/doc/source/admin/support-compute.rst -@@ -478,67 +478,3 @@ - at ``/v2.1`` instead of ``/v2``. The former route supports microversions, - while the latter route is considered the legacy v2.0 compatibility-mode - route which renders all requests as if they were made on the legacy v2.0 API. -- -- --.. _user_token_timeout: -- --User token times out during long-running operations ----------------------------------------------------- -- --Problem --~~~~~~~ -- --Long-running operations such as live migration or snapshot can sometimes --overrun the expiry of the user token. In such cases, post operations such --as cleaning up after a live migration can fail when the nova-compute service --needs to cleanup resources in other services, such as in the block-storage --(cinder) or networking (neutron) services. -- --For example: -- --.. code-block:: console -- -- 2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec -- 2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled -- 2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client -- 2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance -- 2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401) -- --Solution --~~~~~~~~ -- --Configure nova to use service user tokens to supplement the regular user token --used to initiate the operation. The identity service (keystone) will then --authenticate a request using the service user token if the user token has --already expired. -- --To use, create a service user in the identity service similar as you would when --creating the ``nova`` service user. -- --Then configure the :oslo.config:group:`service_user` section of the nova --configuration file, for example: -- --.. code-block:: ini -- -- [service_user] -- send_service_user_token = True -- auth_type = password -- project_domain_name = Default -- project_name = service -- user_domain_name = Default -- password = secretservice -- username = nova -- auth_url = https://104.130.216.102/identity -- ... -- --And configure the other identity options as necessary for the service user, --much like you would configure nova to work with the image service (glance) --or networking service. -- --.. note:: -- -- Please note that the role of the :oslo.config:group:`service_user` you -- configure needs to be a superset of -- :oslo.config:option:`keystone_authtoken.service_token_roles` (The option -- :oslo.config:option:`keystone_authtoken.service_token_roles` is configured -- in cinder, glance and neutron). -diff --git a/doc/source/install/compute-install-obs.rst b/doc/source/install/compute-install-obs.rst -index c5c1d29..c227b6e 100644 ---- a/doc/source/install/compute-install-obs.rst -+++ b/doc/source/install/compute-install-obs.rst -@@ -92,6 +92,26 @@ - Comment out or remove any other options in the ``[keystone_authtoken]`` - section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: - - .. path /etc/nova/nova.conf -diff --git a/doc/source/install/compute-install-rdo.rst b/doc/source/install/compute-install-rdo.rst -index 0a5ad68..0c6203a 100644 ---- a/doc/source/install/compute-install-rdo.rst -+++ b/doc/source/install/compute-install-rdo.rst -@@ -84,6 +84,26 @@ - Comment out or remove any other options in the ``[keystone_authtoken]`` - section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: - - .. path /etc/nova/nova.conf -diff --git a/doc/source/install/compute-install-ubuntu.rst b/doc/source/install/compute-install-ubuntu.rst -index 8605c73..baf0585 100644 ---- a/doc/source/install/compute-install-ubuntu.rst -+++ b/doc/source/install/compute-install-ubuntu.rst -@@ -74,6 +74,26 @@ - Comment out or remove any other options in the - ``[keystone_authtoken]`` section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: - - .. path /etc/nova/nova.conf -diff --git a/doc/source/install/controller-install-obs.rst b/doc/source/install/controller-install-obs.rst -index 1849961..01b7bb0 100644 ---- a/doc/source/install/controller-install-obs.rst -+++ b/doc/source/install/controller-install-obs.rst -@@ -260,6 +260,26 @@ - Comment out or remove any other options in the ``[keystone_authtoken]`` - section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the - management interface IP address of the controller node: - -diff --git a/doc/source/install/controller-install-rdo.rst b/doc/source/install/controller-install-rdo.rst -index fd24196..b6098f1 100644 ---- a/doc/source/install/controller-install-rdo.rst -+++ b/doc/source/install/controller-install-rdo.rst -@@ -247,6 +247,26 @@ - Comment out or remove any other options in the ``[keystone_authtoken]`` - section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the - management interface IP address of the controller node: - -diff --git a/doc/source/install/controller-install-ubuntu.rst b/doc/source/install/controller-install-ubuntu.rst -index 7282b0b..1363a98 100644 ---- a/doc/source/install/controller-install-ubuntu.rst -+++ b/doc/source/install/controller-install-ubuntu.rst -@@ -237,6 +237,26 @@ - Comment out or remove any other options in the ``[keystone_authtoken]`` - section. - -+ * In the ``[service_user]`` section, configure :ref:`service user -+ tokens `: -+ -+ .. path /etc/nova/nova.conf -+ .. code-block:: ini -+ -+ [service_user] -+ send_service_user_token = true -+ auth_url = https://controller/identity -+ auth_strategy = keystone -+ auth_type = password -+ project_domain_name = Default -+ project_name = service -+ user_domain_name = Default -+ username = nova -+ password = NOVA_PASS -+ -+ Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in -+ the Identity service. -+ - * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the - management interface IP address of the controller node: - -diff --git a/nova/cmd/status.py b/nova/cmd/status.py -index 29e4a5d..4a4e28d 100644 ---- a/nova/cmd/status.py -+++ b/nova/cmd/status.py -@@ -271,6 +271,15 @@ - - return upgradecheck.Result(upgradecheck.Code.SUCCESS) - -+ def _check_service_user_token(self): -+ if not CONF.service_user.send_service_user_token: -+ msg = (_(""" -+Service user token configuration is required for all Nova services. -+For more details see the following: -+https://docs.openstack.org/latest/nova/admin/configuration/service-user-token.html""")) # noqa -+ return upgradecheck.Result(upgradecheck.Code.FAILURE, msg) -+ return upgradecheck.Result(upgradecheck.Code.SUCCESS) -+ - # The format of the check functions is to return an upgradecheck.Result - # object with the appropriate upgradecheck.Code and details set. If the - # check hits warnings or failures then those should be stored in the -@@ -294,6 +303,8 @@ - (_('Older than N-1 computes'), _check_old_computes), - # Added in Wallaby - (_('hw_machine_type unset'), _check_machine_type_set), -+ # Added in Bobcat -+ (_('Service User Token Configuration'), _check_service_user_token), - ) - - -diff --git a/nova/tests/unit/cmd/test_status.py b/nova/tests/unit/cmd/test_status.py -index f5fcc16..c6a0ab2 100644 ---- a/nova/tests/unit/cmd/test_status.py -+++ b/nova/tests/unit/cmd/test_status.py -@@ -446,3 +446,19 @@ - upgradecheck.Code.SUCCESS, - result.code - ) -+ -+ -+class TestUpgradeCheckServiceUserToken(test.NoDBTestCase): -+ -+ def setUp(self): -+ super().setUp() -+ self.cmd = status.UpgradeCommands() -+ -+ def test_service_user_token_not_configured(self): -+ result = self.cmd._check_service_user_token() -+ self.assertEqual(upgradecheck.Code.FAILURE, result.code) -+ -+ def test_service_user_token_configured(self): -+ self.flags(send_service_user_token=True, group='service_user') -+ result = self.cmd._check_service_user_token() -+ self.assertEqual(upgradecheck.Code.SUCCESS, result.code) -diff --git a/nova/tests/unit/virt/hyperv/test_vmops.py b/nova/tests/unit/virt/hyperv/test_vmops.py -index 07e1774..1e3e50f 100644 ---- a/nova/tests/unit/virt/hyperv/test_vmops.py -+++ b/nova/tests/unit/virt/hyperv/test_vmops.py -@@ -1129,7 +1129,7 @@ - mock_unplug_vifs.assert_called_once_with( - mock_instance, mock.sentinel.fake_network_info) - mock_disconnect_volumes.assert_called_once_with( -- mock.sentinel.FAKE_BD_INFO) -+ mock.sentinel.FAKE_BD_INFO, force=True) - mock_delete_disk_files.assert_called_once_with( - mock_instance.name) - -diff --git a/nova/tests/unit/virt/hyperv/test_volumeops.py b/nova/tests/unit/virt/hyperv/test_volumeops.py -index 66d2c25..f289d03 100644 ---- a/nova/tests/unit/virt/hyperv/test_volumeops.py -+++ b/nova/tests/unit/virt/hyperv/test_volumeops.py -@@ -141,7 +141,13 @@ - - self._volumeops.disconnect_volumes(block_device_info) - fake_volume_driver.disconnect_volume.assert_called_once_with( -- block_device_mapping[0]['connection_info']) -+ block_device_mapping[0]['connection_info'], force=False) -+ -+ # Verify force=True -+ fake_volume_driver.disconnect_volume.reset_mock() -+ self._volumeops.disconnect_volumes(block_device_info, force=True) -+ fake_volume_driver.disconnect_volume.assert_called_once_with( -+ block_device_mapping[0]['connection_info'], force=True) - - @mock.patch('time.sleep') - @mock.patch.object(volumeops.VolumeOps, '_get_volume_driver') -@@ -181,7 +187,7 @@ - - if attach_failed: - fake_volume_driver.disconnect_volume.assert_called_once_with( -- fake_conn_info) -+ fake_conn_info, force=False) - mock_sleep.assert_has_calls( - [mock.call(CONF.hyperv.volume_attach_retry_interval)] * - CONF.hyperv.volume_attach_retry_count) -@@ -203,7 +209,13 @@ - mock_get_volume_driver.assert_called_once_with( - mock.sentinel.conn_info) - fake_volume_driver.disconnect_volume.assert_called_once_with( -- mock.sentinel.conn_info) -+ mock.sentinel.conn_info, force=False) -+ -+ # Verify force=True -+ fake_volume_driver.disconnect_volume.reset_mock() -+ self._volumeops.disconnect_volume(mock.sentinel.conn_info, force=True) -+ fake_volume_driver.disconnect_volume.assert_called_once_with( -+ mock.sentinel.conn_info, force=True) - - @mock.patch.object(volumeops.VolumeOps, '_get_volume_driver') - def test_detach_volume(self, mock_get_volume_driver): -@@ -347,7 +359,13 @@ - self._base_vol_driver.disconnect_volume(conn_info) - - self._conn.disconnect_volume.assert_called_once_with( -- conn_info['data']) -+ conn_info['data'], force=False) -+ -+ # Verify force=True -+ self._conn.disconnect_volume.reset_mock() -+ self._base_vol_driver.disconnect_volume(conn_info, force=True) -+ self._conn.disconnect_volume.assert_called_once_with( -+ conn_info['data'], force=True) - - @mock.patch.object(volumeops.BaseVolumeDriver, '_get_disk_res_path') - def _test_get_disk_resource_path_by_conn_info(self, -diff --git a/nova/tests/unit/virt/libvirt/test_driver.py b/nova/tests/unit/virt/libvirt/test_driver.py -index 86e3661..9d0a870 100644 ---- a/nova/tests/unit/virt/libvirt/test_driver.py -+++ b/nova/tests/unit/virt/libvirt/test_driver.py -@@ -9584,7 +9584,7 @@ - drvr._disconnect_volume( - self.context, fake_connection_info, fake_instance_1) - mock_volume_driver.disconnect_volume.assert_called_once_with( -- fake_connection_info, fake_instance_1) -+ fake_connection_info, fake_instance_1, force=False) - - @mock.patch.object(libvirt_driver.LibvirtDriver, '_detach_encryptor') - @mock.patch('nova.objects.InstanceList.get_uuids_by_host') -@@ -9958,7 +9958,12 @@ - device_name='vdc', - ), - mock.call.detach_encryptor(**encryption), -- mock.call.disconnect_volume(connection_info, instance)]) -+ mock.call.disconnect_volume( -+ connection_info, -+ instance, -+ force=False, -+ ) -+ ]) - get_device_conf_func = mock_detach_with_retry.mock_calls[0][1][2] - self.assertEqual(mock_guest.get_disk, get_device_conf_func.func) - self.assertEqual(('vdc',), get_device_conf_func.args) -@@ -20257,16 +20262,64 @@ - self.context, - mock.sentinel.connection_info, - instance, -- destroy_secrets=False -+ destroy_secrets=False, -+ force=True - ), - mock.call( - self.context, - mock.sentinel.connection_info, - instance, -- destroy_secrets=True -+ destroy_secrets=True, -+ force=True - ) - ]) - -+ @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._get_volume_driver') -+ @mock.patch( -+ 'nova.virt.libvirt.driver.LibvirtDriver._should_disconnect_target', -+ new=mock.Mock(return_value=True)) -+ @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._detach_encryptor', -+ new=mock.Mock()) -+ @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._undefine_domain', -+ new=mock.Mock()) -+ @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._get_vpmems', -+ new=mock.Mock(return_value=None)) -+ def test_cleanup_disconnect_volume(self, mock_vol_driver): -+ """Verify that we call disconnect_volume() with force=True -+ -+ cleanup() is called by destroy() when an instance is being deleted and -+ force=True should be passed down to os-brick's disconnect_volume() -+ call, which will ensure removal of devices regardless of errors. -+ -+ We need to ensure that devices are removed when an instance is being -+ deleted to avoid leaving leftover devices that could later be -+ erroneously connected by external entities (example: multipathd) to -+ instances that should not have access to the volumes. -+ -+ See https://bugs.launchpad.net/nova/+bug/2004555 for details. -+ """ -+ connection_info = mock.MagicMock() -+ block_device_info = { -+ 'block_device_mapping': [ -+ { -+ 'connection_info': connection_info -+ } -+ ] -+ } -+ instance = objects.Instance(self.context, **self.test_instance) -+ drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI()) -+ -+ drvr.cleanup( -+ self.context, -+ instance, -+ network_info={}, -+ block_device_info=block_device_info, -+ destroy_vifs=False, -+ destroy_disks=False, -+ ) -+ mock_vol_driver.return_value.disconnect_volume.assert_called_once_with( -+ connection_info, instance, force=True) -+ - @mock.patch.object(libvirt_driver.LibvirtDriver, '_get_volume_encryption') - @mock.patch.object(libvirt_driver.LibvirtDriver, '_allow_native_luksv1') - def test_swap_volume_native_luks_blocked(self, mock_allow_native_luksv1, -diff --git a/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py b/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py -index 0606532..5505465 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py -@@ -81,3 +81,23 @@ - self.assertEqual(requested_size, new_size) - libvirt_driver.connector.extend_volume.assert_called_once_with( - connection_info['data']) -+ -+ def test_disconnect_volume(self): -+ device_path = '/dev/fake-dev' -+ connection_info = {'data': {'device_path': device_path}} -+ -+ libvirt_driver = fibrechannel.LibvirtFibreChannelVolumeDriver( -+ self.fake_host) -+ libvirt_driver.connector.disconnect_volume = mock.MagicMock() -+ libvirt_driver.disconnect_volume( -+ connection_info, mock.sentinel.instance) -+ -+ libvirt_driver.connector.disconnect_volume.assert_called_once_with( -+ connection_info['data'], connection_info['data'], force=False) -+ -+ # Verify force=True -+ libvirt_driver.connector.disconnect_volume.reset_mock() -+ libvirt_driver.disconnect_volume( -+ connection_info, mock.sentinel.instance, force=True) -+ libvirt_driver.connector.disconnect_volume.assert_called_once_with( -+ connection_info['data'], connection_info['data'], force=True) -diff --git a/nova/tests/unit/virt/libvirt/volume/test_iscsi.py b/nova/tests/unit/virt/libvirt/volume/test_iscsi.py -index bd516b1..a1111e0 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_iscsi.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_iscsi.py -@@ -57,10 +57,19 @@ - device=device_path)) - libvirt_driver.disconnect_volume(connection_info, - mock.sentinel.instance) -+ libvirt_driver.connector.disconnect_volume.assert_called_once_with( -+ connection_info['data'], None, force=False) - - msg = mock_LOG_warning.call_args_list[0] - self.assertIn('Ignoring VolumeDeviceNotFound', msg[0][0]) - -+ # Verify force=True -+ libvirt_driver.connector.disconnect_volume.reset_mock() -+ libvirt_driver.disconnect_volume( -+ connection_info, mock.sentinel.instance, force=True) -+ libvirt_driver.connector.disconnect_volume.assert_called_once_with( -+ connection_info['data'], None, force=True) -+ - def test_extend_volume(self): - device_path = '/dev/fake-dev' - connection_info = {'data': {'device_path': device_path}} -diff --git a/nova/tests/unit/virt/libvirt/volume/test_lightos.py b/nova/tests/unit/virt/libvirt/volume/test_lightos.py -index 8a85d73..f97a696 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_lightos.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_lightos.py -@@ -62,7 +62,13 @@ - connection_info = {'data': disk_info} - lightos_driver.disconnect_volume(connection_info, None) - lightos_driver.connector.disconnect_volume.assert_called_once_with( -- disk_info, None) -+ disk_info, None, force=False) -+ -+ # Verify force=True -+ lightos_driver.connector.disconnect_volume.reset_mock() -+ lightos_driver.disconnect_volume(connection_info, None, force=True) -+ lightos_driver.connector.disconnect_volume.assert_called_once_with( -+ disk_info, None, force=True) - - @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', - new=mock.Mock(return_value=mock.Mock())) -diff --git a/nova/tests/unit/virt/libvirt/volume/test_nvme.py b/nova/tests/unit/virt/libvirt/volume/test_nvme.py -index 3f59384..42ef0ad 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_nvme.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_nvme.py -@@ -77,7 +77,13 @@ - connection_info = {'data': disk_info} - nvme_driver.disconnect_volume(connection_info, None) - nvme_driver.connector.disconnect_volume.assert_called_once_with( -- disk_info, None) -+ disk_info, None, force=False) -+ -+ # Verify force=True -+ nvme_driver.connector.disconnect_volume.reset_mock() -+ nvme_driver.disconnect_volume(connection_info, None, force=True) -+ nvme_driver.connector.disconnect_volume.assert_called_once_with( -+ disk_info, None, force=True) - - @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', - new=mock.Mock(return_value=mock.Mock())) -diff --git a/nova/tests/unit/virt/libvirt/volume/test_scaleio.py b/nova/tests/unit/virt/libvirt/volume/test_scaleio.py -index f0fcba1..7d93691 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_scaleio.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_scaleio.py -@@ -49,7 +49,13 @@ - conn = {'data': mock.sentinel.conn_data} - sio.disconnect_volume(conn, mock.sentinel.instance) - sio.connector.disconnect_volume.assert_called_once_with( -- mock.sentinel.conn_data, None) -+ mock.sentinel.conn_data, None, force=False) -+ -+ # Verify force=True -+ sio.connector.disconnect_volume.reset_mock() -+ sio.disconnect_volume(conn, mock.sentinel.instance, force=True) -+ sio.connector.disconnect_volume.assert_called_once_with( -+ mock.sentinel.conn_data, None, force=True) - - @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', - new=mock.Mock(return_value=mock.Mock())) -diff --git a/nova/tests/unit/virt/libvirt/volume/test_storpool.py b/nova/tests/unit/virt/libvirt/volume/test_storpool.py -index 678d4f8..a3252b8 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_storpool.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_storpool.py -@@ -53,9 +53,11 @@ - } - return {'type': 'block', 'path': test_attached[v]['path']} - -- def disconnect_volume(self, connection_info, device_info): -+ def disconnect_volume(self, connection_info, device_info, **kwargs): - self.inst.assertIn('client_id', connection_info) - self.inst.assertIn('volume', connection_info) -+ self.inst.assertIn('force', kwargs) -+ self.inst.assertEqual(self.inst.force, kwargs.get('force')) - - v = connection_info['volume'] - if v not in test_attached: -@@ -86,6 +88,11 @@ - class LibvirtStorPoolVolumeDriverTestCase( - test_volume.LibvirtVolumeBaseTestCase): - -+ def setUp(self): -+ super().setUp() -+ # This is for testing the force flag of disconnect_volume() -+ self.force = False -+ - def mock_storpool(f): - def _config_inner_inner1(inst, *args, **kwargs): - @mock.patch( -@@ -175,3 +182,10 @@ - - libvirt_driver.disconnect_volume(ci_2, mock.sentinel.instance) - self.assertDictEqual({}, test_attached) -+ -+ # Connect the volume again so we can detach it again -+ libvirt_driver.connect_volume(ci_2, mock.sentinel.instance) -+ # Verify force=True -+ self.force = True -+ libvirt_driver.disconnect_volume( -+ ci_2, mock.sentinel.instance, force=True) -diff --git a/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py b/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py -index 168efee..c9e455b 100644 ---- a/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py -+++ b/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py -@@ -95,7 +95,13 @@ - conn = {'data': mock.sentinel.conn_data} - drv.disconnect_volume(conn, mock.sentinel.instance) - drv.connector.disconnect_volume.assert_called_once_with( -- mock.sentinel.conn_data, None) -+ mock.sentinel.conn_data, None, force=False) -+ -+ # Verify force=True -+ drv.connector.disconnect_volume.reset_mock() -+ drv.disconnect_volume(conn, mock.sentinel.instance, force=True) -+ drv.connector.disconnect_volume.assert_called_once_with( -+ mock.sentinel.conn_data, None, force=True) - - def test_libvirt_vzstorage_driver_get_config(self): - libvirt_driver = vzstorage.LibvirtVZStorageVolumeDriver(self.fake_host) -diff --git a/nova/virt/hyperv/vmops.py b/nova/virt/hyperv/vmops.py -index 3ec7e90..08adead 100644 ---- a/nova/virt/hyperv/vmops.py -+++ b/nova/virt/hyperv/vmops.py -@@ -747,7 +747,7 @@ - # should be disconnected even if the VM doesn't exist anymore, - # so they are not leaked. - self.unplug_vifs(instance, network_info) -- self._volumeops.disconnect_volumes(block_device_info) -+ self._volumeops.disconnect_volumes(block_device_info, force=True) - - if destroy_disks: - self._delete_disk_files(instance_name) -diff --git a/nova/virt/hyperv/volumeops.py b/nova/virt/hyperv/volumeops.py -index da5b40f..d2bfed2 100644 ---- a/nova/virt/hyperv/volumeops.py -+++ b/nova/virt/hyperv/volumeops.py -@@ -59,10 +59,10 @@ - for vol in volumes: - self.attach_volume(vol['connection_info'], instance_name) - -- def disconnect_volumes(self, block_device_info): -+ def disconnect_volumes(self, block_device_info, force=False): - mapping = driver.block_device_info_get_mapping(block_device_info) - for vol in mapping: -- self.disconnect_volume(vol['connection_info']) -+ self.disconnect_volume(vol['connection_info'], force=force) - - def attach_volume(self, connection_info, instance_name, - disk_bus=constants.CTRL_TYPE_SCSI): -@@ -116,9 +116,9 @@ - volume_driver.set_disk_qos_specs(connection_info, - qos_specs) - -- def disconnect_volume(self, connection_info): -+ def disconnect_volume(self, connection_info, force=False): - volume_driver = self._get_volume_driver(connection_info) -- volume_driver.disconnect_volume(connection_info) -+ volume_driver.disconnect_volume(connection_info, force=force) - - def detach_volume(self, connection_info, instance_name): - LOG.debug("Detaching volume: %(connection_info)s " -@@ -231,8 +231,8 @@ - def connect_volume(self, connection_info): - return self._connector.connect_volume(connection_info['data']) - -- def disconnect_volume(self, connection_info): -- self._connector.disconnect_volume(connection_info['data']) -+ def disconnect_volume(self, connection_info, force=False): -+ self._connector.disconnect_volume(connection_info['data'], force=force) - - def get_disk_resource_path(self, connection_info): - disk_paths = self._connector.get_volume_paths(connection_info['data']) -diff --git a/nova/virt/libvirt/driver.py b/nova/virt/libvirt/driver.py -index e247107..1cbeda4 100644 ---- a/nova/virt/libvirt/driver.py -+++ b/nova/virt/libvirt/driver.py -@@ -1644,7 +1644,7 @@ - try: - self._disconnect_volume( - context, connection_info, instance, -- destroy_secrets=destroy_secrets) -+ destroy_secrets=destroy_secrets, force=True) - except Exception as exc: - with excutils.save_and_reraise_exception() as ctxt: - if cleanup_instance_disks: -@@ -1961,7 +1961,7 @@ - return (False if connection_count > 1 else True) - - def _disconnect_volume(self, context, connection_info, instance, -- encryption=None, destroy_secrets=True): -+ encryption=None, destroy_secrets=True, force=False): - self._detach_encryptor( - context, - connection_info, -@@ -1973,7 +1973,8 @@ - multiattach = connection_info.get('multiattach', False) - if self._should_disconnect_target( - context, instance, multiattach, vol_driver, volume_id): -- vol_driver.disconnect_volume(connection_info, instance) -+ vol_driver.disconnect_volume( -+ connection_info, instance, force=force) - else: - LOG.info('Detected multiple connections on this host for ' - 'volume: %(volume)s, skipping target disconnect.', -diff --git a/nova/virt/libvirt/volume/fibrechannel.py b/nova/virt/libvirt/volume/fibrechannel.py -index b50db3a..1f890c9 100644 ---- a/nova/virt/libvirt/volume/fibrechannel.py -+++ b/nova/virt/libvirt/volume/fibrechannel.py -@@ -59,7 +59,7 @@ - connection_info['data']['multipath_id'] = \ - device_info['multipath_id'] - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Detach the volume from instance_name.""" - - LOG.debug("calling os-brick to detach FC Volume", instance=instance) -@@ -69,11 +69,12 @@ - # the 2nd param of disconnect_volume and be consistent - # with the rest of the connectors. - self.connector.disconnect_volume(connection_info['data'], -- connection_info['data']) -+ connection_info['data'], -+ force=force) - LOG.debug("Disconnected FC Volume", instance=instance) - - super(LibvirtFibreChannelVolumeDriver, -- self).disconnect_volume(connection_info, instance) -+ self).disconnect_volume(connection_info, instance, force=force) - - def extend_volume(self, connection_info, instance, requested_size): - """Extend the volume.""" -diff --git a/nova/virt/libvirt/volume/fs.py b/nova/virt/libvirt/volume/fs.py -index 5fb9af4..992ef45 100644 ---- a/nova/virt/libvirt/volume/fs.py -+++ b/nova/virt/libvirt/volume/fs.py -@@ -116,7 +116,7 @@ - connection_info['data']['device_path'] = \ - self._get_device_path(connection_info) - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Disconnect the volume.""" - vol_name = connection_info['data']['name'] - mountpoint = self._get_mount_path(connection_info) -diff --git a/nova/virt/libvirt/volume/iscsi.py b/nova/virt/libvirt/volume/iscsi.py -index 564bac1..2b25972a 100644 ---- a/nova/virt/libvirt/volume/iscsi.py -+++ b/nova/virt/libvirt/volume/iscsi.py -@@ -66,19 +66,20 @@ - - connection_info['data']['device_path'] = device_info['path'] - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Detach the volume from instance_name.""" - - LOG.debug("calling os-brick to detach iSCSI Volume", instance=instance) - try: -- self.connector.disconnect_volume(connection_info['data'], None) -+ self.connector.disconnect_volume( -+ connection_info['data'], None, force=force) - except os_brick_exception.VolumeDeviceNotFound as exc: - LOG.warning('Ignoring VolumeDeviceNotFound: %s', exc) - return - LOG.debug("Disconnected iSCSI Volume", instance=instance) - - super(LibvirtISCSIVolumeDriver, -- self).disconnect_volume(connection_info, instance) -+ self).disconnect_volume(connection_info, instance, force=force) - - def extend_volume(self, connection_info, instance, requested_size): - """Extend the volume.""" -diff --git a/nova/virt/libvirt/volume/lightos.py b/nova/virt/libvirt/volume/lightos.py -index d6d3939..6a22bf6 100644 ---- a/nova/virt/libvirt/volume/lightos.py -+++ b/nova/virt/libvirt/volume/lightos.py -@@ -42,14 +42,15 @@ - LOG.debug("Connecting NVMe volume with device_info %s", device_info) - connection_info['data']['device_path'] = device_info['path'] - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Detach the volume from the instance.""" - LOG.debug("Disconnecting NVMe disk. instance:%s, volume_id:%s", - connection_info.get("instance", ""), - connection_info.get("volume_id", "")) -- self.connector.disconnect_volume(connection_info['data'], None) -+ self.connector.disconnect_volume( -+ connection_info['data'], None, force=force) - super(LibvirtLightOSVolumeDriver, self).disconnect_volume( -- connection_info, instance) -+ connection_info, instance, force=force) - - def extend_volume(self, connection_info, instance, requested_size=None): - """Extend the volume.""" -diff --git a/nova/virt/libvirt/volume/nvme.py b/nova/virt/libvirt/volume/nvme.py -index 7436552..e2977c3 100644 ---- a/nova/virt/libvirt/volume/nvme.py -+++ b/nova/virt/libvirt/volume/nvme.py -@@ -45,13 +45,13 @@ - - connection_info['data']['device_path'] = device_info['path'] - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Detach the volume from the instance.""" - LOG.debug("Disconnecting NVMe disk", instance=instance) - self.connector.disconnect_volume( -- connection_info['data'], None) -+ connection_info['data'], None, force=force) - super(LibvirtNVMEVolumeDriver, -- self).disconnect_volume(connection_info, instance) -+ self).disconnect_volume(connection_info, instance, force=force) - - def extend_volume(self, connection_info, instance, requested_size): - """Extend the volume.""" -diff --git a/nova/virt/libvirt/volume/quobyte.py b/nova/virt/libvirt/volume/quobyte.py -index bb7a770..2eb4bcfb 100644 ---- a/nova/virt/libvirt/volume/quobyte.py -+++ b/nova/virt/libvirt/volume/quobyte.py -@@ -189,7 +189,7 @@ - instance=instance) - - @utils.synchronized('connect_qb_volume') -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Disconnect the volume.""" - - mount_path = self._get_mount_path(connection_info) -diff --git a/nova/virt/libvirt/volume/scaleio.py b/nova/virt/libvirt/volume/scaleio.py -index 7c414c2..04a9423 100644 ---- a/nova/virt/libvirt/volume/scaleio.py -+++ b/nova/virt/libvirt/volume/scaleio.py -@@ -57,12 +57,13 @@ - instance=instance) - connection_info['data']['device_path'] = device_info['path'] - -- def disconnect_volume(self, connection_info, instance): -- self.connector.disconnect_volume(connection_info['data'], None) -+ def disconnect_volume(self, connection_info, instance, force=False): -+ self.connector.disconnect_volume( -+ connection_info['data'], None, force=force) - LOG.debug("Disconnected volume", instance=instance) - - super(LibvirtScaleIOVolumeDriver, self).disconnect_volume( -- connection_info, instance) -+ connection_info, instance, force=force) - - def extend_volume(self, connection_info, instance, requested_size): - LOG.debug("calling os-brick to extend ScaleIO Volume", -diff --git a/nova/virt/libvirt/volume/smbfs.py b/nova/virt/libvirt/volume/smbfs.py -index d112af7..9de1ce2 100644 ---- a/nova/virt/libvirt/volume/smbfs.py -+++ b/nova/virt/libvirt/volume/smbfs.py -@@ -52,7 +52,7 @@ - device_path = self._get_device_path(connection_info) - connection_info['data']['device_path'] = device_path - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Disconnect the volume.""" - smbfs_share = connection_info['data']['export'] - mount_path = self._get_mount_path(connection_info) -diff --git a/nova/virt/libvirt/volume/storpool.py b/nova/virt/libvirt/volume/storpool.py -index 0e71221..e6dffca 100644 ---- a/nova/virt/libvirt/volume/storpool.py -+++ b/nova/virt/libvirt/volume/storpool.py -@@ -47,10 +47,11 @@ - device_info, instance=instance) - connection_info['data']['device_path'] = device_info['path'] - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - LOG.debug("Detaching StorPool volume %s", - connection_info['data']['volume'], instance=instance) -- self.connector.disconnect_volume(connection_info['data'], None) -+ self.connector.disconnect_volume( -+ connection_info['data'], None, force=force) - LOG.debug("Detached StorPool volume", instance=instance) - - def extend_volume(self, connection_info, instance, requested_size): -diff --git a/nova/virt/libvirt/volume/volume.py b/nova/virt/libvirt/volume/volume.py -index 6d650c8..f76c361 100644 ---- a/nova/virt/libvirt/volume/volume.py -+++ b/nova/virt/libvirt/volume/volume.py -@@ -135,7 +135,7 @@ - """Connect the volume.""" - pass - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Disconnect the volume.""" - pass - -diff --git a/nova/virt/libvirt/volume/vzstorage.py b/nova/virt/libvirt/volume/vzstorage.py -index 85ffb45..babfdef 100644 ---- a/nova/virt/libvirt/volume/vzstorage.py -+++ b/nova/virt/libvirt/volume/vzstorage.py -@@ -126,9 +126,10 @@ - - return _connect_volume(connection_info, instance) - -- def disconnect_volume(self, connection_info, instance): -+ def disconnect_volume(self, connection_info, instance, force=False): - """Detach the volume from instance_name.""" - LOG.debug("calling os-brick to detach Vzstorage Volume", - instance=instance) -- self.connector.disconnect_volume(connection_info['data'], None) -+ self.connector.disconnect_volume( -+ connection_info['data'], None, force=force) - LOG.debug("Disconnected Vzstorage Volume", instance=instance) -diff --git a/releasenotes/notes/service-user-token-421d067c16257782.yaml b/releasenotes/notes/service-user-token-421d067c16257782.yaml -new file mode 100644 -index 0000000..d3af14f ---- /dev/null -+++ b/releasenotes/notes/service-user-token-421d067c16257782.yaml -@@ -0,0 +1,11 @@ -+upgrade: -+ - | -+ Configuration of service user tokens is now **required** for all Nova services -+ to ensure security of block-storage volume data. -+ -+ All Nova configuration files must configure the ``[service_user]`` section as -+ described in the `documentation`__. -+ -+ See https://bugs.launchpad.net/nova/+bug/2004555 for more details. -+ -+ __ https://docs.openstack.org/nova/latest/admin/configuration/service-user-token.html diff -Nru nova-26.1.0/debian/patches/CVE-2024-32498_1_nova-stable-2023.1_Reject_qcow_files_with_data-file_attributes.patch nova-26.2.2/debian/patches/CVE-2024-32498_1_nova-stable-2023.1_Reject_qcow_files_with_data-file_attributes.patch --- nova-26.1.0/debian/patches/CVE-2024-32498_1_nova-stable-2023.1_Reject_qcow_files_with_data-file_attributes.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-32498_1_nova-stable-2023.1_Reject_qcow_files_with_data-file_attributes.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,87 @@ +Description: [PATCH 1/3] Reject qcow files with data-file attributes +Author: Dan Smith +Date: Mon, 1 Apr 2024 07:32:11 -0700 +Change-Id: Ic3fa16f55acc38cf6c1a4ac1dce4487225e66d04 +Bug: https://launchpad.net/bugs/2059809 +Bug-Debian: https://bugs.debian.org/1074762 +Origin: upstream, https://review.opendev.org/c/openstack/nova/+/923273 +Last-Update: 2024-06-30 + +diff --git a/nova/tests/unit/virt/libvirt/test_utils.py b/nova/tests/unit/virt/libvirt/test_utils.py +index 37744ea9f7..015da7e222 100644 +--- a/nova/tests/unit/virt/libvirt/test_utils.py ++++ b/nova/tests/unit/virt/libvirt/test_utils.py +@@ -419,6 +419,7 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + FakeImgInfo.file_format = file_format + FakeImgInfo.backing_file = backing_file + FakeImgInfo.virtual_size = 1 ++ FakeImgInfo.format_specific = None if file_format == 'raw' else {} + + return FakeImgInfo() + +diff --git a/nova/tests/unit/virt/test_images.py b/nova/tests/unit/virt/test_images.py +index 62a61c1e8b..272a1cae36 100644 +--- a/nova/tests/unit/virt/test_images.py ++++ b/nova/tests/unit/virt/test_images.py +@@ -112,6 +112,37 @@ class QemuTestCase(test.NoDBTestCase): + images.fetch_to_raw, + None, 'href123', '/no/path') + ++ @mock.patch.object(images, 'convert_image', ++ side_effect=exception.ImageUnacceptable) ++ @mock.patch.object(images, 'qemu_img_info') ++ @mock.patch.object(images, 'fetch') ++ def test_fetch_to_raw_data_file(self, convert_image, qemu_img_info_fn, ++ fetch): ++ # NOTE(danms): the above test needs the following line as well, as it ++ # is broken without it. ++ qemu_img_info = qemu_img_info_fn.return_value ++ qemu_img_info.backing_file = None ++ qemu_img_info.file_format = 'qcow2' ++ qemu_img_info.virtual_size = 20 ++ qemu_img_info.format_specific = {'data': {'data-file': 'somefile'}} ++ self.assertRaisesRegex(exception.ImageUnacceptable, ++ 'Image href123 is unacceptable.*somefile', ++ images.fetch_to_raw, ++ None, 'href123', '/no/path') ++ ++ @mock.patch('os.rename') ++ @mock.patch.object(images, 'qemu_img_info') ++ @mock.patch.object(images, 'fetch') ++ def test_fetch_to_raw_from_raw(self, fetch, qemu_img_info_fn, mock_rename): ++ # Make sure we support a case where we fetch an already-raw image and ++ # qemu-img returns None for "format_specific". ++ qemu_img_info = qemu_img_info_fn.return_value ++ qemu_img_info.file_format = 'raw' ++ qemu_img_info.backing_file = None ++ qemu_img_info.format_specific = None ++ images.fetch_to_raw(None, 'href123', '/no/path') ++ mock_rename.assert_called_once_with('/no/path.part', '/no/path') ++ + @mock.patch.object(compute_utils, 'disk_ops_semaphore') + @mock.patch('nova.privsep.utils.supports_direct_io', return_value=True) + @mock.patch('oslo_concurrency.processutils.execute') +diff --git a/nova/virt/images.py b/nova/virt/images.py +index f13c872290..5f80a1d075 100644 +--- a/nova/virt/images.py ++++ b/nova/virt/images.py +@@ -157,6 +157,15 @@ def fetch_to_raw(context, image_href, path, trusted_certs=None): + reason=(_("fmt=%(fmt)s backed by: %(backing_file)s") % + {'fmt': fmt, 'backing_file': backing_file})) + ++ try: ++ data_file = data.format_specific['data']['data-file'] ++ except (KeyError, TypeError, AttributeError): ++ data_file = None ++ if data_file is not None: ++ raise exception.ImageUnacceptable(image_id=image_href, ++ reason=(_("fmt=%(fmt)s has data-file: %(data_file)s") % ++ {'fmt': fmt, 'data_file': data_file})) ++ + if fmt == 'vmdk': + check_vmdk_image(image_href, data) + +-- +2.41.0 + + diff -Nru nova-26.1.0/debian/patches/CVE-2024-32498_2_nova-stable-2023.1_Check_images_with_format_inspector_for_safety_2.patch nova-26.2.2/debian/patches/CVE-2024-32498_2_nova-stable-2023.1_Check_images_with_format_inspector_for_safety_2.patch --- nova-26.1.0/debian/patches/CVE-2024-32498_2_nova-stable-2023.1_Check_images_with_format_inspector_for_safety_2.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-32498_2_nova-stable-2023.1_Check_images_with_format_inspector_for_safety_2.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,1316 @@ +From 67e5376dd64407f5aaf1ea5f8c896e356064a2c9 Mon Sep 17 00:00:00 2001 +From: Dan Smith +Date: Wed, 17 Apr 2024 07:06:13 -0700 +Subject: [PATCH] Check images with format_inspector for safety + +It has been asserted that we should not be calling qemu-img info +on untrusted files. That means we need to know if they have a +backing_file, data_file or other unsafe configuration *before* we use +qemu-img to probe or convert them. + +This grafts glance's format_inspector module into nova/images so we +can use it to check the file early for safety. The expectation is that +this will be moved to oslo.utils (or something) later and thus we will +just delete the file from nova and change our import when that happens. + +NOTE: This includes whitespace changes from the glance version of +format_inspector.py because of autopep8 demands. + +Change-Id: Iaefbe41b4c4bf0cf95d8f621653fdf65062aaa59 +Closes-Bug: #2059809 +(cherry picked from commit 9cdce715945619fc851ab3f43c97fab4bae4e35a) +(cherry picked from commit f07fa55fd86726eeafcd4c0c687bc49dd4df9f4c) +(cherry picked from commit 0acf5ee7b5dfb6ff0f9a9745f5ad2a0ed2bf65bf) +--- + +Index: nova/nova/conf/workarounds.py +=================================================================== +--- nova.orig/nova/conf/workarounds.py ++++ nova/nova/conf/workarounds.py +@@ -439,6 +439,16 @@ Howerver, if you don't use automatic cle + extra delay before and Ironic node is available for building a + new Nova instance. + """), ++ cfg.BoolOpt( ++ 'disable_deep_image_inspection', ++ default=False, ++ help=""" ++This disables the additional deep image inspection that the compute node does ++when downloading from glance. This includes backing-file, data-file, and ++known-features detection *before* passing the image to qemu-img. Generally, ++this inspection should be enabled for maximum safety, but this workaround ++option allows disabling it if there is a compatibility concern. ++"""), + ] + + +Index: nova/nova/image/format_inspector.py +=================================================================== +--- /dev/null ++++ nova/nova/image/format_inspector.py +@@ -0,0 +1,889 @@ ++# Copyright 2020 Red Hat, Inc ++# All Rights Reserved. ++# ++# Licensed under the Apache License, Version 2.0 (the "License"); you may ++# not use this file except in compliance with the License. You may obtain ++# a copy of the License at ++# ++# http://www.apache.org/licenses/LICENSE-2.0 ++# ++# Unless required by applicable law or agreed to in writing, software ++# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT ++# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the ++# License for the specific language governing permissions and limitations ++# under the License. ++ ++""" ++This is a python implementation of virtual disk format inspection routines ++gathered from various public specification documents, as well as qemu disk ++driver code. It attempts to store and parse the minimum amount of data ++required, and in a streaming-friendly manner to collect metadata about ++complex-format images. ++""" ++ ++import struct ++ ++from oslo_log import log as logging ++ ++LOG = logging.getLogger(__name__) ++ ++ ++def chunked_reader(fileobj, chunk_size=512): ++ while True: ++ chunk = fileobj.read(chunk_size) ++ if not chunk: ++ break ++ yield chunk ++ ++ ++class CaptureRegion(object): ++ """Represents a region of a file we want to capture. ++ ++ A region of a file we want to capture requires a byte offset into ++ the file and a length. This is expected to be used by a data ++ processing loop, calling capture() with the most recently-read ++ chunk. This class handles the task of grabbing the desired region ++ of data across potentially multiple fractional and unaligned reads. ++ ++ :param offset: Byte offset into the file starting the region ++ :param length: The length of the region ++ """ ++ ++ def __init__(self, offset, length): ++ self.offset = offset ++ self.length = length ++ self.data = b'' ++ ++ @property ++ def complete(self): ++ """Returns True when we have captured the desired data.""" ++ return self.length == len(self.data) ++ ++ def capture(self, chunk, current_position): ++ """Process a chunk of data. ++ ++ This should be called for each chunk in the read loop, at least ++ until complete returns True. ++ ++ :param chunk: A chunk of bytes in the file ++ :param current_position: The position of the file processed by the ++ read loop so far. Note that this will be ++ the position in the file *after* the chunk ++ being presented. ++ """ ++ read_start = current_position - len(chunk) ++ if (read_start <= self.offset <= current_position or ++ self.offset <= read_start <= (self.offset + self.length)): ++ if read_start < self.offset: ++ lead_gap = self.offset - read_start ++ else: ++ lead_gap = 0 ++ self.data += chunk[lead_gap:] ++ self.data = self.data[:self.length] ++ ++ ++class ImageFormatError(Exception): ++ """An unrecoverable image format error that aborts the process.""" ++ pass ++ ++ ++class TraceDisabled(object): ++ """A logger-like thing that swallows tracing when we do not want it.""" ++ ++ def debug(self, *a, **k): ++ pass ++ ++ info = debug ++ warning = debug ++ error = debug ++ ++ ++class FileInspector(object): ++ """A stream-based disk image inspector. ++ ++ This base class works on raw images and is subclassed for more ++ complex types. It is to be presented with the file to be examined ++ one chunk at a time, during read processing and will only store ++ as much data as necessary to determine required attributes of ++ the file. ++ """ ++ ++ def __init__(self, tracing=False): ++ self._total_count = 0 ++ ++ # NOTE(danms): The logging in here is extremely verbose for a reason, ++ # but should never really be enabled at that level at runtime. To ++ # retain all that work and assist in future debug, we have a separate ++ # debug flag that can be passed from a manual tool to turn it on. ++ if tracing: ++ self._log = logging.getLogger(str(self)) ++ else: ++ self._log = TraceDisabled() ++ self._capture_regions = {} ++ ++ def _capture(self, chunk, only=None): ++ for name, region in self._capture_regions.items(): ++ if only and name not in only: ++ continue ++ if not region.complete: ++ region.capture(chunk, self._total_count) ++ ++ def eat_chunk(self, chunk): ++ """Call this to present chunks of the file to the inspector.""" ++ pre_regions = set(self._capture_regions.keys()) ++ ++ # Increment our position-in-file counter ++ self._total_count += len(chunk) ++ ++ # Run through the regions we know of to see if they want this ++ # data ++ self._capture(chunk) ++ ++ # Let the format do some post-read processing of the stream ++ self.post_process() ++ ++ # Check to see if the post-read processing added new regions ++ # which may require the current chunk. ++ new_regions = set(self._capture_regions.keys()) - pre_regions ++ if new_regions: ++ self._capture(chunk, only=new_regions) ++ ++ def post_process(self): ++ """Post-read hook to process what has been read so far. ++ ++ This will be called after each chunk is read and potentially captured ++ by the defined regions. If any regions are defined by this call, ++ those regions will be presented with the current chunk in case it ++ is within one of the new regions. ++ """ ++ pass ++ ++ def region(self, name): ++ """Get a CaptureRegion by name.""" ++ return self._capture_regions[name] ++ ++ def new_region(self, name, region): ++ """Add a new CaptureRegion by name.""" ++ if self.has_region(name): ++ # This is a bug, we tried to add the same region twice ++ raise ImageFormatError('Inspector re-added region %s' % name) ++ self._capture_regions[name] = region ++ ++ def has_region(self, name): ++ """Returns True if named region has been defined.""" ++ return name in self._capture_regions ++ ++ @property ++ def format_match(self): ++ """Returns True if the file appears to be the expected format.""" ++ return True ++ ++ @property ++ def virtual_size(self): ++ """Returns the virtual size of the disk image, or zero if unknown.""" ++ return self._total_count ++ ++ @property ++ def actual_size(self): ++ """Returns the total size of the file, usually smaller than ++ virtual_size. NOTE: this will only be accurate if the entire ++ file is read and processed. ++ """ ++ return self._total_count ++ ++ @property ++ def complete(self): ++ """Returns True if we have all the information needed.""" ++ return all(r.complete for r in self._capture_regions.values()) ++ ++ def __str__(self): ++ """The string name of this file format.""" ++ return 'raw' ++ ++ @property ++ def context_info(self): ++ """Return info on amount of data held in memory for auditing. ++ ++ This is a dict of region:sizeinbytes items that the inspector ++ uses to examine the file. ++ """ ++ return {name: len(region.data) for name, region in ++ self._capture_regions.items()} ++ ++ @classmethod ++ def from_file(cls, filename): ++ """Read as much of a file as necessary to complete inspection. ++ ++ NOTE: Because we only read as much of the file as necessary, the ++ actual_size property will not reflect the size of the file, but the ++ amount of data we read before we satisfied the inspector. ++ ++ Raises ImageFormatError if we cannot parse the file. ++ """ ++ inspector = cls() ++ with open(filename, 'rb') as f: ++ for chunk in chunked_reader(f): ++ inspector.eat_chunk(chunk) ++ if inspector.complete: ++ # No need to eat any more data ++ break ++ if not inspector.complete or not inspector.format_match: ++ raise ImageFormatError('File is not in requested format') ++ return inspector ++ ++ def safety_check(self): ++ """Perform some checks to determine if this file is safe. ++ ++ Returns True if safe, False otherwise. It may raise ImageFormatError ++ if safety cannot be guaranteed because of parsing or other errors. ++ """ ++ return True ++ ++ ++# The qcow2 format consists of a big-endian 72-byte header, of which ++# only a small portion has information we care about: ++# ++# Dec Hex Name ++# 0 0x00 Magic 4-bytes 'QFI\xfb' ++# 4 0x04 Version (uint32_t, should always be 2 for modern files) ++# . . . ++# 8 0x08 Backing file offset (uint64_t) ++# 24 0x18 Size in bytes (unint64_t) ++# . . . ++# 72 0x48 Incompatible features bitfield (6 bytes) ++# ++# https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qcow2.txt ++class QcowInspector(FileInspector): ++ """QEMU QCOW2 Format ++ ++ This should only require about 32 bytes of the beginning of the file ++ to determine the virtual size, and 104 bytes to perform the safety check. ++ """ ++ ++ BF_OFFSET = 0x08 ++ BF_OFFSET_LEN = 8 ++ I_FEATURES = 0x48 ++ I_FEATURES_LEN = 8 ++ I_FEATURES_DATAFILE_BIT = 3 ++ I_FEATURES_MAX_BIT = 4 ++ ++ def __init__(self, *a, **k): ++ super(QcowInspector, self).__init__(*a, **k) ++ self.new_region('header', CaptureRegion(0, 512)) ++ ++ def _qcow_header_data(self): ++ magic, version, bf_offset, bf_sz, cluster_bits, size = ( ++ struct.unpack('>4sIQIIQ', self.region('header').data[:32])) ++ return magic, size ++ ++ @property ++ def has_header(self): ++ return self.region('header').complete ++ ++ @property ++ def virtual_size(self): ++ if not self.region('header').complete: ++ return 0 ++ if not self.format_match: ++ return 0 ++ magic, size = self._qcow_header_data() ++ return size ++ ++ @property ++ def format_match(self): ++ if not self.region('header').complete: ++ return False ++ magic, size = self._qcow_header_data() ++ return magic == b'QFI\xFB' ++ ++ @property ++ def has_backing_file(self): ++ if not self.region('header').complete: ++ return None ++ if not self.format_match: ++ return False ++ bf_offset_bytes = self.region('header').data[ ++ self.BF_OFFSET:self.BF_OFFSET + self.BF_OFFSET_LEN] ++ # nonzero means "has a backing file" ++ bf_offset, = struct.unpack('>Q', bf_offset_bytes) ++ return bf_offset != 0 ++ ++ @property ++ def has_unknown_features(self): ++ if not self.region('header').complete: ++ return None ++ if not self.format_match: ++ return False ++ i_features = self.region('header').data[ ++ self.I_FEATURES:self.I_FEATURES + self.I_FEATURES_LEN] ++ ++ # This is the maximum byte number we should expect any bits to be set ++ max_byte = self.I_FEATURES_MAX_BIT // 8 ++ ++ # The flag bytes are in big-endian ordering, so if we process ++ # them in index-order, they're reversed ++ for i, byte_num in enumerate(reversed(range(self.I_FEATURES_LEN))): ++ if byte_num == max_byte: ++ # If we're in the max-allowed byte, allow any bits less than ++ # the maximum-known feature flag bit to be set ++ allow_mask = ((1 << self.I_FEATURES_MAX_BIT) - 1) ++ elif byte_num > max_byte: ++ # If we're above the byte with the maximum known feature flag ++ # bit, then we expect all zeroes ++ allow_mask = 0x0 ++ else: ++ # Any earlier-than-the-maximum byte can have any of the flag ++ # bits set ++ allow_mask = 0xFF ++ ++ if i_features[i] & ~allow_mask: ++ LOG.warning('Found unknown feature bit in byte %i: %s/%s', ++ byte_num, bin(i_features[byte_num] & ~allow_mask), ++ bin(allow_mask)) ++ return True ++ ++ return False ++ ++ @property ++ def has_data_file(self): ++ if not self.region('header').complete: ++ return None ++ if not self.format_match: ++ return False ++ i_features = self.region('header').data[ ++ self.I_FEATURES:self.I_FEATURES + self.I_FEATURES_LEN] ++ ++ # First byte of bitfield, which is i_features[7] ++ byte = self.I_FEATURES_LEN - 1 - self.I_FEATURES_DATAFILE_BIT // 8 ++ # Third bit of bitfield, which is 0x04 ++ bit = 1 << (self.I_FEATURES_DATAFILE_BIT - 1 % 8) ++ return bool(i_features[byte] & bit) ++ ++ def __str__(self): ++ return 'qcow2' ++ ++ def safety_check(self): ++ return (not self.has_backing_file and ++ not self.has_data_file and ++ not self.has_unknown_features) ++ ++ ++# The VHD (or VPC as QEMU calls it) format consists of a big-endian ++# 512-byte "footer" at the beginning of the file with various ++# information, most of which does not matter to us: ++# ++# Dec Hex Name ++# 0 0x00 Magic string (8-bytes, always 'conectix') ++# 40 0x28 Disk size (uint64_t) ++# ++# https://github.com/qemu/qemu/blob/master/block/vpc.c ++class VHDInspector(FileInspector): ++ """Connectix/MS VPC VHD Format ++ ++ This should only require about 512 bytes of the beginning of the file ++ to determine the virtual size. ++ """ ++ ++ def __init__(self, *a, **k): ++ super(VHDInspector, self).__init__(*a, **k) ++ self.new_region('header', CaptureRegion(0, 512)) ++ ++ @property ++ def format_match(self): ++ return self.region('header').data.startswith(b'conectix') ++ ++ @property ++ def virtual_size(self): ++ if not self.region('header').complete: ++ return 0 ++ ++ if not self.format_match: ++ return 0 ++ ++ return struct.unpack('>Q', self.region('header').data[40:48])[0] ++ ++ def __str__(self): ++ return 'vhd' ++ ++ ++# The VHDX format consists of a complex dynamic little-endian ++# structure with multiple regions of metadata and data, linked by ++# offsets with in the file (and within regions), identified by MSFT ++# GUID strings. The header is a 320KiB structure, only a few pieces of ++# which we actually need to capture and interpret: ++# ++# Dec Hex Name ++# 0 0x00000 Identity (Technically 9-bytes, padded to 64KiB, the first ++# 8 bytes of which are 'vhdxfile') ++# 196608 0x30000 The Region table (64KiB of a 32-byte header, followed ++# by up to 2047 36-byte region table entry structures) ++# ++# The region table header includes two items we need to read and parse, ++# which are: ++# ++# 196608 0x30000 4-byte signature ('regi') ++# 196616 0x30008 Entry count (uint32-t) ++# ++# The region table entries follow the region table header immediately ++# and are identified by a 16-byte GUID, and provide an offset of the ++# start of that region. We care about the "metadata region", identified ++# by the METAREGION class variable. The region table entry is (offsets ++# from the beginning of the entry, since it could be in multiple places): ++# ++# 0 0x00000 16-byte MSFT GUID ++# 16 0x00010 Offset of the actual metadata region (uint64_t) ++# ++# When we find the METAREGION table entry, we need to grab that offset ++# and start examining the region structure at that point. That ++# consists of a metadata table of structures, which point to places in ++# the data in an unstructured space that follows. The header is ++# (offsets relative to the region start): ++# ++# 0 0x00000 8-byte signature ('metadata') ++# . . . ++# 16 0x00010 2-byte entry count (up to 2047 entries max) ++# ++# This header is followed by the specified number of metadata entry ++# structures, identified by GUID: ++# ++# 0 0x00000 16-byte MSFT GUID ++# 16 0x00010 4-byte offset (uint32_t, relative to the beginning of ++# the metadata region) ++# ++# We need to find the "Virtual Disk Size" metadata item, identified by ++# the GUID in the VIRTUAL_DISK_SIZE class variable, grab the offset, ++# add it to the offset of the metadata region, and examine that 8-byte ++# chunk of data that follows. ++# ++# The "Virtual Disk Size" is a naked uint64_t which contains the size ++# of the virtual disk, and is our ultimate target here. ++# ++# https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-vhdx/83e061f8-f6e2-4de1-91bd-5d518a43d477 ++class VHDXInspector(FileInspector): ++ """MS VHDX Format ++ ++ This requires some complex parsing of the stream. The first 256KiB ++ of the image is stored to get the header and region information, ++ and then we capture the first metadata region to read those ++ records, find the location of the virtual size data and parse ++ it. This needs to store the metadata table entries up until the ++ VDS record, which may consist of up to 2047 32-byte entries at ++ max. Finally, it must store a chunk of data at the offset of the ++ actual VDS uint64. ++ ++ """ ++ METAREGION = '8B7CA206-4790-4B9A-B8FE-575F050F886E' ++ VIRTUAL_DISK_SIZE = '2FA54224-CD1B-4876-B211-5DBED83BF4B8' ++ VHDX_METADATA_TABLE_MAX_SIZE = 32 * 2048 # From qemu ++ ++ def __init__(self, *a, **k): ++ super(VHDXInspector, self).__init__(*a, **k) ++ self.new_region('ident', CaptureRegion(0, 32)) ++ self.new_region('header', CaptureRegion(192 * 1024, 64 * 1024)) ++ ++ def post_process(self): ++ # After reading a chunk, we may have the following conditions: ++ # ++ # 1. We may have just completed the header region, and if so, ++ # we need to immediately read and calculate the location of ++ # the metadata region, as it may be starting in the same ++ # read we just did. ++ # 2. We may have just completed the metadata region, and if so, ++ # we need to immediately calculate the location of the ++ # "virtual disk size" record, as it may be starting in the ++ # same read we just did. ++ if self.region('header').complete and not self.has_region('metadata'): ++ region = self._find_meta_region() ++ if region: ++ self.new_region('metadata', region) ++ elif self.has_region('metadata') and not self.has_region('vds'): ++ region = self._find_meta_entry(self.VIRTUAL_DISK_SIZE) ++ if region: ++ self.new_region('vds', region) ++ ++ @property ++ def format_match(self): ++ return self.region('ident').data.startswith(b'vhdxfile') ++ ++ @staticmethod ++ def _guid(buf): ++ """Format a MSFT GUID from the 16-byte input buffer.""" ++ guid_format = '= 2048: ++ raise ImageFormatError('Region count is %i (limit 2047)' % count) ++ ++ # Process the regions until we find the metadata one; grab the ++ # offset and return ++ self._log.debug('Region entry first is %x', region_entry_first) ++ self._log.debug('Region entries %i', count) ++ meta_offset = 0 ++ for i in range(0, count): ++ entry_start = region_entry_first + (i * 32) ++ entry_end = entry_start + 32 ++ entry = self.region('header').data[entry_start:entry_end] ++ self._log.debug('Entry offset is %x', entry_start) ++ ++ # GUID is the first 16 bytes ++ guid = self._guid(entry[:16]) ++ if guid == self.METAREGION: ++ # This entry is the metadata region entry ++ meta_offset, meta_len, meta_req = struct.unpack( ++ '= 2048: ++ raise ImageFormatError( ++ 'Metadata item count is %i (limit 2047)' % count) ++ ++ for i in range(0, count): ++ entry_offset = 32 + (i * 32) ++ guid = self._guid(meta_buffer[entry_offset:entry_offset + 16]) ++ if guid == desired_guid: ++ # Found the item we are looking for by id. ++ # Stop our region from capturing ++ item_offset, item_length, _reserved = struct.unpack( ++ ' +Date: Mon, 24 Jun 2024 09:09:36 -0700 +Subject: [PATCH] Additional qemu safety checking on base images + +There is an additional way we can be fooled into using a qcow2 file +with a data-file, which is uploading it as raw to glance and then +booting an instance from it. Because when we go to create the +ephemeral disk from a cached base image, we've lost the information +about the original source's format, we probe the image's file type +without a strict format specified. If a qcow2 file is listed in +glance as a raw, we won't notice it until it is too late. + +This brings over another piece of code (proposed against) glance's +format inspector which provides a safe format detection routine. This +patch uses that to detect the format of and run a safety check on the +base image each time we go to use it to create an ephemeral disk +image from it. + +This also detects QED files and always marks them as unsafe as we do +not support that format at all. Since we could be fooled into +downloading one and passing it to qemu-img if we don't recognize it, +we need to detect and reject it as unsafe. + +Change-Id: I4881c8cbceb30c1ff2d2b859c554e0d02043f1f5 +(cherry picked from commit b1b88bf001757546fbbea959f4b73cb344407dfb) +(cherry picked from commit 8a0d5f2afaf40c4554419a0b2488ce092eda7a1a) +(cherry picked from commit 0269234dc42fe2c320dc4696123cf5132642f9b7) +--- + +Index: nova/nova/image/format_inspector.py +=================================================================== +--- nova.orig/nova/image/format_inspector.py ++++ nova/nova/image/format_inspector.py +@@ -368,6 +368,23 @@ class QcowInspector(FileInspector): + not self.has_unknown_features) + + ++class QEDInspector(FileInspector): ++ def __init__(self, tracing=False): ++ super().__init__(tracing) ++ self.new_region('header', CaptureRegion(0, 512)) ++ ++ @property ++ def format_match(self): ++ if not self.region('header').complete: ++ return False ++ return self.region('header').data.startswith(b'QED\x00') ++ ++ def safety_check(self): ++ # QED format is not supported by anyone, but we want to detect it ++ # and mark it as just always unsafe. ++ return False ++ ++ + # The VHD (or VPC as QEMU calls it) format consists of a big-endian + # 512-byte "footer" at the beginning of the file with various + # information, most of which does not matter to us: +@@ -871,19 +888,52 @@ class InfoWrapper(object): + self._source.close() + + ++ALL_FORMATS = { ++ 'raw': FileInspector, ++ 'qcow2': QcowInspector, ++ 'vhd': VHDInspector, ++ 'vhdx': VHDXInspector, ++ 'vmdk': VMDKInspector, ++ 'vdi': VDIInspector, ++ 'qed': QEDInspector, ++} ++ ++ + def get_inspector(format_name): + """Returns a FormatInspector class based on the given name. + + :param format_name: The name of the disk_format (raw, qcow2, etc). + :returns: A FormatInspector or None if unsupported. + """ +- formats = { +- 'raw': FileInspector, +- 'qcow2': QcowInspector, +- 'vhd': VHDInspector, +- 'vhdx': VHDXInspector, +- 'vmdk': VMDKInspector, +- 'vdi': VDIInspector, +- } + +- return formats.get(format_name) ++ return ALL_FORMATS.get(format_name) ++ ++ ++def detect_file_format(filename): ++ """Attempts to detect the format of a file. ++ ++ This runs through a file one time, running all the known inspectors in ++ parallel. It stops reading the file once one of them matches or all of ++ them are sure they don't match. ++ ++ Returns the FileInspector that matched, if any. None if 'raw'. ++ """ ++ inspectors = {k: v() for k, v in ALL_FORMATS.items()} ++ with open(filename, 'rb') as f: ++ for chunk in chunked_reader(f): ++ for format, inspector in list(inspectors.items()): ++ try: ++ inspector.eat_chunk(chunk) ++ except ImageFormatError: ++ # No match, so stop considering this format ++ inspectors.pop(format) ++ continue ++ if (inspector.format_match and inspector.complete and ++ format != 'raw'): ++ # First complete match (other than raw) wins ++ return inspector ++ if all(i.complete for i in inspectors.values()): ++ # If all the inspectors are sure they are not a match, avoid ++ # reading to the end of the file to settle on 'raw'. ++ break ++ return inspectors['raw'] +Index: nova/nova/tests/unit/virt/libvirt/test_driver.py +=================================================================== +--- nova.orig/nova/tests/unit/virt/libvirt/test_driver.py ++++ nova/nova/tests/unit/virt/libvirt/test_driver.py +@@ -14275,10 +14275,11 @@ class LibvirtConnTestCase(test.NoDBTestC + '/fake/instance/dir', disk_info) + self.assertFalse(mock_fetch_image.called) + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch('nova.privsep.path.utime') + @mock.patch('nova.virt.libvirt.utils.create_image') + def test_create_images_and_backing_ephemeral_gets_created( +- self, mock_create_cow_image, mock_utime): ++ self, mock_create_cow_image, mock_utime, mock_detect): + drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), False) + + base_dir = os.path.join(CONF.instances_path, +@@ -16018,11 +16019,13 @@ class LibvirtConnTestCase(test.NoDBTestC + fake_mkfs.assert_has_calls([mock.call('ext4', '/dev/something', + 'myVol')]) + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch('nova.privsep.path.utime') + @mock.patch('nova.virt.libvirt.utils.fetch_image') + @mock.patch('nova.virt.libvirt.utils.create_image') + def test_create_ephemeral_specified_fs_not_valid( +- self, mock_create_cow_image, mock_fetch_image, mock_utime): ++ self, mock_create_cow_image, mock_fetch_image, mock_utime, ++ mock_detect): + CONF.set_override('default_ephemeral_format', 'ext4') + ephemerals = [{'device_type': 'disk', + 'disk_bus': 'virtio', +Index: nova/nova/tests/unit/virt/libvirt/test_imagebackend.py +=================================================================== +--- nova.orig/nova/tests/unit/virt/libvirt/test_imagebackend.py ++++ nova/nova/tests/unit/virt/libvirt/test_imagebackend.py +@@ -524,13 +524,15 @@ class Qcow2TestCase(_ImageTestCase, test + + mock_exists.assert_has_calls(exist_calls) + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(imagebackend.utils, 'synchronized') + @mock.patch('nova.virt.libvirt.utils.create_image') + @mock.patch.object(os.path, 'exists', side_effect=[]) + @mock.patch.object(imagebackend.Image, 'verify_base_size') + @mock.patch('nova.privsep.path.utime') + def test_create_image( +- self, mock_utime, mock_verify, mock_exist, mock_create, mock_sync ++ self, mock_utime, mock_verify, mock_exist, mock_create, mock_sync, ++ mock_detect_format + ): + mock_sync.side_effect = lambda *a, **kw: self._fake_deco + fn = mock.MagicMock() +@@ -551,7 +553,10 @@ class Qcow2TestCase(_ImageTestCase, test + mock_exist.assert_has_calls(exist_calls) + self.assertTrue(mock_sync.called) + mock_utime.assert_called() ++ mock_detect_format.assert_called_once() ++ mock_detect_format.return_value.safety_check.assert_called_once_with() + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(imagebackend.utils, 'synchronized') + @mock.patch('nova.virt.libvirt.utils.create_image') + @mock.patch.object(imagebackend.disk, 'extend') +@@ -559,7 +564,8 @@ class Qcow2TestCase(_ImageTestCase, test + @mock.patch.object(imagebackend.Qcow2, 'get_disk_size') + @mock.patch('nova.privsep.path.utime') + def test_create_image_too_small(self, mock_utime, mock_get, mock_exist, +- mock_extend, mock_create, mock_sync): ++ mock_extend, mock_create, mock_sync, ++ mock_detect_format): + mock_sync.side_effect = lambda *a, **kw: self._fake_deco + mock_get.return_value = self.SIZE + fn = mock.MagicMock() +@@ -576,7 +582,9 @@ class Qcow2TestCase(_ImageTestCase, test + self.assertTrue(mock_sync.called) + self.assertFalse(mock_create.called) + self.assertFalse(mock_extend.called) ++ mock_detect_format.assert_called_once() + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(imagebackend.utils, 'synchronized') + @mock.patch('nova.virt.libvirt.utils.create_image') + @mock.patch('nova.virt.libvirt.utils.get_disk_backing_file') +@@ -588,7 +596,8 @@ class Qcow2TestCase(_ImageTestCase, test + def test_generate_resized_backing_files(self, mock_utime, mock_copy, + mock_verify, mock_exist, + mock_extend, mock_get, +- mock_create, mock_sync): ++ mock_create, mock_sync, ++ mock_detect_format): + mock_sync.side_effect = lambda *a, **kw: self._fake_deco + mock_get.return_value = self.QCOW2_BASE + fn = mock.MagicMock() +@@ -615,7 +624,9 @@ class Qcow2TestCase(_ImageTestCase, test + self.assertTrue(mock_sync.called) + self.assertFalse(mock_create.called) + mock_utime.assert_called() ++ mock_detect_format.assert_called_once() + ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(imagebackend.utils, 'synchronized') + @mock.patch('nova.virt.libvirt.utils.create_image') + @mock.patch('nova.virt.libvirt.utils.get_disk_backing_file') +@@ -626,7 +637,8 @@ class Qcow2TestCase(_ImageTestCase, test + def test_qcow2_exists_and_has_no_backing_file(self, mock_utime, + mock_verify, mock_exist, + mock_extend, mock_get, +- mock_create, mock_sync): ++ mock_create, mock_sync, ++ mock_detect_format): + mock_sync.side_effect = lambda *a, **kw: self._fake_deco + mock_get.return_value = None + fn = mock.MagicMock() +@@ -647,6 +659,31 @@ class Qcow2TestCase(_ImageTestCase, test + self.assertTrue(mock_sync.called) + self.assertFalse(mock_create.called) + self.assertFalse(mock_extend.called) ++ mock_detect_format.assert_called_once() ++ ++ @mock.patch('nova.image.format_inspector.detect_file_format') ++ @mock.patch.object(imagebackend.utils, 'synchronized') ++ @mock.patch('nova.virt.libvirt.utils.create_image') ++ @mock.patch('nova.virt.libvirt.utils.get_disk_backing_file') ++ @mock.patch.object(imagebackend.disk, 'extend') ++ @mock.patch.object(os.path, 'exists', side_effect=[]) ++ @mock.patch.object(imagebackend.Image, 'verify_base_size') ++ def test_qcow2_exists_and_fails_safety_check(self, ++ mock_verify, mock_exist, ++ mock_extend, mock_get, ++ mock_create, mock_sync, ++ mock_detect_format): ++ mock_detect_format.return_value.safety_check.return_value = False ++ mock_sync.side_effect = lambda *a, **kw: self._fake_deco ++ mock_get.return_value = None ++ fn = mock.MagicMock() ++ mock_exist.side_effect = [False, True, False, True, True] ++ image = self.image_class(self.INSTANCE, self.NAME) ++ ++ self.assertRaises(exception.InvalidDiskInfo, ++ image.create_image, fn, self.TEMPLATE_PATH, ++ self.SIZE) ++ mock_verify.assert_not_called() + + def test_resolve_driver_format(self): + image = self.image_class(self.INSTANCE, self.NAME) +Index: nova/nova/tests/unit/virt/libvirt/test_utils.py +=================================================================== +--- nova.orig/nova/tests/unit/virt/libvirt/test_utils.py ++++ nova/nova/tests/unit/virt/libvirt/test_utils.py +@@ -107,16 +107,29 @@ class LibvirtUtilsTestCase(test.NoDBTest + @mock.patch('tempfile.NamedTemporaryFile') + @mock.patch('oslo_concurrency.processutils.execute') + @mock.patch('nova.virt.images.qemu_img_info') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + def _test_create_image( +- self, path, disk_format, disk_size, mock_info, mock_execute, +- mock_ntf, backing_file=None, encryption=None ++ self, path, disk_format, disk_size, mock_detect, mock_info, ++ mock_execute, mock_ntf, backing_file=None, encryption=None, ++ safety_check=True + ): ++ if isinstance(backing_file, dict): ++ backing_info = backing_file ++ backing_file = backing_info.pop('file', None) ++ else: ++ backing_info = {} ++ backing_backing_file = backing_info.pop('backing_file', None) ++ + mock_info.return_value = mock.Mock( + file_format=mock.sentinel.backing_fmt, + cluster_size=mock.sentinel.cluster_size, ++ backing_file=backing_backing_file, ++ format_specific=backing_info, + ) + fh = mock_ntf.return_value.__enter__.return_value + ++ mock_detect.return_value.safety_check.return_value = safety_check ++ + libvirt_utils.create_image( + path, disk_format, disk_size, backing_file=backing_file, + encryption=encryption, +@@ -130,7 +143,7 @@ class LibvirtUtilsTestCase(test.NoDBTest + mock_info.assert_called_once_with(backing_file) + cow_opts = [ + '-o', +- f'backing_file={mock.sentinel.backing_file},' ++ f'backing_file={backing_file},' + f'backing_fmt={mock.sentinel.backing_fmt},' + f'cluster_size={mock.sentinel.cluster_size}', + ] +@@ -166,6 +179,8 @@ class LibvirtUtilsTestCase(test.NoDBTest + expected_args += (disk_size,) + + self.assertEqual([(expected_args,)], mock_execute.call_args_list) ++ if backing_file: ++ mock_detect.return_value.safety_check.assert_called_once_with() + + def test_create_image_raw(self): + self._test_create_image('/some/path', 'raw', '10G') +@@ -181,6 +196,25 @@ class LibvirtUtilsTestCase(test.NoDBTest + backing_file=mock.sentinel.backing_file, + ) + ++ def test_create_image_base_has_backing_file(self): ++ self.assertRaises( ++ exception.InvalidDiskInfo, ++ self._test_create_image, ++ '/some/stuff', 'qcow2', '1234567891234', ++ backing_file={'file': mock.sentinel.backing_file, ++ 'backing_file': mock.sentinel.backing_backing_file}, ++ ) ++ ++ def test_create_image_base_has_data_file(self): ++ self.assertRaises( ++ exception.InvalidDiskInfo, ++ self._test_create_image, ++ '/some/stuff', 'qcow2', '1234567891234', ++ backing_file={'file': mock.sentinel.backing_file, ++ 'backing_file': mock.sentinel.backing_backing_file, ++ 'data': {'data-file': mock.sentinel.data_file}}, ++ ) ++ + def test_create_image_size_none(self): + self._test_create_image( + '/some/stuff', 'qcow2', None, +Index: nova/nova/virt/libvirt/imagebackend.py +=================================================================== +--- nova.orig/nova/virt/libvirt/imagebackend.py ++++ nova/nova/virt/libvirt/imagebackend.py +@@ -34,6 +34,7 @@ from oslo_utils import units + import nova.conf + from nova import exception + from nova.i18n import _ ++from nova.image import format_inspector + from nova.image import glance + import nova.privsep.libvirt + import nova.privsep.path +@@ -660,6 +661,20 @@ class Qcow2(Image): + if not os.path.exists(base): + prepare_template(target=base, *args, **kwargs) + ++ # NOTE(danms): We need to perform safety checks on the base image ++ # before we inspect it for other attributes. We do this each time ++ # because additional safety checks could have been added since we ++ # downloaded the image. ++ if not CONF.workarounds.disable_deep_image_inspection: ++ inspector = format_inspector.detect_file_format(base) ++ if not inspector.safety_check(): ++ LOG.warning('Base image %s failed safety check', base) ++ # NOTE(danms): This is the same exception as would be raised ++ # by qemu_img_info() if the disk format was unreadable or ++ # otherwise unsuitable. ++ raise exception.InvalidDiskInfo( ++ reason=_('Base image failed safety check')) ++ + # NOTE(ankit): Update the mtime of the base file so the image + # cache manager knows it is in use. + _update_utime_ignore_eacces(base) +Index: nova/nova/virt/libvirt/utils.py +=================================================================== +--- nova.orig/nova/virt/libvirt/utils.py ++++ nova/nova/virt/libvirt/utils.py +@@ -35,6 +35,7 @@ import nova.conf + from nova import context as nova_context + from nova import exception + from nova.i18n import _ ++from nova.image import format_inspector + from nova import objects + from nova.objects import fields as obj_fields + import nova.privsep.fs +@@ -135,7 +136,34 @@ def create_image( + ] + + if backing_file: ++ # NOTE(danms): We need to perform safety checks on the base image ++ # before we inspect it for other attributes. We do this each time ++ # because additional safety checks could have been added since we ++ # downloaded the image. ++ if not CONF.workarounds.disable_deep_image_inspection: ++ inspector = format_inspector.detect_file_format(backing_file) ++ if not inspector.safety_check(): ++ LOG.warning('Base image %s failed safety check', backing_file) ++ # NOTE(danms): This is the same exception as would be raised ++ # by qemu_img_info() if the disk format was unreadable or ++ # otherwise unsuitable. ++ raise exception.InvalidDiskInfo( ++ reason=_('Base image failed safety check')) ++ + base_details = images.qemu_img_info(backing_file) ++ if base_details.backing_file is not None: ++ LOG.warning('Base image %s failed safety check', backing_file) ++ raise exception.InvalidDiskInfo( ++ reason=_('Base image failed safety check')) ++ try: ++ data_file = base_details.format_specific['data']['data-file'] ++ except (KeyError, TypeError, AttributeError): ++ data_file = None ++ if data_file is not None: ++ LOG.warning('Base image %s failed safety check', backing_file) ++ raise exception.InvalidDiskInfo( ++ reason=_('Base image failed safety check')) ++ + cow_opts = [ + f'backing_file={backing_file}', + f'backing_fmt={base_details.file_format}' diff -Nru nova-26.1.0/debian/patches/CVE-2024-32498_4_Fix-vmdk_allowed_types-checking_2.patch nova-26.2.2/debian/patches/CVE-2024-32498_4_Fix-vmdk_allowed_types-checking_2.patch --- nova-26.1.0/debian/patches/CVE-2024-32498_4_Fix-vmdk_allowed_types-checking_2.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-32498_4_Fix-vmdk_allowed_types-checking_2.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,85 @@ +From f732f8476851e6272d8ad9937f54b918795844e8 Mon Sep 17 00:00:00 2001 +From: Dan Smith +Date: Mon, 01 Jul 2024 09:06:40 -0700 +Subject: [PATCH] Fix vmdk_allowed_types checking + +This restores the vmdk_allowed_types checking in create_image() +that was unintentionally lost by tightening the +qemu-type-matches-glance code in the fetch patch recently. Since we +are still detecting the format of base images without metadata, we +would have treated a vmdk file that claims to be raw as raw in fetch, +but then read it like a vmdk once it was used as a base image for +something else. + +Change-Id: I07b332a7edb814f6a91661651d9d24bfd6651ae7 +Related-Bug: #2059809 +(cherry picked from commit 08be7b2a0dc1d7728d8034bc2aab0428c4fb642e) +(cherry picked from commit 11301e7e3f0d81a3368632f90608e30d9c647111) +(cherry picked from commit 70a435fd519a0ebcc3ac9ad5254fefbf19c93e48) +--- + +Index: nova/nova/tests/unit/virt/libvirt/test_utils.py +=================================================================== +--- nova.orig/nova/tests/unit/virt/libvirt/test_utils.py ++++ nova/nova/tests/unit/virt/libvirt/test_utils.py +@@ -119,9 +119,11 @@ class LibvirtUtilsTestCase(test.NoDBTest + else: + backing_info = {} + backing_backing_file = backing_info.pop('backing_file', None) ++ backing_fmt = backing_info.pop('backing_fmt', ++ mock.sentinel.backing_fmt) + + mock_info.return_value = mock.Mock( +- file_format=mock.sentinel.backing_fmt, ++ file_format=backing_fmt, + cluster_size=mock.sentinel.cluster_size, + backing_file=backing_backing_file, + format_specific=backing_info, +@@ -144,7 +146,7 @@ class LibvirtUtilsTestCase(test.NoDBTest + cow_opts = [ + '-o', + f'backing_file={backing_file},' +- f'backing_fmt={mock.sentinel.backing_fmt},' ++ f'backing_fmt={backing_fmt},' + f'cluster_size={mock.sentinel.cluster_size}', + ] + +@@ -221,6 +223,25 @@ class LibvirtUtilsTestCase(test.NoDBTest + backing_file=mock.sentinel.backing_file, + ) + ++ def test_create_image_vmdk(self): ++ self._test_create_image( ++ '/some/vmdk', 'vmdk', '1234567891234', ++ backing_file={'file': mock.sentinel.backing_file, ++ 'backing_fmt': 'vmdk', ++ 'backing_file': None, ++ 'data': {'create-type': 'monolithicSparse'}} ++ ) ++ ++ def test_create_image_vmdk_invalid_type(self): ++ self.assertRaises(exception.ImageUnacceptable, ++ self._test_create_image, ++ '/some/vmdk', 'vmdk', '1234567891234', ++ backing_file={'file': mock.sentinel.backing_file, ++ 'backing_fmt': 'vmdk', ++ 'backing_file': None, ++ 'data': {'create-type': 'monolithicFlat'}} ++ ) ++ + def test_create_image_encryption(self): + encryption = { + 'secret': 'a_secret', +Index: nova/nova/virt/libvirt/utils.py +=================================================================== +--- nova.orig/nova/virt/libvirt/utils.py ++++ nova/nova/virt/libvirt/utils.py +@@ -151,6 +151,8 @@ def create_image( + reason=_('Base image failed safety check')) + + base_details = images.qemu_img_info(backing_file) ++ if base_details.file_format == 'vmdk': ++ images.check_vmdk_image('base', base_details) + if base_details.backing_file is not None: + LOG.warning('Base image %s failed safety check', backing_file) + raise exception.InvalidDiskInfo( diff -Nru nova-26.1.0/debian/patches/CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch nova-26.2.2/debian/patches/CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch --- nova-26.1.0/debian/patches/CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,557 @@ +From: Sean Mooney +Date: Thu, 04 Jul 2024 12:38:39 +0100 +Description: CVE-2024-40767: port format inspector tests from glance + This commit is a direct port of the format inspector + unit tests from glance as of commit + 0d8e79b713bc31a78f0f4eac14ee594ca8520999 + . + the only changes to the test are as follows + . + "from glance.common import format_inspector" was updated to + "from nova.image import format_inspector" + . + "from glance.tests import utils as test_utils" + was replaced with "from nova import test" + . + "test_utils.BaseTestCase" was replaced with "test.NoDBTestCase" + . + "glance-unittest-formatinspector-" was replaced with + "nova-unittest-formatinspector-" + . + This makes the test funtional in nova. + . + TestFormatInspectors requries qemu-img to be installed on the + host which would be a new depency for executing unit tests. + to avoid that we skip TestFormatInspectors if qemu-img + is not installed. + TestFormatInspectorInfra and TestFormatInspectorsTargeted + do not have a qemu-img dependency so + no changes to the test assertions were required. +Change-Id: Ia34203f246f0bc574e11476287dfb33fda7954fe +Origin: upstream, https://review.opendev.org/c/openstack/nova/+/923731 +Bug-Debian: https://bugs.debian.org/1076774 +Last-Update: 2024-07-10 + +diff --git a/nova/tests/unit/image/test_format_inspector.py b/nova/tests/unit/image/test_format_inspector.py +new file mode 100644 +index 0000000..4bda796 +--- /dev/null ++++ b/nova/tests/unit/image/test_format_inspector.py +@@ -0,0 +1,517 @@ ++# Copyright 2020 Red Hat, Inc ++# All Rights Reserved. ++# ++# Licensed under the Apache License, Version 2.0 (the "License"); you may ++# not use this file except in compliance with the License. You may obtain ++# a copy of the License at ++# ++# http://www.apache.org/licenses/LICENSE-2.0 ++# ++# Unless required by applicable law or agreed to in writing, software ++# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT ++# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the ++# License for the specific language governing permissions and limitations ++# under the License. ++ ++import io ++import os ++import re ++import struct ++import subprocess ++import tempfile ++from unittest import mock ++ ++from oslo_utils import units ++ ++from nova.image import format_inspector ++from nova import test ++ ++ ++def get_size_from_qemu_img(filename): ++ output = subprocess.check_output('qemu-img info "%s"' % filename, ++ shell=True) ++ for line in output.split(b'\n'): ++ m = re.search(b'^virtual size: .* .([0-9]+) bytes', line.strip()) ++ if m: ++ return int(m.group(1)) ++ ++ raise Exception('Could not find virtual size with qemu-img') ++ ++ ++class TestFormatInspectors(test.NoDBTestCase): ++ def setUp(self): ++ super(TestFormatInspectors, self).setUp() ++ # these tests depend on qemu-img being installed ++ # and in the path, if it is not installed, skip ++ try: ++ subprocess.check_output('qemu-img --version', shell=True) ++ except Exception: ++ self.skipTest('qemu-img not installed') ++ ++ self._created_files = [] ++ ++ def tearDown(self): ++ super(TestFormatInspectors, self).tearDown() ++ for fn in self._created_files: ++ try: ++ os.remove(fn) ++ except Exception: ++ pass ++ ++ def _create_img(self, fmt, size, subformat=None, options=None, ++ backing_file=None): ++ if fmt == 'vhd': ++ # QEMU calls the vhd format vpc ++ fmt = 'vpc' ++ ++ if options is None: ++ options = {} ++ opt = '' ++ prefix = 'nova-unittest-formatinspector-' ++ ++ if subformat: ++ options['subformat'] = subformat ++ prefix += subformat + '-' ++ ++ if options: ++ opt += '-o ' + ','.join('%s=%s' % (k, v) ++ for k, v in options.items()) ++ ++ if backing_file is not None: ++ opt += ' -b %s -F raw' % backing_file ++ ++ fn = tempfile.mktemp(prefix=prefix, ++ suffix='.%s' % fmt) ++ self._created_files.append(fn) ++ subprocess.check_output( ++ 'qemu-img create -f %s %s %s %i' % (fmt, opt, fn, size), ++ shell=True) ++ return fn ++ ++ def _create_allocated_vmdk(self, size_mb, subformat=None): ++ # We need a "big" VMDK file to exercise some parts of the code of the ++ # format_inspector. A way to create one is to first create an empty ++ # file, and then to convert it with the -S 0 option. ++ ++ if subformat is None: ++ # Matches qemu-img default, see `qemu-img convert -O vmdk -o help` ++ subformat = 'monolithicSparse' ++ ++ prefix = 'nova-unittest-formatinspector-%s-' % subformat ++ fn = tempfile.mktemp(prefix=prefix, suffix='.vmdk') ++ self._created_files.append(fn) ++ raw = tempfile.mktemp(prefix=prefix, suffix='.raw') ++ self._created_files.append(raw) ++ ++ # Create a file with pseudo-random data, otherwise it will get ++ # compressed in the streamOptimized format ++ subprocess.check_output( ++ 'dd if=/dev/urandom of=%s bs=1M count=%i' % (raw, size_mb), ++ shell=True) ++ ++ # Convert it to VMDK ++ subprocess.check_output( ++ 'qemu-img convert -f raw -O vmdk -o subformat=%s -S 0 %s %s' % ( ++ subformat, raw, fn), ++ shell=True) ++ return fn ++ ++ def _test_format_at_block_size(self, format_name, img, block_size): ++ fmt = format_inspector.get_inspector(format_name)() ++ self.assertIsNotNone(fmt, ++ 'Did not get format inspector for %s' % ( ++ format_name)) ++ wrapper = format_inspector.InfoWrapper(open(img, 'rb'), fmt) ++ ++ while True: ++ chunk = wrapper.read(block_size) ++ if not chunk: ++ break ++ ++ wrapper.close() ++ return fmt ++ ++ def _test_format_at_image_size(self, format_name, image_size, ++ subformat=None): ++ img = self._create_img(format_name, image_size, subformat=subformat) ++ ++ # Some formats have internal alignment restrictions making this not ++ # always exactly like image_size, so get the real value for comparison ++ virtual_size = get_size_from_qemu_img(img) ++ ++ # Read the format in various sizes, some of which will read whole ++ # sections in a single read, others will be completely unaligned, etc. ++ for block_size in (64 * units.Ki, 512, 17, 1 * units.Mi): ++ fmt = self._test_format_at_block_size(format_name, img, block_size) ++ self.assertTrue(fmt.format_match, ++ 'Failed to match %s at size %i block %i' % ( ++ format_name, image_size, block_size)) ++ self.assertEqual(virtual_size, fmt.virtual_size, ++ ('Failed to calculate size for %s at size %i ' ++ 'block %i') % (format_name, image_size, ++ block_size)) ++ memory = sum(fmt.context_info.values()) ++ self.assertLess(memory, 512 * units.Ki, ++ 'Format used more than 512KiB of memory: %s' % ( ++ fmt.context_info)) ++ ++ def _test_format(self, format_name, subformat=None): ++ # Try a few different image sizes, including some odd and very small ++ # sizes ++ for image_size in (512, 513, 2057, 7): ++ self._test_format_at_image_size(format_name, image_size * units.Mi, ++ subformat=subformat) ++ ++ def test_qcow2(self): ++ self._test_format('qcow2') ++ ++ def test_vhd(self): ++ self._test_format('vhd') ++ ++ def test_vhdx(self): ++ self._test_format('vhdx') ++ ++ def test_vmdk(self): ++ self._test_format('vmdk') ++ ++ def test_vmdk_stream_optimized(self): ++ self._test_format('vmdk', 'streamOptimized') ++ ++ def test_from_file_reads_minimum(self): ++ img = self._create_img('qcow2', 10 * units.Mi) ++ file_size = os.stat(img).st_size ++ fmt = format_inspector.QcowInspector.from_file(img) ++ # We know everything we need from the first 512 bytes of a QCOW image, ++ # so make sure that we did not read the whole thing when we inspect ++ # a local file. ++ self.assertLess(fmt.actual_size, file_size) ++ ++ def test_qed_always_unsafe(self): ++ img = self._create_img('qed', 10 * units.Mi) ++ fmt = format_inspector.get_inspector('qed').from_file(img) ++ self.assertTrue(fmt.format_match) ++ self.assertFalse(fmt.safety_check()) ++ ++ def _test_vmdk_bad_descriptor_offset(self, subformat=None): ++ format_name = 'vmdk' ++ image_size = 10 * units.Mi ++ descriptorOffsetAddr = 0x1c ++ BAD_ADDRESS = 0x400 ++ img = self._create_img(format_name, image_size, subformat=subformat) ++ ++ # Corrupt the header ++ fd = open(img, 'r+b') ++ fd.seek(descriptorOffsetAddr) ++ fd.write(struct.pack(' +Date: Thu, 04 Jul 2024 13:55:41 +0100 +Change-Id: I56d8b9980b4871941ba5de91e60a7df6a40106a8 +Origin: upstream, https://review.opendev.org/c/openstack/nova/+/923732 +Bug-Debian: https://bugs.debian.org/1076774 +Last-Update: 2024-07-10 + +diff --git a/nova/tests/unit/image/test_format_inspector.py b/nova/tests/unit/image/test_format_inspector.py +index 4bda796..9bd99c0 100644 +--- a/nova/tests/unit/image/test_format_inspector.py ++++ b/nova/tests/unit/image/test_format_inspector.py +@@ -27,6 +27,9 @@ + from nova import test + + ++TEST_IMAGE_PREFIX = 'nova-unittest-formatinspector-' ++ ++ + def get_size_from_qemu_img(filename): + output = subprocess.check_output('qemu-img info "%s"' % filename, + shell=True) +@@ -41,13 +44,6 @@ + class TestFormatInspectors(test.NoDBTestCase): + def setUp(self): + super(TestFormatInspectors, self).setUp() +- # these tests depend on qemu-img being installed +- # and in the path, if it is not installed, skip +- try: +- subprocess.check_output('qemu-img --version', shell=True) +- except Exception: +- self.skipTest('qemu-img not installed') +- + self._created_files = [] + + def tearDown(self): +@@ -58,8 +54,55 @@ + except Exception: + pass + ++ def _create_iso(self, image_size, subformat='iso-9660'): ++ # these tests depend on mkisofs ++ # being installed and in the path, ++ # if it is not installed, skip ++ try: ++ subprocess.check_output('mkisofs --version', shell=True) ++ except Exception: ++ self.skipTest('mkisofs not installed') ++ ++ size = image_size // units.Mi ++ base_cmd = "mkisofs" ++ if subformat == 'udf': ++ # depending on the distribution mkisofs may not support udf ++ # and may be provided by genisoimage instead. As a result we ++ # need to check if the command supports udf via help ++ # instead of checking the installed version. ++ # mkisofs --help outputs to stderr so we need to ++ # redirect it to stdout to use grep. ++ try: ++ subprocess.check_output( ++ 'mkisofs --help 2>&1 | grep udf', shell=True) ++ except Exception: ++ self.skipTest('mkisofs does not support udf format') ++ base_cmd += " -udf" ++ prefix = TEST_IMAGE_PREFIX ++ prefix += '-%s-' % subformat ++ fn = tempfile.mktemp(prefix=prefix, suffix='.iso') ++ self._created_files.append(fn) ++ subprocess.check_output( ++ 'dd if=/dev/zero of=%s bs=1M count=%i' % (fn, size), ++ shell=True) ++ subprocess.check_output( ++ '%s -o %s -V "TEST" -J -r %s' % (base_cmd, fn, fn), ++ shell=True) ++ return fn ++ + def _create_img(self, fmt, size, subformat=None, options=None, + backing_file=None): ++ if fmt == 'iso': ++ return self._create_iso(size, subformat) ++ ++ # these tests depend on qemu-img ++ # being installed and in the path, ++ # if it is not installed, skip ++ try: ++ subprocess.check_output('qemu-img --version', shell=True) ++ except Exception: ++ self.skipTest('qemu-img not installed') ++ + if fmt == 'vhd': + # QEMU calls the vhd format vpc + fmt = 'vpc' +@@ -67,7 +110,7 @@ + if options is None: + options = {} + opt = '' +- prefix = 'nova-unittest-formatinspector-' ++ prefix = TEST_IMAGE_PREFIX + + if subformat: + options['subformat'] = subformat +@@ -97,7 +140,8 @@ + # Matches qemu-img default, see `qemu-img convert -O vmdk -o help` + subformat = 'monolithicSparse' + +- prefix = 'nova-unittest-formatinspector-%s-' % subformat ++ prefix = TEST_IMAGE_PREFIX ++ prefix += '-%s-' % subformat + fn = tempfile.mktemp(prefix=prefix, suffix='.vmdk') + self._created_files.append(fn) + raw = tempfile.mktemp(prefix=prefix, suffix='.raw') +@@ -165,6 +209,16 @@ + def test_qcow2(self): + self._test_format('qcow2') + ++ def test_iso_9660(self): ++ # reproduce iso-9660 format regression ++ self.assertRaises( ++ TypeError, self._test_format, 'iso', subformat='iso-9660') ++ ++ def test_udf(self): ++ # reproduce udf format regression ++ self.assertRaises( ++ TypeError, self._test_format, 'iso', subformat='udf') ++ + def test_vhd(self): + self._test_format('vhd') + diff -Nru nova-26.1.0/debian/patches/CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch nova-26.2.2/debian/patches/CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch --- nova-26.1.0/debian/patches/CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,375 @@ +Description: CVE-2024-40767: Add iso file format inspector + This change includes unit tests for the ISO + format inspector using mkisofs to generate + the iso files. + . + A test for stashing qcow content in the system_area + of an iso file is also included. + . + This change modifies format_inspector.detect_file_format + to evaluate all inspectors until they are complete and + raise an InvalidDiskInfo exception if multiple formats + match. +Author: Sean Mooney +Date: Thu, 04 Jul 2024 20:09:31 +0100 +Bug: https://launchpad.net/bugs/2059809 +Change-Id: I7e12718fb3e1f77eb8d1cfcb9fa64e8ddeb9e712 +Origin: upstream, https://review.opendev.org/c/openstack/nova/+/923733 +Bug-Debian: https://bug.debian.org/1076774 +Last-Update: 2024-07-10 + +diff --git a/nova/image/format_inspector.py b/nova/image/format_inspector.py +index 8e57d7e..49cb759 100644 +--- a/nova/image/format_inspector.py ++++ b/nova/image/format_inspector.py +@@ -24,6 +24,7 @@ + import struct + + from oslo_log import log as logging ++from oslo_utils import units + + LOG = logging.getLogger(__name__) + +@@ -843,6 +844,93 @@ + return 'vdi' + + ++class ISOInspector(FileInspector): ++ """ISO 9660 and UDF format ++ ++ we need to check the first 32KB + descriptor size ++ to look for the ISO 9660 or UDF signature. ++ ++ http://wiki.osdev.org/ISO_9660 ++ http://wiki.osdev.org/UDF ++ mkisofs --help | grep udf ++ ++ The Universal Disc Format or UDF is the filesystem used on DVDs and ++ Blu-Ray discs.UDF is an extension of ISO 9660 and shares the same ++ header structure and initial layout. ++ ++ Like the CDFS(ISO 9660) file system, ++ the UDF file system uses a 2048 byte sector size, ++ and it designates that the first 16 sectors can be used by the OS ++ to store proprietary data or boot logic. ++ ++ That means we need to check the first 32KB + descriptor size ++ to look for the ISO 9660 or UDF signature. ++ both formats have an extent based layout, so we can't determine ++ ahead of time where the descriptor will be located. ++ ++ fortunately, the ISO 9660 and UDF formats have a Primary Volume Descriptor ++ located at the beginning of the image, which contains the volume size. ++ ++ """ ++ ++ def __init__(self, *a, **k): ++ super(ISOInspector, self).__init__(*a, **k) ++ self.new_region('system_area', CaptureRegion(0, 32 * units.Ki)) ++ self.new_region('header', CaptureRegion(32 * units.Ki, 2 * units.Ki)) ++ ++ @property ++ def format_match(self): ++ if not self.complete: ++ return False ++ signature = self.region('header').data[1:6] ++ assert len(signature) == 5 ++ return signature in (b'CD001', b'NSR02', b'NSR03') ++ ++ @property ++ def virtual_size(self): ++ if not self.complete: ++ return 0 ++ if not self.format_match: ++ return 0 ++ ++ # the header size is 2KB or 1 sector ++ # the first header field is the descriptor type which is 1 byte ++ # the second field is the standard identifier which is 5 bytes ++ # the third field is the version which is 1 byte ++ # the rest of the header contains type specific data is 2041 bytes ++ # see http://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor ++ ++ # we need to check that the descriptor type is 1 ++ # to ensure that this is a primary volume descriptor ++ descriptor_type = self.region('header').data[0] ++ if descriptor_type != 1: ++ return 0 ++ # The size in bytes of a logical block is stored at offset 128 ++ # and is 2 bytes long encoded in both little and big endian ++ # int16_LSB-MSB so the field is 4 bytes long ++ logical_block_size_data = self.region('header').data[128:132] ++ assert len(logical_block_size_data) == 4 ++ # given the encoding we only need to read half the field so we ++ # can use the first 2 bytes which are the little endian part ++ # this is normally 2048 or 2KB but we need to check as it can be ++ # different according to the ISO 9660 standard. ++ logical_block_size, = struct.unpack(' 1: ++ all_formats = [str(inspector) for inspector in detections] ++ raise ImageFormatError( ++ 'Multiple formats detected: %s' % ', '.join(all_formats)) ++ ++ return inspectors['raw'] if not detections else detections[0] +diff --git a/nova/tests/unit/image/test_format_inspector.py b/nova/tests/unit/image/test_format_inspector.py +index 9bd99c0..4001250 100644 +--- a/nova/tests/unit/image/test_format_inspector.py ++++ b/nova/tests/unit/image/test_format_inspector.py +@@ -54,7 +54,13 @@ + except Exception: + pass + +- def _create_iso(self, image_size, subformat='iso-9660'): ++ def _create_iso(self, image_size, subformat='9660'): ++ """Create an ISO file of the given size. ++ ++ :param image_size: The size of the image to create in bytes ++ :param subformat: The subformat to use, if any ++ """ ++ + # these tests depend on mkisofs + # being installed and in the path, + # if it is not installed, skip +@@ -86,12 +92,22 @@ + 'dd if=/dev/zero of=%s bs=1M count=%i' % (fn, size), + shell=True) + subprocess.check_output( +- '%s -o %s -V "TEST" -J -r %s' % (base_cmd, fn, fn), ++ '%s -V "TEST" -o %s %s' % (base_cmd, fn, fn), + shell=True) + return fn + +- def _create_img(self, fmt, size, subformat=None, options=None, +- backing_file=None): ++ def _create_img( ++ self, fmt, size, subformat=None, options=None, ++ backing_file=None): ++ """Create an image file of the given format and size. ++ ++ :param fmt: The format to create ++ :param size: The size of the image to create in bytes ++ :param subformat: The subformat to use, if any ++ :param options: A dictionary of options to pass to the format ++ :param backing_file: The backing file to use, if any ++ """ ++ + if fmt == 'iso': + return self._create_iso(size, subformat) + +@@ -177,6 +193,13 @@ + + def _test_format_at_image_size(self, format_name, image_size, + subformat=None): ++ """Test the format inspector for the given format at the ++ given image size. ++ ++ :param format_name: The format to test ++ :param image_size: The size of the image to create in bytes ++ :param subformat: The subformat to use, if any ++ """ + img = self._create_img(format_name, image_size, subformat=subformat) + + # Some formats have internal alignment restrictions making this not +@@ -185,7 +208,15 @@ + + # Read the format in various sizes, some of which will read whole + # sections in a single read, others will be completely unaligned, etc. +- for block_size in (64 * units.Ki, 512, 17, 1 * units.Mi): ++ block_sizes = [64 * units.Ki, 1 * units.Mi] ++ # ISO images have a 32KB system area at the beginning of the image ++ # as a result reading that in 17 or 512 byte blocks takes too long, ++ # causing the test to fail. The 64KiB block size is enough to read ++ # the system area and header in a single read. the 1MiB block size ++ # adds very little time to the test so we include it. ++ if format_name != 'iso': ++ block_sizes.extend([17, 512]) ++ for block_size in block_sizes: + fmt = self._test_format_at_block_size(format_name, img, block_size) + self.assertTrue(fmt.format_match, + 'Failed to match %s at size %i block %i' % ( +@@ -210,14 +241,63 @@ + self._test_format('qcow2') + + def test_iso_9660(self): +- # reproduce iso-9660 format regression +- self.assertRaises( +- TypeError, self._test_format, 'iso', subformat='iso-9660') ++ self._test_format('iso', subformat='9660') + +- def test_udf(self): +- # reproduce udf format regression +- self.assertRaises( +- TypeError, self._test_format, 'iso', subformat='udf') ++ def test_iso_udf(self): ++ self._test_format('iso', subformat='udf') ++ ++ def _generate_bad_iso(self): ++ # we want to emulate a malicious user who uploads a an ++ # ISO file has a qcow2 header in the system area ++ # of the ISO file ++ # we will create a qcow2 image and an ISO file ++ # and then copy the qcow2 header to the ISO file ++ # e.g. ++ # mkisofs -o orig.iso /etc/resolv.conf ++ # qemu-img create orig.qcow2 -f qcow2 64M ++ # dd if=orig.qcow2 of=outcome bs=32K count=1 ++ # dd if=orig.iso of=outcome bs=32K skip=1 seek=1 ++ ++ qcow = self._create_img('qcow2', 10 * units.Mi) ++ iso = self._create_iso(64 * units.Mi, subformat='9660') ++ # first ensure the files are valid ++ iso_fmt = self._test_format_at_block_size('iso', iso, 4 * units.Ki) ++ self.assertTrue(iso_fmt.format_match) ++ qcow_fmt = self._test_format_at_block_size('qcow2', qcow, 4 * units.Ki) ++ self.assertTrue(qcow_fmt.format_match) ++ # now copy the qcow2 header to an ISO file ++ prefix = TEST_IMAGE_PREFIX ++ prefix += '-bad-' ++ fn = tempfile.mktemp(prefix=prefix, suffix='.iso') ++ self._created_files.append(fn) ++ subprocess.check_output( ++ 'dd if=%s of=%s bs=32K count=1' % (qcow, fn), ++ shell=True) ++ subprocess.check_output( ++ 'dd if=%s of=%s bs=32K skip=1 seek=1' % (iso, fn), ++ shell=True) ++ return qcow, iso, fn ++ ++ def test_bad_iso_qcow2(self): ++ ++ _, _, fn = self._generate_bad_iso() ++ ++ iso_check = self._test_format_at_block_size('iso', fn, 4 * units.Ki) ++ qcow_check = self._test_format_at_block_size('qcow2', fn, 4 * units.Ki) ++ # this system area of the ISO file is not considered part of the format ++ # the qcow2 header is in the system area of the ISO file ++ # so the ISO file is still valid ++ self.assertTrue(iso_check.format_match) ++ # the qcow2 header is in the system area of the ISO file ++ # but that will be parsed by the qcow2 format inspector ++ # and it will match ++ self.assertTrue(qcow_check.format_match) ++ # if we call format_inspector.detect_file_format it should detect ++ # and raise an exception because both match internally. ++ e = self.assertRaises( ++ format_inspector.ImageFormatError, ++ format_inspector.detect_file_format, fn) ++ self.assertIn('Multiple formats detected', str(e)) + + def test_vhd(self): + self._test_format('vhd') +diff --git a/nova/tests/unit/virt/test_images.py b/nova/tests/unit/virt/test_images.py +index 46c9f9a..cc285dc 100644 +--- a/nova/tests/unit/virt/test_images.py ++++ b/nova/tests/unit/virt/test_images.py +@@ -235,6 +235,34 @@ + images.fetch_to_raw, None, 'foo', 'anypath') + self.assertIn('Invalid VMDK create-type specified', str(e)) + ++ @mock.patch('os.rename') ++ @mock.patch.object(images, 'IMAGE_API') ++ @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch.object(images, 'fetch') ++ @mock.patch('nova.privsep.qemu.unprivileged_qemu_img_info') ++ def test_fetch_iso_is_raw(self, mock_info, mock_fetch, mock_gi, ++ mock_glance, mock_rename): ++ mock_glance.get.return_value = {'disk_format': 'iso'} ++ inspector = mock_gi.return_value.from_file.return_value ++ inspector.safety_check.return_value = True ++ # qemu-img does not have a parser for iso so it is treated as raw ++ info = { ++ "virtual-size": 356352, ++ "filename": "foo.iso", ++ "format": "raw", ++ "actual-size": 356352, ++ "dirty-flag": False ++ } ++ mock_info.return_value = jsonutils.dumps(info) ++ with mock.patch('os.path.exists', return_value=True): ++ images.fetch_to_raw(None, 'foo', 'anypath') ++ # Make sure we called info with -f raw for an iso, since qemu-img does ++ # not support iso ++ mock_info.assert_called_once_with('anypath.part', format='raw') ++ # Make sure that since we considered this to be a raw file, we did the ++ # just-rename-don't-convert path ++ mock_rename.assert_called_once_with('anypath.part', 'anypath') ++ + @mock.patch.object(images, 'IMAGE_API') + @mock.patch('nova.image.format_inspector.get_inspector') + @mock.patch.object(images, 'qemu_img_info') +diff --git a/nova/virt/images.py b/nova/virt/images.py +index 5ec0dc0..813696e 100644 +--- a/nova/virt/images.py ++++ b/nova/virt/images.py +@@ -171,6 +171,11 @@ + raise exception.ImageUnacceptable( + image_id=image_href, + reason=_('Image not in a supported format')) ++ ++ if disk_format == 'iso': ++ # ISO image passed safety check; qemu will treat this as raw from here ++ disk_format = 'raw' ++ + return disk_format + + diff -Nru nova-26.1.0/debian/patches/CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch nova-26.2.2/debian/patches/CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch --- nova-26.1.0/debian/patches/CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,461 @@ +Author: Sean Mooney +Date: Wed, 10 Jul 2024 14:23:33 +0100 +Description: CVE-2024-40767: Change force_format strategy to catch mismatches +From: Dan Smith + When we moved the qemu-img command in fetch_to_raw() to force the + format to what we expect, we lost the ability to identify and react + to situations where qemu-img detected a file as a format that is not + supported by us (i.e. identfied and safety-checked by + format_inspector). In the case of some of the other VMDK variants + that we don't support, we need to be sure to catch any case where + qemu-img thinks it's something other than raw when we think it is, + which will be the case for those formats we don't support. + . + Note this also moves us from explicitly using the format_inspector + that we're told by glance is appropriate, to using our own detection. + We assert that we agree with glance and as above, qemu agrees with + us. This helps us avoid cases where the uploader lies about the + image format, causing us to not run the appropriate safety check. + AMI formats are a liability here since we have a very hard time + asserting what they are and what they will be detected as later in + the pipeline, so there is still special-casing for those. +Bug: https://bugs.launchpad.net/nova/+bug/2071734 +Change-Id: I4b792c5bc959a904854c21565682ed3a687baa1a +Origin: upstream, https://bugs.launchpad.net/nova/+bug/2071734 +Bug-Debian: https://bugs.debian.org/1076774 +Last-Update: 2024-07-10 + +diff --git a/nova/tests/unit/virt/libvirt/test_utils.py b/nova/tests/unit/virt/libvirt/test_utils.py +index bc95e55cd4..6609db6432 100644 +--- a/nova/tests/unit/virt/libvirt/test_utils.py ++++ b/nova/tests/unit/virt/libvirt/test_utils.py +@@ -443,12 +443,12 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + _context, image_id, target, trusted_certs) + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch.object(format_inspector, 'get_inspector') ++ @mock.patch.object(format_inspector, 'detect_file_format') + @mock.patch.object(compute_utils, 'disk_ops_semaphore') + @mock.patch('nova.privsep.utils.supports_direct_io', return_value=True) + @mock.patch('nova.privsep.qemu.unprivileged_convert_image') + def test_fetch_raw_image(self, mock_convert_image, mock_direct_io, +- mock_disk_op_sema, mock_gi, mock_glance): ++ mock_disk_op_sema, mock_detect, mock_glance): + + def fake_rename(old, new): + self.executes.append(('mv', old, new)) +@@ -488,7 +488,7 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + self.stub_out('oslo_utils.fileutils.delete_if_exists', + fake_rm_on_error) + +- mock_inspector = mock_gi.return_value.from_file.return_value ++ mock_inspector = mock_detect.return_value + + # Since the remove param of fileutils.remove_path_on_error() + # is initialized at load time, we must provide a wrapper +@@ -502,6 +502,7 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + + # Make sure qcow2 gets converted to raw + mock_inspector.safety_check.return_value = True ++ mock_inspector.__str__.return_value = 'qcow2' + mock_glance.get.return_value = {'disk_format': 'qcow2'} + target = 't.qcow2' + self.executes = [] +@@ -515,12 +516,13 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + CONF.instances_path, False) + mock_convert_image.reset_mock() + mock_inspector.safety_check.assert_called_once_with() +- mock_gi.assert_called_once_with('qcow2') ++ mock_detect.assert_called_once_with('t.qcow2.part') + + # Make sure raw does not get converted +- mock_gi.reset_mock() ++ mock_detect.reset_mock() + mock_inspector.safety_check.reset_mock() + mock_inspector.safety_check.return_value = True ++ mock_inspector.__str__.return_value = 'raw' + mock_glance.get.return_value = {'disk_format': 'raw'} + target = 't.raw' + self.executes = [] +@@ -529,12 +531,13 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + self.assertEqual(self.executes, expected_commands) + mock_convert_image.assert_not_called() + mock_inspector.safety_check.assert_called_once_with() +- mock_gi.assert_called_once_with('raw') ++ mock_detect.assert_called_once_with('t.raw.part') + + # Make sure safety check failure prevents us from proceeding +- mock_gi.reset_mock() ++ mock_detect.reset_mock() + mock_inspector.safety_check.reset_mock() + mock_inspector.safety_check.return_value = False ++ mock_inspector.__str__.return_value = 'qcow2' + mock_glance.get.return_value = {'disk_format': 'qcow2'} + target = 'backing.qcow2' + self.executes = [] +@@ -544,10 +547,10 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + self.assertEqual(self.executes, expected_commands) + mock_convert_image.assert_not_called() + mock_inspector.safety_check.assert_called_once_with() +- mock_gi.assert_called_once_with('qcow2') ++ mock_detect.assert_called_once_with('backing.qcow2.part') + + # Make sure a format mismatch prevents us from proceeding +- mock_gi.reset_mock() ++ mock_detect.reset_mock() + mock_inspector.safety_check.reset_mock() + mock_inspector.safety_check.side_effect = ( + format_inspector.ImageFormatError) +@@ -560,7 +563,7 @@ class LibvirtUtilsTestCase(test.NoDBTestCase): + self.assertEqual(self.executes, expected_commands) + mock_convert_image.assert_not_called() + mock_inspector.safety_check.assert_called_once_with() +- mock_gi.assert_called_once_with('qcow2') ++ mock_detect.assert_called_once_with('backing.qcow2.part') + + del self.executes + +diff --git a/nova/tests/unit/virt/test_images.py b/nova/tests/unit/virt/test_images.py +index cc285dc4fe..2e6a518cd7 100644 +--- a/nova/tests/unit/virt/test_images.py ++++ b/nova/tests/unit/virt/test_images.py +@@ -21,7 +21,6 @@ from oslo_utils import imageutils + + from nova.compute import utils as compute_utils + from nova import exception +-from nova.image import format_inspector + from nova import test + from nova.virt import images + +@@ -101,15 +100,16 @@ class QemuTestCase(test.NoDBTestCase): + mocked_execute.assert_called_once() + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'convert_image', + side_effect=exception.ImageUnacceptable) + @mock.patch.object(images, 'qemu_img_info') + @mock.patch.object(images, 'fetch') + def test_fetch_to_raw_errors(self, convert_image, qemu_img_info, fetch, +- get_inspector, glance): +- inspector = get_inspector.return_value.from_file.return_value ++ mock_detect, glance): ++ inspector = mock_detect.return_value + inspector.safety_check.return_value = True ++ inspector.__str__.return_value = 'qcow2' + glance.get.return_value = {'disk_format': 'qcow2'} + qemu_img_info.backing_file = None + qemu_img_info.file_format = 'qcow2' +@@ -120,16 +120,17 @@ class QemuTestCase(test.NoDBTestCase): + None, 'href123', '/no/path') + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'convert_image', + side_effect=exception.ImageUnacceptable) + @mock.patch.object(images, 'qemu_img_info') + @mock.patch.object(images, 'fetch') + def test_fetch_to_raw_data_file(self, convert_image, qemu_img_info_fn, +- fetch, mock_gi, mock_glance): ++ fetch, mock_detect, mock_glance): + mock_glance.get.return_value = {'disk_format': 'qcow2'} +- inspector = mock_gi.return_value.from_file.return_value ++ inspector = mock_detect.return_value + inspector.safety_check.return_value = True ++ inspector.__str__.return_value = 'qcow2' + # NOTE(danms): the above test needs the following line as well, as it + # is broken without it. + qemu_img_info = qemu_img_info_fn.return_value +@@ -142,16 +143,17 @@ class QemuTestCase(test.NoDBTestCase): + images.fetch_to_raw, + None, 'href123', '/no/path') + +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'IMAGE_API') + @mock.patch('os.rename') + @mock.patch.object(images, 'qemu_img_info') + @mock.patch.object(images, 'fetch') + def test_fetch_to_raw_from_raw(self, fetch, qemu_img_info_fn, mock_rename, +- mock_glance, mock_gi): ++ mock_glance, mock_detect): + # Make sure we support a case where we fetch an already-raw image and + # qemu-img returns None for "format_specific". + mock_glance.get.return_value = {'disk_format': 'raw'} ++ mock_detect.return_value.__str__.return_value = 'raw' + qemu_img_info = qemu_img_info_fn.return_value + qemu_img_info.file_format = 'raw' + qemu_img_info.backing_file = None +@@ -215,14 +217,15 @@ class QemuTestCase(test.NoDBTestCase): + format='json')) + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'fetch') + @mock.patch('nova.privsep.qemu.unprivileged_qemu_img_info') +- def test_fetch_checks_vmdk_rules(self, mock_info, mock_fetch, mock_gi, ++ def test_fetch_checks_vmdk_rules(self, mock_info, mock_fetch, mock_detect, + mock_glance): + mock_glance.get.return_value = {'disk_format': 'vmdk'} +- inspector = mock_gi.return_value.from_file.return_value ++ inspector = mock_detect.return_value + inspector.safety_check.return_value = True ++ inspector.__str__.return_value = 'vmdk' + info = {'format': 'vmdk', + 'format-specific': { + 'type': 'vmdk', +@@ -238,13 +241,17 @@ class QemuTestCase(test.NoDBTestCase): + @mock.patch('os.rename') + @mock.patch.object(images, 'IMAGE_API') + @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'fetch') + @mock.patch('nova.privsep.qemu.unprivileged_qemu_img_info') +- def test_fetch_iso_is_raw(self, mock_info, mock_fetch, mock_gi, +- mock_glance, mock_rename): ++ def test_fetch_iso_is_raw( ++ self, mock_info, mock_fetch, mock_detect_file_format, mock_gi, ++ mock_glance, mock_rename): + mock_glance.get.return_value = {'disk_format': 'iso'} + inspector = mock_gi.return_value.from_file.return_value + inspector.safety_check.return_value = True ++ inspector.__str__.return_value = 'iso' ++ mock_detect_file_format.return_value = inspector + # qemu-img does not have a parser for iso so it is treated as raw + info = { + "virtual-size": 356352, +@@ -258,27 +265,27 @@ class QemuTestCase(test.NoDBTestCase): + images.fetch_to_raw(None, 'foo', 'anypath') + # Make sure we called info with -f raw for an iso, since qemu-img does + # not support iso +- mock_info.assert_called_once_with('anypath.part', format='raw') ++ mock_info.assert_called_once_with('anypath.part', format=None) + # Make sure that since we considered this to be a raw file, we did the + # just-rename-don't-convert path + mock_rename.assert_called_once_with('anypath.part', 'anypath') + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'qemu_img_info') + @mock.patch.object(images, 'fetch') +- def test_fetch_to_raw_inspector(self, fetch, qemu_img_info, mock_gi, ++ def test_fetch_to_raw_inspector(self, fetch, qemu_img_info, mock_detect, + mock_glance): + # Image claims to be qcow2, is qcow2, but fails safety check, so we + # abort before qemu-img-info + mock_glance.get.return_value = {'disk_format': 'qcow2'} +- inspector = mock_gi.return_value.from_file.return_value ++ inspector = mock_detect.return_value + inspector.safety_check.return_value = False ++ inspector.__str__.return_value = 'qcow2' + self.assertRaises(exception.ImageUnacceptable, + images.fetch_to_raw, None, 'href123', '/no.path') + qemu_img_info.assert_not_called() +- mock_gi.assert_called_once_with('qcow2') +- mock_gi.return_value.from_file.assert_called_once_with('/no.path.part') ++ mock_detect.assert_called_once_with('/no.path.part') + inspector.safety_check.assert_called_once_with() + mock_glance.get.assert_called_once_with(None, 'href123') + +@@ -292,18 +299,17 @@ class QemuTestCase(test.NoDBTestCase): + # Image claims to be qcow2 in glance, but the image is something else, + # so we abort before qemu-img-info + qemu_img_info.reset_mock() +- mock_gi.reset_mock() ++ mock_detect.reset_mock() + inspector.safety_check.reset_mock() +- mock_gi.return_value.from_file.side_effect = ( +- format_inspector.ImageFormatError) ++ mock_detect.return_value.__str__.return_value = 'vmdk' + self.assertRaises(exception.ImageUnacceptable, + images.fetch_to_raw, None, 'href123', '/no.path') +- mock_gi.assert_called_once_with('qcow2') +- inspector.safety_check.assert_not_called() ++ mock_detect.assert_called_once_with('/no.path.part') ++ inspector.safety_check.assert_called_once_with() + qemu_img_info.assert_not_called() + + @mock.patch.object(images, 'IMAGE_API') +- @mock.patch('nova.image.format_inspector.get_inspector') ++ @mock.patch('nova.image.format_inspector.detect_file_format') + @mock.patch.object(images, 'qemu_img_info') + @mock.patch.object(images, 'fetch') + def test_fetch_to_raw_inspector_disabled(self, fetch, qemu_img_info, +@@ -316,36 +322,41 @@ class QemuTestCase(test.NoDBTestCase): + # If deep inspection is disabled, we should never call the inspector + mock_gi.assert_not_called() + # ... and we let qemu-img detect the format itself. +- qemu_img_info.assert_called_once_with('/no.path.part', +- format=None) ++ qemu_img_info.assert_called_once_with('/no.path.part') + mock_glance.get.assert_not_called() + + @mock.patch.object(images, 'IMAGE_API') + @mock.patch.object(images, 'qemu_img_info') +- def test_fetch_inspect_ami(self, imginfo, glance): ++ @mock.patch('nova.image.format_inspector.detect_file_format') ++ def test_fetch_inspect_ami(self, detect, imginfo, glance): + glance.get.return_value = {'disk_format': 'ami'} ++ detect.return_value.__str__.return_value = 'raw' + self.assertRaises(exception.ImageUnacceptable, + images.fetch_to_raw, None, 'href123', '/no.path') + # Make sure 'ami was translated into 'raw' before we call qemu-img +- imginfo.assert_called_once_with('/no.path.part', format='raw') ++ imginfo.assert_called_once_with('/no.path.part') + + @mock.patch.object(images, 'IMAGE_API') + @mock.patch.object(images, 'qemu_img_info') +- def test_fetch_inspect_aki(self, imginfo, glance): ++ @mock.patch('nova.image.format_inspector.detect_file_format') ++ def test_fetch_inspect_aki(self, detect, imginfo, glance): + glance.get.return_value = {'disk_format': 'aki'} ++ detect.return_value.__str__.return_value = 'raw' + self.assertRaises(exception.ImageUnacceptable, + images.fetch_to_raw, None, 'href123', '/no.path') + # Make sure 'aki was translated into 'raw' before we call qemu-img +- imginfo.assert_called_once_with('/no.path.part', format='raw') ++ imginfo.assert_called_once_with('/no.path.part') + + @mock.patch.object(images, 'IMAGE_API') + @mock.patch.object(images, 'qemu_img_info') +- def test_fetch_inspect_ari(self, imginfo, glance): ++ @mock.patch('nova.image.format_inspector.detect_file_format') ++ def test_fetch_inspect_ari(self, detect, imginfo, glance): + glance.get.return_value = {'disk_format': 'ari'} ++ detect.return_value.__str__.return_value = 'raw' + self.assertRaises(exception.ImageUnacceptable, + images.fetch_to_raw, None, 'href123', '/no.path') + # Make sure 'aki was translated into 'raw' before we call qemu-img +- imginfo.assert_called_once_with('/no.path.part', format='raw') ++ imginfo.assert_called_once_with('/no.path.part') + + @mock.patch.object(images, 'IMAGE_API') + @mock.patch.object(images, 'qemu_img_info') +@@ -358,13 +369,16 @@ class QemuTestCase(test.NoDBTestCase): + + @mock.patch.object(images, 'IMAGE_API') + @mock.patch.object(images, 'qemu_img_info') +- @mock.patch('nova.image.format_inspector.get_inspector') +- def test_fetch_inspect_disagrees_qemu(self, mock_gi, imginfo, glance): ++ @mock.patch('nova.image.format_inspector.detect_file_format') ++ def test_fetch_inspect_disagrees_qemu(self, mock_detect, imginfo, glance): + glance.get.return_value = {'disk_format': 'qcow2'} ++ mock_detect.return_value.__str__.return_value = 'qcow2' + # Glance and inspector think it is a qcow2 file, but qemu-img does not +- # agree. It was forced to interpret as a qcow2, but returned no +- # format information as a result. ++ # agree. + imginfo.return_value.data_file = None +- self.assertRaises(exception.ImageUnacceptable, +- images.fetch_to_raw, None, 'href123', '/no.path') +- imginfo.assert_called_once_with('/no.path.part', format='qcow2') ++ imginfo.return_value.file_format = 'vmdk' ++ ex = self.assertRaises(exception.ImageUnacceptable, ++ images.fetch_to_raw, ++ None, 'href123', '/no.path') ++ self.assertIn('content does not match disk_format', str(ex)) ++ imginfo.assert_called_once_with('/no.path.part') +diff --git a/nova/virt/images.py b/nova/virt/images.py +index 813696ed7d..fc9b9192da 100644 +--- a/nova/virt/images.py ++++ b/nova/virt/images.py +@@ -140,42 +140,49 @@ def check_vmdk_image(image_id, data): + + + def do_image_deep_inspection(img, image_href, path): ++ ami_formats = ('ami', 'aki', 'ari') + disk_format = img['disk_format'] + try: + # NOTE(danms): Use our own cautious inspector module to make sure + # the image file passes safety checks. + # See https://bugs.launchpad.net/nova/+bug/2059809 for details. +- inspector_cls = format_inspector.get_inspector(disk_format) +- if not inspector_cls.from_file(path).safety_check(): ++ ++ # Make sure we have a format inspector for the claimed format, else ++ # it is something we do not support and must reject. AMI is excluded. ++ if (disk_format not in ami_formats and ++ not format_inspector.get_inspector(disk_format)): ++ raise exception.ImageUnacceptable( ++ image_id=image_href, ++ reason=_('Image not in a supported format')) ++ ++ inspector = format_inspector.detect_file_format(path) ++ if not inspector.safety_check(): + raise exception.ImageUnacceptable( + image_id=image_href, + reason=(_('Image does not pass safety check'))) ++ ++ # AMI formats can be other things, so don't obsess over this ++ # requirement for them. Otherwise, make sure our detection agrees ++ # with glance. ++ if disk_format not in ami_formats and str(inspector) != disk_format: ++ # If we detected the image as something other than glance claimed, ++ # we abort. ++ raise exception.ImageUnacceptable( ++ image_id=image_href, ++ reason=_('Image content does not match disk_format')) + except format_inspector.ImageFormatError: + # If the inspector we chose based on the image's metadata does not + # think the image is the proper format, we refuse to use it. + raise exception.ImageUnacceptable( + image_id=image_href, + reason=_('Image content does not match disk_format')) +- except AttributeError: +- # No inspector was found +- LOG.warning('Unable to perform deep image inspection on type %r', +- img['disk_format']) +- if disk_format in ('ami', 'aki', 'ari'): +- # A lot of things can be in a UEC, although it is typically a raw +- # filesystem. We really have nothing we can do other than treat it +- # like a 'raw', which is what qemu-img will detect a filesystem as +- # anyway. If someone puts a qcow2 inside, we should fail because +- # we won't do our inspection. +- disk_format = 'raw' +- else: +- raise exception.ImageUnacceptable( +- image_id=image_href, +- reason=_('Image not in a supported format')) +- ++ except Exception: ++ raise exception.ImageUnacceptable( ++ image_id=image_href, ++ reason=_('Image not in a supported format')) + if disk_format == 'iso': + # ISO image passed safety check; qemu will treat this as raw from here + disk_format = 'raw' +- + return disk_format + + +@@ -194,12 +201,22 @@ def fetch_to_raw(context, image_href, path, trusted_certs=None): + + # Only run qemu-img after we have done deep inspection (if enabled). + # If it was not enabled, we will let it detect the format. +- data = qemu_img_info(path_tmp, format=force_format) ++ data = qemu_img_info(path_tmp) + fmt = data.file_format + if fmt is None: + raise exception.ImageUnacceptable( + reason=_("'qemu-img info' parsing failed."), + image_id=image_href) ++ elif force_format is not None and fmt != force_format: ++ # Format inspector and qemu-img must agree on the format, else ++ # we reject. This will catch VMDK some variants that we don't ++ # explicitly support because qemu will identify them as such ++ # and we will not. ++ LOG.warning('Image %s detected by qemu as %s but we expected %s', ++ image_href, fmt, force_format) ++ raise exception.ImageUnacceptable( ++ image_id=image_href, ++ reason=_('Image content does not match disk_format')) + + backing_file = data.backing_file + if backing_file is not None: +-- +2.42.0 + diff -Nru nova-26.1.0/debian/patches/libvirt_Add_encryption_support_to_qemu-img_create_command.patch nova-26.2.2/debian/patches/libvirt_Add_encryption_support_to_qemu-img_create_command.patch --- nova-26.1.0/debian/patches/libvirt_Add_encryption_support_to_qemu-img_create_command.patch 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/debian/patches/libvirt_Add_encryption_support_to_qemu-img_create_command.patch 2024-07-10 21:49:31.000000000 +0000 @@ -0,0 +1,197 @@ +Description: [PATCH] libvirt: Add encryption support to qemu-img create command + This adds handling of options needed for qemu-img create to generate + encrypted images. +Author: Lee Yarwood +Date: Thu, 27 Jan 2022 13:18:37 +0000 +Co-Authored-By: melanie witt +Change-Id: I5d6d2a7b03b5ace0826af80c4004de852579ff12 +Origin: upstream, https://review.opendev.org/c/openstack/nova/+/826752 +Last-Update: 2024-07-02 + +diff --git a/nova/tests/unit/virt/libvirt/test_utils.py b/nova/tests/unit/virt/libvirt/test_utils.py +index 0b80bde..c648108 100644 +--- a/nova/tests/unit/virt/libvirt/test_utils.py ++++ b/nova/tests/unit/virt/libvirt/test_utils.py +@@ -103,19 +103,23 @@ + def test_valid_hostname_bad(self): + self.assertFalse(libvirt_utils.is_valid_hostname("foo/?com=/bin/sh")) + ++ @mock.patch('tempfile.NamedTemporaryFile') + @mock.patch('oslo_concurrency.processutils.execute') + @mock.patch('nova.virt.images.qemu_img_info') + def _test_create_image( + self, path, disk_format, disk_size, mock_info, mock_execute, +- backing_file=None ++ mock_ntf, backing_file=None, encryption=None + ): + mock_info.return_value = mock.Mock( + file_format=mock.sentinel.backing_fmt, + cluster_size=mock.sentinel.cluster_size, + ) ++ fh = mock_ntf.return_value.__enter__.return_value + + libvirt_utils.create_image( +- path, disk_format, disk_size, backing_file=backing_file) ++ path, disk_format, disk_size, backing_file=backing_file, ++ encryption=encryption, ++ ) + + cow_opts = [] + +@@ -130,9 +134,32 @@ + f'cluster_size={mock.sentinel.cluster_size}', + ] + ++ encryption_opts = [] ++ ++ if encryption: ++ encryption_opts = [ ++ '--object', f"secret,id=sec,file={fh.name}", ++ '-o', 'encrypt.key-secret=sec', ++ '-o', f"encrypt.format={encryption.get('format')}", ++ ] ++ ++ encryption_options = { ++ 'cipher-alg': 'aes-256', ++ 'cipher-mode': 'xts', ++ 'hash-alg': 'sha256', ++ 'iter-time': 2000, ++ 'ivgen-alg': 'plain64', ++ 'ivgen-hash-alg': 'sha256', ++ } ++ for option, value in encryption_options.items(): ++ encryption_opts += [ ++ '-o', ++ f'encrypt.{option}={value}', ++ ] ++ + expected_args = ( + 'env', 'LC_ALL=C', 'LANG=C', 'qemu-img', 'create', '-f', +- disk_format, *cow_opts, path, ++ disk_format, *cow_opts, *encryption_opts, path, + ) + if disk_size is not None: + expected_args += (disk_size,) +@@ -159,6 +186,16 @@ + backing_file=mock.sentinel.backing_file, + ) + ++ def test_create_image_encryption(self): ++ encryption = { ++ 'secret': 'a_secret', ++ 'format': 'luks', ++ } ++ self._test_create_image( ++ '/some/stuff', 'qcow2', '1234567891234', ++ encryption=encryption, ++ ) ++ + @ddt.unpack + @ddt.data({'fs_type': 'some_fs_type', + 'default_eph_format': None, +diff --git a/nova/virt/libvirt/utils.py b/nova/virt/libvirt/utils.py +index c673818..10f9380 100644 +--- a/nova/virt/libvirt/utils.py ++++ b/nova/virt/libvirt/utils.py +@@ -22,6 +22,7 @@ + import os + import pwd + import re ++import tempfile + import typing as ty + import uuid + +@@ -114,6 +115,7 @@ + disk_format: str, + disk_size: ty.Optional[ty.Union[str, int]], + backing_file: ty.Optional[str] = None, ++ encryption: ty.Optional[ty.Dict[str, ty.Any]] = None + ) -> None: + """Disk image creation with qemu-img + :param path: Desired location of the disk image +@@ -125,15 +127,16 @@ + If no suffix is given, it will be interpreted as bytes. + Can be None in the case of a COW image. + :param backing_file: (Optional) Backing file to use. ++ :param encryption: (Optional) Dict detailing various encryption attributes ++ such as the format and passphrase. + """ +- base_cmd = [ ++ cmd = [ + 'env', 'LC_ALL=C', 'LANG=C', 'qemu-img', 'create', '-f', disk_format + ] +- cow_opts = [] + + if backing_file: + base_details = images.qemu_img_info(backing_file) +- cow_opts += [ ++ cow_opts = [ + f'backing_file={backing_file}', + f'backing_fmt={base_details.file_format}' + ] +@@ -147,12 +150,60 @@ + + # Format as a comma separated list + csv_opts = ",".join(cow_opts) +- cow_opts = ['-o', csv_opts] ++ cmd += ['-o', csv_opts] + +- cmd = base_cmd + cow_opts + [path] +- if disk_size is not None: +- cmd += [str(disk_size)] +- processutils.execute(*cmd) ++ # Disk size can be None in the case of a COW image ++ disk_size_arg = [str(disk_size)] if disk_size is not None else [] ++ ++ if encryption: ++ with tempfile.NamedTemporaryFile(mode='tr+', encoding='utf-8') as f: ++ # Write out the passphrase secret to a temp file ++ f.write(encryption.get('secret')) ++ ++ # Ensure the secret is written to disk, we can't .close() here as ++ # that removes the file when using NamedTemporaryFile ++ f.flush() ++ ++ # The basic options include the secret and encryption format ++ encryption_opts = [ ++ '--object', f"secret,id=sec,file={f.name}", ++ '-o', 'encrypt.key-secret=sec', ++ '-o', f"encrypt.format={encryption.get('format')}", ++ ] ++ # Supported luks options: ++ # cipher-alg= - Name of cipher algorithm and key length ++ # cipher-mode= - Name of encryption cipher mode ++ # hash-alg= - Name of hash algorithm to use for PBKDF ++ # iter-time= - Time to spend in PBKDF in milliseconds ++ # ivgen-alg= - Name of IV generator algorithm ++ # ivgen-hash-alg= - Name of IV generator hash algorithm ++ # ++ # NOTE(melwitt): Sensible defaults (that match the qemu defaults) ++ # are hardcoded at this time for simplicity and consistency when ++ # instances are migrated. Configuration of luks options could be ++ # added in a future release. ++ encryption_options = { ++ 'cipher-alg': 'aes-256', ++ 'cipher-mode': 'xts', ++ 'hash-alg': 'sha256', ++ 'iter-time': 2000, ++ 'ivgen-alg': 'plain64', ++ 'ivgen-hash-alg': 'sha256', ++ } ++ ++ for option, value in encryption_options.items(): ++ encryption_opts += [ ++ '-o', ++ f'encrypt.{option}={value}', ++ ] ++ ++ # We need to execute the command while the NamedTemporaryFile still ++ # exists ++ cmd += encryption_opts + [path] + disk_size_arg ++ processutils.execute(*cmd) ++ else: ++ cmd += [path] + disk_size_arg ++ processutils.execute(*cmd) + + + def create_ploop_image( diff -Nru nova-26.1.0/debian/patches/series nova-26.2.2/debian/patches/series --- nova-26.1.0/debian/patches/series 2023-05-12 08:15:11.000000000 +0000 +++ nova-26.2.2/debian/patches/series 2024-07-10 21:49:31.000000000 +0000 @@ -1,4 +1,12 @@ Install-missed-files.patch remove-svg-converter-from-doc-conf.py.patch Add-a-healtcheck-url.patch -CVE-2023-2088_Use_force_True_for_os-brick_disconnect_during_delete.patch +libvirt_Add_encryption_support_to_qemu-img_create_command.patch +CVE-2024-32498_1_nova-stable-2023.1_Reject_qcow_files_with_data-file_attributes.patch +CVE-2024-32498_2_nova-stable-2023.1_Check_images_with_format_inspector_for_safety_2.patch +CVE-2024-32498_3_nova-stable-2023.1_Additional-qemu-safety-checking-on-base-images_2.patch +CVE-2024-32498_4_Fix-vmdk_allowed_types-checking_2.patch +CVE-2024-40767_1_port_format_inspector_tests_from_glance_antelope.patch +CVE-2024-40767_2_Reproduce_iso_regression_with_deep_format_inspection_antelope.patch +CVE-2024-40767_3_Add-iso-file-format-inspector_antelope.patch +CVE-2024-40767_4_Change-force_format-strategy-to-catch-mismatches_antelope.patch diff -Nru nova-26.1.0/doc/source/admin/configuration/cross-cell-resize.rst nova-26.2.2/doc/source/admin/configuration/cross-cell-resize.rst --- nova-26.1.0/doc/source/admin/configuration/cross-cell-resize.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/admin/configuration/cross-cell-resize.rst 2024-01-16 10:19:41.000000000 +0000 @@ -284,7 +284,7 @@ Timeouts ~~~~~~~~ -Configure a :ref:`service user ` in case the user token +Configure a :ref:`service user ` in case the user token times out, e.g. during the snapshot and download of a large server image. If RPC calls are timing out with a ``MessagingTimeout`` error in the logs, diff -Nru nova-26.1.0/doc/source/admin/configuration/index.rst nova-26.2.2/doc/source/admin/configuration/index.rst --- nova-26.1.0/doc/source/admin/configuration/index.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/admin/configuration/index.rst 2024-01-16 10:19:41.000000000 +0000 @@ -19,6 +19,7 @@ .. toctree:: :maxdepth: 1 + /admin/configuration/service-user-token /admin/configuration/api /admin/configuration/resize /admin/configuration/cross-cell-resize diff -Nru nova-26.1.0/doc/source/admin/configuration/service-user-token.rst nova-26.2.2/doc/source/admin/configuration/service-user-token.rst --- nova-26.1.0/doc/source/admin/configuration/service-user-token.rst 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/doc/source/admin/configuration/service-user-token.rst 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,59 @@ +.. _service_user_token: + +=================== +Service User Tokens +=================== + +.. note:: + + Configuration of service user tokens is **required** for every Nova service + for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for + details. + +Configure Nova to send service user tokens alongside regular user tokens when +making REST API calls to other services. The identity service (Keystone) will +authenticate a request using the service user token if the regular user token +has expired. + +This is important when long-running operations such as live migration or +snapshot take long enough to exceed the expiry of the user token. Without the +service token, if a long-running operation exceeds the expiry of the user +token, post operations such as cleanup after a live migration could fail when +Nova calls other service APIs like block-storage (Cinder) or networking +(Neutron). + +The service token is also used by services to validate whether the API caller +is a service. Some service APIs are restricted to service users only. + +To set up service tokens, create a ``nova`` service user and ``service`` role +in the identity service (Keystone) and assign the ``service`` role to the +``nova`` service user. + +Then, configure the :oslo.config:group:`service_user` section of the Nova +configuration file, for example: + +.. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://104.130.216.102/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = secretservice + ... + +And configure the other identity options as necessary for the service user, +much like you would configure nova to work with the image service (Glance) or +networking service (Neutron). + +.. note:: + + Please note that the role assigned to the :oslo.config:group:`service_user` + needs to be in the configured + :oslo.config:option:`keystone_authtoken.service_token_roles` of other + services such as block-storage (Cinder), image (Glance), and networking + (Neutron). diff -Nru nova-26.1.0/doc/source/admin/live-migration-usage.rst nova-26.2.2/doc/source/admin/live-migration-usage.rst --- nova-26.1.0/doc/source/admin/live-migration-usage.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/admin/live-migration-usage.rst 2024-01-16 10:19:41.000000000 +0000 @@ -320,4 +320,4 @@ If live migrations routinely timeout or fail during cleanup operations due to the user token timing out, consider configuring nova to use -:ref:`service user tokens `. +:ref:`service user tokens `. diff -Nru nova-26.1.0/doc/source/admin/migrate-instance-with-snapshot.rst nova-26.2.2/doc/source/admin/migrate-instance-with-snapshot.rst --- nova-26.1.0/doc/source/admin/migrate-instance-with-snapshot.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/admin/migrate-instance-with-snapshot.rst 2024-01-16 10:19:41.000000000 +0000 @@ -67,7 +67,7 @@ If snapshot operations routinely fail because the user token times out while uploading a large disk image, consider configuring nova to use - :ref:`service user tokens `. + :ref:`service user tokens `. #. Use the :command:`openstack image list` command to check the status until the status is ``ACTIVE``: diff -Nru nova-26.1.0/doc/source/admin/support-compute.rst nova-26.2.2/doc/source/admin/support-compute.rst --- nova-26.1.0/doc/source/admin/support-compute.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/admin/support-compute.rst 2024-01-16 10:19:41.000000000 +0000 @@ -478,67 +478,3 @@ at ``/v2.1`` instead of ``/v2``. The former route supports microversions, while the latter route is considered the legacy v2.0 compatibility-mode route which renders all requests as if they were made on the legacy v2.0 API. - - -.. _user_token_timeout: - -User token times out during long-running operations ---------------------------------------------------- - -Problem -~~~~~~~ - -Long-running operations such as live migration or snapshot can sometimes -overrun the expiry of the user token. In such cases, post operations such -as cleaning up after a live migration can fail when the nova-compute service -needs to cleanup resources in other services, such as in the block-storage -(cinder) or networking (neutron) services. - -For example: - -.. code-block:: console - - 2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec - 2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled - 2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client - 2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance - 2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401) - -Solution -~~~~~~~~ - -Configure nova to use service user tokens to supplement the regular user token -used to initiate the operation. The identity service (keystone) will then -authenticate a request using the service user token if the user token has -already expired. - -To use, create a service user in the identity service similar as you would when -creating the ``nova`` service user. - -Then configure the :oslo.config:group:`service_user` section of the nova -configuration file, for example: - -.. code-block:: ini - - [service_user] - send_service_user_token = True - auth_type = password - project_domain_name = Default - project_name = service - user_domain_name = Default - password = secretservice - username = nova - auth_url = https://104.130.216.102/identity - ... - -And configure the other identity options as necessary for the service user, -much like you would configure nova to work with the image service (glance) -or networking service. - -.. note:: - - Please note that the role of the :oslo.config:group:`service_user` you - configure needs to be a superset of - :oslo.config:option:`keystone_authtoken.service_token_roles` (The option - :oslo.config:option:`keystone_authtoken.service_token_roles` is configured - in cinder, glance and neutron). diff -Nru nova-26.1.0/doc/source/install/compute-install-obs.rst nova-26.2.2/doc/source/install/compute-install-obs.rst --- nova-26.1.0/doc/source/install/compute-install-obs.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/compute-install-obs.rst 2024-01-16 10:19:41.000000000 +0000 @@ -92,6 +92,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: .. path /etc/nova/nova.conf diff -Nru nova-26.1.0/doc/source/install/compute-install-rdo.rst nova-26.2.2/doc/source/install/compute-install-rdo.rst --- nova-26.1.0/doc/source/install/compute-install-rdo.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/compute-install-rdo.rst 2024-01-16 10:19:41.000000000 +0000 @@ -84,6 +84,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: .. path /etc/nova/nova.conf diff -Nru nova-26.1.0/doc/source/install/compute-install-ubuntu.rst nova-26.2.2/doc/source/install/compute-install-ubuntu.rst --- nova-26.1.0/doc/source/install/compute-install-ubuntu.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/compute-install-ubuntu.rst 2024-01-16 10:19:41.000000000 +0000 @@ -74,6 +74,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option: .. path /etc/nova/nova.conf diff -Nru nova-26.1.0/doc/source/install/controller-install-obs.rst nova-26.2.2/doc/source/install/controller-install-obs.rst --- nova-26.1.0/doc/source/install/controller-install-obs.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/controller-install-obs.rst 2024-01-16 10:19:41.000000000 +0000 @@ -260,6 +260,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the management interface IP address of the controller node: diff -Nru nova-26.1.0/doc/source/install/controller-install-rdo.rst nova-26.2.2/doc/source/install/controller-install-rdo.rst --- nova-26.1.0/doc/source/install/controller-install-rdo.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/controller-install-rdo.rst 2024-01-16 10:19:41.000000000 +0000 @@ -247,6 +247,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the management interface IP address of the controller node: diff -Nru nova-26.1.0/doc/source/install/controller-install-ubuntu.rst nova-26.2.2/doc/source/install/controller-install-ubuntu.rst --- nova-26.1.0/doc/source/install/controller-install-ubuntu.rst 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/doc/source/install/controller-install-ubuntu.rst 2024-01-16 10:19:41.000000000 +0000 @@ -237,6 +237,26 @@ Comment out or remove any other options in the ``[keystone_authtoken]`` section. + * In the ``[service_user]`` section, configure :ref:`service user + tokens `: + + .. path /etc/nova/nova.conf + .. code-block:: ini + + [service_user] + send_service_user_token = true + auth_url = https://controller/identity + auth_strategy = keystone + auth_type = password + project_domain_name = Default + project_name = service + user_domain_name = Default + username = nova + password = NOVA_PASS + + Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in + the Identity service. + * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the management interface IP address of the controller node: diff -Nru nova-26.1.0/nova/api/openstack/compute/flavor_access.py nova-26.2.2/nova/api/openstack/compute/flavor_access.py --- nova-26.1.0/nova/api/openstack/compute/flavor_access.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/api/openstack/compute/flavor_access.py 2024-01-16 10:19:41.000000000 +0000 @@ -93,7 +93,14 @@ vals = body['removeTenantAccess'] tenant = vals['tenant'] - identity.verify_project_id(context, tenant) + # It doesn't really matter if project exists or not: we can delete + # it from flavor's access list in both cases. + try: + identity.verify_project_id(context, tenant) + except webob.exc.HTTPBadRequest as identity_exc: + msg = "Project ID %s is not a valid project." % tenant + if msg not in identity_exc.explanation: + raise # NOTE(gibi): We have to load a flavor from the db here as # flavor.remove_access() will try to emit a notification and that needs diff -Nru nova-26.1.0/nova/api/openstack/compute/remote_consoles.py nova-26.2.2/nova/api/openstack/compute/remote_consoles.py --- nova-26.1.0/nova/api/openstack/compute/remote_consoles.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/api/openstack/compute/remote_consoles.py 2024-01-16 10:19:41.000000000 +0000 @@ -56,6 +56,9 @@ raise webob.exc.HTTPNotFound(explanation=e.format_message()) except exception.InstanceNotReady as e: raise webob.exc.HTTPConflict(explanation=e.format_message()) + except exception.InstanceInvalidState as e: + common.raise_http_conflict_for_instance_invalid_state( + e, 'get_vnc_console', id) except NotImplementedError: common.raise_feature_not_supported() diff -Nru nova-26.1.0/nova/api/openstack/identity.py nova-26.2.2/nova/api/openstack/identity.py --- nova-26.1.0/nova/api/openstack/identity.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/api/openstack/identity.py 2024-01-16 10:19:41.000000000 +0000 @@ -27,24 +27,27 @@ """verify that a project_id exists. This attempts to verify that a project id exists. If it does not, - an HTTPBadRequest is emitted. + an HTTPBadRequest is emitted. Also HTTPBadRequest is emitted + if Keystone identity service version 3.0 is not found. """ adap = utils.get_ksa_adapter( 'identity', ksa_auth=context.get_auth_plugin(), min_version=(3, 0), max_version=(3, 'latest')) - failure = webob.exc.HTTPBadRequest( - explanation=_("Project ID %s is not a valid project.") % - project_id) try: resp = adap.get('/projects/%s' % project_id) except kse.EndpointNotFound: LOG.error( - "Keystone identity service version 3.0 was not found. This might " - "be because your endpoint points to the v2.0 versioned endpoint " - "which is not supported. Please fix this.") - raise failure + "Keystone identity service version 3.0 was not found. This " + "might be caused by Nova misconfiguration or Keystone " + "problems.") + msg = _("Nova was unable to find Keystone service endpoint.") + # TODO(astupnik). It may be reasonable to switch to HTTP 503 + # (HTTP Service Unavailable) instead of HTTP Bad Request here. + # If proper Keystone servie is inaccessible, then technially + # this is a server side error and not an error in Nova. + raise webob.exc.HTTPBadRequest(explanation=msg) except kse.ClientException: # something is wrong, like there isn't a keystone v3 endpoint, # or nova isn't configured for the interface to talk to it; @@ -57,7 +60,8 @@ return True elif resp.status_code == 404: # we got access, and we know this project is not there - raise failure + msg = _("Project ID %s is not a valid project.") % project_id + raise webob.exc.HTTPBadRequest(explanation=msg) elif resp.status_code == 403: # we don't have enough permission to verify this, so default # to "it's ok". diff -Nru nova-26.1.0/nova/api/openstack/wsgi.py nova-26.2.2/nova/api/openstack/wsgi.py --- nova-26.1.0/nova/api/openstack/wsgi.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/api/openstack/wsgi.py 2024-01-16 10:19:41.000000000 +0000 @@ -538,12 +538,6 @@ with ResourceExceptionHandler(): action_result = self.dispatch(meth, request, action_args) except Fault as ex: - LOG.debug(f'Request method failure captured:\n' - f' request: {request}\n' - f' method: {meth}\n' - f' exception: {ex}\n' - f' action_args: {action_args}\n', - exc_info=1) response = ex if not response: diff -Nru nova-26.1.0/nova/cmd/status.py nova-26.2.2/nova/cmd/status.py --- nova-26.1.0/nova/cmd/status.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/cmd/status.py 2024-01-16 10:19:41.000000000 +0000 @@ -271,6 +271,15 @@ return upgradecheck.Result(upgradecheck.Code.SUCCESS) + def _check_service_user_token(self): + if not CONF.service_user.send_service_user_token: + msg = (_(""" +Service user token configuration is required for all Nova services. +For more details see the following: +https://docs.openstack.org/latest/nova/admin/configuration/service-user-token.html""")) # noqa + return upgradecheck.Result(upgradecheck.Code.FAILURE, msg) + return upgradecheck.Result(upgradecheck.Code.SUCCESS) + # The format of the check functions is to return an upgradecheck.Result # object with the appropriate upgradecheck.Code and details set. If the # check hits warnings or failures then those should be stored in the @@ -294,6 +303,8 @@ (_('Older than N-1 computes'), _check_old_computes), # Added in Wallaby (_('hw_machine_type unset'), _check_machine_type_set), + # Added in Bobcat + (_('Service User Token Configuration'), _check_service_user_token), ) diff -Nru nova-26.1.0/nova/compute/api.py nova-26.2.2/nova/compute/api.py --- nova-26.1.0/nova/compute/api.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/api.py 2024-01-16 10:19:41.000000000 +0000 @@ -4697,6 +4697,7 @@ allow_bfv_rescue=False): """Rescue the given instance.""" + image_meta = None if rescue_image_ref: try: image_meta = image_meta_obj.ImageMeta.from_image_ref( @@ -4717,6 +4718,8 @@ "image properties set") raise exception.UnsupportedRescueImage( image=rescue_image_ref) + else: + image_meta = instance.image_meta bdms = objects.BlockDeviceMappingList.get_by_instance_uuid( context, instance.uuid) @@ -4725,6 +4728,9 @@ volume_backed = compute_utils.is_volume_backed_instance( context, instance, bdms) + allow_bfv_rescue &= 'hw_rescue_bus' in image_meta.properties and \ + 'hw_rescue_device' in image_meta.properties + if volume_backed and allow_bfv_rescue: cn = objects.ComputeNode.get_by_host_and_nodename( context, instance.host, instance.node) diff -Nru nova-26.1.0/nova/compute/build_results.py nova-26.2.2/nova/compute/build_results.py --- nova-26.1.0/nova/compute/build_results.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/build_results.py 2024-01-16 10:19:41.000000000 +0000 @@ -24,3 +24,11 @@ ACTIVE = 'active' # Instance is running FAILED = 'failed' # Instance failed to build and was not rescheduled RESCHEDULED = 'rescheduled' # Instance failed to build, but was rescheduled +# Instance failed by policy violation (such as affinity or anti-affinity) +# and was not rescheduled. In this case, the node's failed count won't be +# increased. +FAILED_BY_POLICY = 'failed_by_policy' +# Instance failed by policy violation (such as affinity or anti-affinity) +# but was rescheduled. In this case, the node's failed count won't be +# increased. +RESCHEDULED_BY_POLICY = 'rescheduled_by_policy' diff -Nru nova-26.1.0/nova/compute/manager.py nova-26.2.2/nova/compute/manager.py --- nova-26.1.0/nova/compute/manager.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/manager.py 2024-01-16 10:19:41.000000000 +0000 @@ -1803,11 +1803,8 @@ else: max_server = 1 if len(members_on_host) >= max_server: - msg = _("Anti-affinity instance group policy " - "was violated.") - raise exception.RescheduledException( - instance_uuid=instance.uuid, - reason=msg) + raise exception.GroupAffinityViolation( + instance_uuid=instance.uuid, policy='Anti-affinity') # NOTE(ganso): The check for affinity below does not work and it # can easily be violated because the lock happens in different @@ -1817,10 +1814,8 @@ elif group.policy and 'affinity' == group.policy: group_hosts = group.get_hosts(exclude=[instance.uuid]) if group_hosts and self.host not in group_hosts: - msg = _("Affinity instance group policy was violated.") - raise exception.RescheduledException( - instance_uuid=instance.uuid, - reason=msg) + raise exception.GroupAffinityViolation( + instance_uuid=instance.uuid, policy='Affinity') _do_validation(context, instance, group) @@ -2260,6 +2255,9 @@ self.reportclient.delete_allocation_for_instance( context, instance.uuid, force=True) + if result in (build_results.FAILED_BY_POLICY, + build_results.RESCHEDULED_BY_POLICY): + return if result in (build_results.FAILED, build_results.RESCHEDULED): self._build_failed(node) @@ -2358,6 +2356,8 @@ self._nil_out_instance_obj_host_and_node(instance) self._set_instance_obj_error_state(instance, clean_task_state=True) + if isinstance(e, exception.RescheduledByPolicyException): + return build_results.FAILED_BY_POLICY return build_results.FAILED LOG.debug(e.format_message(), instance=instance) # This will be used for logging the exception @@ -2384,6 +2384,10 @@ injected_files, requested_networks, security_groups, block_device_mapping, request_spec=request_spec, host_lists=[host_list]) + + if isinstance(e, exception.RescheduledByPolicyException): + return build_results.RESCHEDULED_BY_POLICY + return build_results.RESCHEDULED except (exception.InstanceNotFound, exception.UnexpectedDeletingTaskStateError): @@ -2601,6 +2605,17 @@ bdms=block_device_mapping) raise exception.BuildAbortException(instance_uuid=instance.uuid, reason=e.format_message()) + except exception.GroupAffinityViolation as e: + LOG.exception('Failed to build and run instance', + instance=instance) + self._notify_about_instance_usage(context, instance, + 'create.error', fault=e) + compute_utils.notify_about_instance_create( + context, instance, self.host, + phase=fields.NotificationPhase.ERROR, exception=e, + bdms=block_device_mapping) + raise exception.RescheduledByPolicyException( + instance_uuid=instance.uuid, reason=str(e)) except Exception as e: LOG.exception('Failed to build and run instance', instance=instance) @@ -2736,7 +2751,8 @@ block_device_mapping) resources['block_device_info'] = block_device_info except (exception.InstanceNotFound, - exception.UnexpectedDeletingTaskStateError): + exception.UnexpectedDeletingTaskStateError, + exception.ComputeResourcesUnavailable): with excutils.save_and_reraise_exception(): self._build_resources_cleanup(instance, network_info) except (exception.UnexpectedTaskStateError, @@ -3621,7 +3637,7 @@ bdms, recreate, on_shared_storage, preserve_ephemeral, migration, scheduled_node, limits, request_spec, accel_uuids, - reimage_boot_volume): + reimage_boot_volume=None): """Destroy and re-make this instance. A 'rebuild' effectively purges all existing data from the system and @@ -3655,7 +3671,7 @@ :param accel_uuids: a list of cyborg ARQ uuids :param reimage_boot_volume: Boolean to specify whether the user has explicitly requested to rebuild a boot - volume + volume or None if RPC version is <=6.0 """ # recreate=True means the instance is being evacuated from a failed @@ -6793,9 +6809,9 @@ instance.power_state = current_power_state # NOTE(mriedem): The vm_state has to be set before updating the - # resource tracker, see vm_states.ALLOW_RESOURCE_REMOVAL. The host/node - # values cannot be nulled out until after updating the resource tracker - # though. + # resource tracker, see vm_states.allow_resource_removal(). The + # host/node values cannot be nulled out until after updating the + # resource tracker though. instance.vm_state = vm_states.SHELVED_OFFLOADED instance.task_state = None instance.save(expected_task_state=[task_states.SHELVING, diff -Nru nova-26.1.0/nova/compute/resource_tracker.py nova-26.2.2/nova/compute/resource_tracker.py --- nova-26.1.0/nova/compute/resource_tracker.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/resource_tracker.py 2024-01-16 10:19:41.000000000 +0000 @@ -1546,7 +1546,8 @@ # NOTE(sfinucan): Both brand new instances as well as instances that # are being unshelved will have is_new_instance == True is_removed_instance = not is_new_instance and (is_removed or - instance['vm_state'] in vm_states.ALLOW_RESOURCE_REMOVAL) + vm_states.allow_resource_removal( + vm_state=instance['vm_state'], task_state=instance.task_state)) if is_new_instance: self.tracked_instances.add(uuid) @@ -1605,7 +1606,9 @@ instance_by_uuid = {} for instance in instances: - if instance.vm_state not in vm_states.ALLOW_RESOURCE_REMOVAL: + if not vm_states.allow_resource_removal( + vm_state=instance['vm_state'], + task_state=instance.task_state): self._update_usage_from_instance(context, instance, nodename) instance_by_uuid[instance.uuid] = instance return instance_by_uuid diff -Nru nova-26.1.0/nova/compute/stats.py nova-26.2.2/nova/compute/stats.py --- nova-26.1.0/nova/compute/stats.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/stats.py 2024-01-16 10:19:41.000000000 +0000 @@ -105,7 +105,8 @@ (vm_state, task_state, os_type, project_id) = \ self._extract_state_from_instance(instance) - if is_removed or vm_state in vm_states.ALLOW_RESOURCE_REMOVAL: + if is_removed or vm_states.allow_resource_removal( + vm_state=vm_state, task_state=task_state): self._decrement("num_instances") self.states.pop(uuid) else: diff -Nru nova-26.1.0/nova/compute/vm_states.py nova-26.2.2/nova/compute/vm_states.py --- nova-26.1.0/nova/compute/vm_states.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/compute/vm_states.py 2024-01-16 10:19:41.000000000 +0000 @@ -27,6 +27,7 @@ See http://wiki.openstack.org/VMState """ +from nova.compute import task_states from nova.objects import fields @@ -74,5 +75,11 @@ # states we allow to trigger crash dump ALLOW_TRIGGER_CRASH_DUMP = [ACTIVE, PAUSED, RESCUED, RESIZED, ERROR] -# states we allow resources to be freed in -ALLOW_RESOURCE_REMOVAL = [DELETED, SHELVED_OFFLOADED] + +def allow_resource_removal(vm_state, task_state=None): + """(vm_state, task_state) combinations we allow resources to be freed in""" + + return ( + vm_state == DELETED or + vm_state == SHELVED_OFFLOADED and task_state != task_states.SPAWNING + ) diff -Nru nova-26.1.0/nova/conf/workarounds.py nova-26.2.2/nova/conf/workarounds.py --- nova-26.1.0/nova/conf/workarounds.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/conf/workarounds.py 2024-01-16 10:19:41.000000000 +0000 @@ -410,6 +410,13 @@ 4.4.0, libvirt will do the correct thing with respect to checking CPU compatibility on the destination host during live migration. """), + cfg.BoolOpt('skip_cpu_compare_at_startup', + default=False, + help=""" +This will skip the CPU comparison call at the startup of Compute +service and lets libvirt handle it. +"""), + cfg.BoolOpt( 'skip_hypervisor_version_check_on_lm', default=False, diff -Nru nova-26.1.0/nova/exception.py nova-26.2.2/nova/exception.py --- nova-26.1.0/nova/exception.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/exception.py 2024-01-16 10:19:41.000000000 +0000 @@ -1487,6 +1487,15 @@ "%(reason)s") +class RescheduledByPolicyException(RescheduledException): + msg_fmt = _("Build of instance %(instance_uuid)s was re-scheduled: " + "%(reason)s") + + +class GroupAffinityViolation(NovaException): + msg_fmt = _("%(policy)s instance group policy was violated") + + class InstanceFaultRollback(NovaException): def __init__(self, inner_exception=None): message = _("Instance rollback performed due to: %s") diff -Nru nova-26.1.0/nova/network/neutron.py nova-26.2.2/nova/network/neutron.py --- nova-26.1.0/nova/network/neutron.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/network/neutron.py 2024-01-16 10:19:41.000000000 +0000 @@ -222,13 +222,15 @@ # support some services (metadata API) where an admin context is used # without an auth token. global _ADMIN_AUTH + user_auth = None if admin or (context.is_admin and not context.auth_token): if not _ADMIN_AUTH: _ADMIN_AUTH = _load_auth_plugin(CONF) - return _ADMIN_AUTH + user_auth = _ADMIN_AUTH - if context.auth_token: - return service_auth.get_auth_plugin(context) + if context.auth_token or user_auth: + # When user_auth = None, user_auth will be extracted from the context. + return service_auth.get_auth_plugin(context, user_auth=user_auth) # We did not get a user token and we should not be using # an admin token so log an error @@ -1586,6 +1588,13 @@ 'pf_mac_address': pf_mac, 'vf_num': vf_num, }) + + # Update port binding capabilities using PCI device's network + # capabilities if they exist. + pci_net_caps = pci_dev.network_caps + if pci_net_caps: + vf_profile.update({'capabilities': pci_net_caps}) + return vf_profile def _get_pci_device_profile(self, pci_dev): @@ -3894,7 +3903,7 @@ 'Failed to get segment IDs for network %s' % network_id) from e # The segment field of an unconfigured subnet could be None return [subnet['segment_id'] for subnet in subnets - if subnet['segment_id'] is not None] + if subnet.get('segment_id') is not None] def get_segment_id_for_subnet( self, diff -Nru nova-26.1.0/nova/objects/flavor.py nova-26.2.2/nova/objects/flavor.py --- nova-26.1.0/nova/objects/flavor.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/objects/flavor.py 2024-01-16 10:19:41.000000000 +0000 @@ -270,8 +270,9 @@ return flavor @staticmethod - @api_db_api.context_manager.reader def _flavor_get_query_from_db(context): + # We don't use a database context decorator on this method because this + # method is not executing a query, it's only building one. query = context.session.query(api_models.Flavors).options( orm.joinedload(api_models.Flavors.extra_specs) ) @@ -285,6 +286,7 @@ @staticmethod @db_utils.require_context + @api_db_api.context_manager.reader def _flavor_get_from_db(context, id): """Returns a dict describing specific flavor.""" result = Flavor._flavor_get_query_from_db(context).\ @@ -296,6 +298,7 @@ @staticmethod @db_utils.require_context + @api_db_api.context_manager.reader def _flavor_get_by_name_from_db(context, name): """Returns a dict describing specific flavor.""" result = Flavor._flavor_get_query_from_db(context).\ @@ -307,6 +310,7 @@ @staticmethod @db_utils.require_context + @api_db_api.context_manager.reader def _flavor_get_by_flavor_id_from_db(context, flavor_id): """Returns a dict describing specific flavor_id.""" result = Flavor._flavor_get_query_from_db(context).\ diff -Nru nova-26.1.0/nova/objects/instance.py nova-26.2.2/nova/objects/instance.py --- nova-26.1.0/nova/objects/instance.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/objects/instance.py 2024-01-16 10:19:41.000000000 +0000 @@ -1090,6 +1090,11 @@ def obj_load_attr(self, attrname): # NOTE(danms): We can't lazy-load anything without a context and a uuid if not self._context: + if 'uuid' in self: + LOG.debug( + "Lazy-load of '%s' attempted by orphaned instance", + attrname, instance=self + ) raise exception.OrphanedObjectError(method='obj_load_attr', objtype=self.obj_name()) if 'uuid' not in self: diff -Nru nova-26.1.0/nova/objects/pci_device.py nova-26.2.2/nova/objects/pci_device.py --- nova-26.1.0/nova/objects/pci_device.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/objects/pci_device.py 2024-01-16 10:19:41.000000000 +0000 @@ -588,6 +588,13 @@ """ return self.extra_info.get('mac_address') + @property + def network_caps(self): + """PCI device network capabilities or empty list if not available""" + caps_json = self.extra_info.get('capabilities', '{}') + caps = jsonutils.loads(caps_json) + return caps.get('network', []) + @base.NovaObjectRegistry.register class PciDeviceList(base.ObjectListBase, base.NovaObject): diff -Nru nova-26.1.0/nova/service_auth.py nova-26.2.2/nova/service_auth.py --- nova-26.1.0/nova/service_auth.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/service_auth.py 2024-01-16 10:19:41.000000000 +0000 @@ -30,8 +30,10 @@ _SERVICE_AUTH = None -def get_auth_plugin(context): - user_auth = context.get_auth_plugin() +def get_auth_plugin(context, user_auth=None): + # user_auth may be passed in when the RequestContext is anonymous, such as + # when get_admin_context() is used for API calls by nova-manage. + user_auth = user_auth or context.get_auth_plugin() if CONF.service_user.send_service_user_token: global _SERVICE_AUTH diff -Nru nova-26.1.0/nova/tests/fixtures/libvirt_data.py nova-26.2.2/nova/tests/fixtures/libvirt_data.py --- nova-26.1.0/nova/tests/fixtures/libvirt_data.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/fixtures/libvirt_data.py 2024-01-16 10:19:41.000000000 +0000 @@ -2182,6 +2182,7 @@ + """, # noqa:E501 diff -Nru nova-26.1.0/nova/tests/fixtures/nova.py nova-26.2.2/nova/tests/fixtures/nova.py --- nova-26.1.0/nova/tests/fixtures/nova.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/fixtures/nova.py 2024-01-16 10:19:41.000000000 +0000 @@ -1316,6 +1316,77 @@ nova.privsep.sys_admin_pctxt, 'client_mode', False)) +class CGroupsFixture(fixtures.Fixture): + """Mocks checks made for available subsystems on the host's control group. + + The fixture mocks all calls made on the host to verify the capabilities + provided by its kernel. Through this, one can simulate the underlying + system hosts work on top of and have tests react to expected outcomes from + such. + + Use sample: + >>> cgroups = self.useFixture(CGroupsFixture()) + >>> cgroups = self.useFixture(CGroupsFixture(version=2)) + >>> cgroups = self.useFixture(CGroupsFixture()) + ... cgroups.version = 2 + + :attr version: Arranges mocks to simulate the host interact with nova + following the given version of cgroups. + Available values are: + - 0: All checks related to cgroups will return False. + - 1: Checks related to cgroups v1 will return True. + - 2: Checks related to cgroups v2 will return True. + Defaults to 1. + """ + + def __init__(self, version=1): + self._cpuv1 = None + self._cpuv2 = None + + self._version = version + + @property + def version(self): + return self._version + + @version.setter + def version(self, value): + self._version = value + self._update_mocks() + + def setUp(self): + super().setUp() + self._cpuv1 = self.useFixture(fixtures.MockPatch( + 'nova.virt.libvirt.host.Host._has_cgroupsv1_cpu_controller')).mock + self._cpuv2 = self.useFixture(fixtures.MockPatch( + 'nova.virt.libvirt.host.Host._has_cgroupsv2_cpu_controller')).mock + self._update_mocks() + + def _update_mocks(self): + if not self._cpuv1: + return + + if not self._cpuv2: + return + + if self.version == 0: + self._cpuv1.return_value = False + self._cpuv2.return_value = False + return + + if self.version == 1: + self._cpuv1.return_value = True + self._cpuv2.return_value = False + return + + if self.version == 2: + self._cpuv1.return_value = False + self._cpuv2.return_value = True + return + + raise ValueError(f"Unknown cgroups version: '{self.version}'.") + + class NoopQuotaDriverFixture(fixtures.Fixture): """A fixture to run tests using the NoopQuotaDriver. diff -Nru nova-26.1.0/nova/tests/functional/api_sample_tests/test_remote_consoles.py nova-26.2.2/nova/tests/functional/api_sample_tests/test_remote_consoles.py --- nova-26.1.0/nova/tests/functional/api_sample_tests/test_remote_consoles.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/api_sample_tests/test_remote_consoles.py 2024-01-16 10:19:41.000000000 +0000 @@ -13,6 +13,10 @@ # License for the specific language governing permissions and limitations # under the License. +from unittest import mock + +from nova.compute import api as compute +from nova import exception from nova.tests.functional.api_sample_tests import test_servers HTTP_RE = r'(https?://)([\w\d:#@%/;$()~_?\+-=\\.&](#!)?)*' @@ -38,6 +42,22 @@ self._verify_response('get-vnc-console-post-resp', {'url': HTTP_RE}, response, 200) + @mock.patch.object(compute.API, 'get_vnc_console') + def test_get_vnc_console_instance_invalid_state(self, + mock_get_vnc_console): + uuid = self._post_server() + + def fake_get_vnc_console(*args, **kwargs): + raise exception.InstanceInvalidState( + attr='fake_attr', state='fake_state', method='fake_method', + instance_uuid=uuid) + + mock_get_vnc_console.side_effect = fake_get_vnc_console + response = self._do_post('servers/%s/action' % uuid, + 'get-vnc-console-post-req', + {'action': 'os-getVNCConsole'}) + self.assertEqual(409, response.status_code) + def test_get_spice_console(self): uuid = self._post_server() response = self._do_post('servers/%s/action' % uuid, diff -Nru nova-26.1.0/nova/tests/functional/libvirt/base.py nova-26.2.2/nova/tests/functional/libvirt/base.py --- nova-26.1.0/nova/tests/functional/libvirt/base.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/libvirt/base.py 2024-01-16 10:19:41.000000000 +0000 @@ -42,6 +42,7 @@ super(ServersTestBase, self).setUp() self.useFixture(nova_fixtures.LibvirtImageBackendFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) self.libvirt = self.useFixture(nova_fixtures.LibvirtFixture()) self.useFixture(nova_fixtures.OSBrickFixture()) diff -Nru nova-26.1.0/nova/tests/functional/libvirt/test_evacuate.py nova-26.2.2/nova/tests/functional/libvirt/test_evacuate.py --- nova-26.1.0/nova/tests/functional/libvirt/test_evacuate.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/libvirt/test_evacuate.py 2024-01-16 10:19:41.000000000 +0000 @@ -427,6 +427,7 @@ self.useFixture(nova_fixtures.NeutronFixture(self)) self.useFixture(nova_fixtures.GlanceFixture(self)) self.useFixture(func_fixtures.PlacementFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) fake_network.set_stub_network_methods(self) api_fixture = self.useFixture( diff -Nru nova-26.1.0/nova/tests/functional/libvirt/test_vpmem.py nova-26.2.2/nova/tests/functional/libvirt/test_vpmem.py --- nova-26.1.0/nova/tests/functional/libvirt/test_vpmem.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/libvirt/test_vpmem.py 2024-01-16 10:19:41.000000000 +0000 @@ -75,6 +75,7 @@ 'nova.privsep.libvirt.get_pmem_namespaces', return_value=self.fake_pmem_namespaces)) self.useFixture(nova_fixtures.LibvirtImageBackendFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) self.useFixture(fixtures.MockPatch( 'nova.virt.libvirt.LibvirtDriver._get_local_gb_info', return_value={'total': 128, diff -Nru nova-26.1.0/nova/tests/functional/regressions/test_bug_1595962.py nova-26.2.2/nova/tests/functional/regressions/test_bug_1595962.py --- nova-26.1.0/nova/tests/functional/regressions/test_bug_1595962.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/regressions/test_bug_1595962.py 2024-01-16 10:19:41.000000000 +0000 @@ -47,6 +47,7 @@ 'nova.virt.libvirt.guest.libvirt', fakelibvirt)) self.useFixture(nova_fixtures.LibvirtFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) self.admin_api = api_fixture.admin_api self.api = api_fixture.api diff -Nru nova-26.1.0/nova/tests/functional/regressions/test_bug_1951656.py nova-26.2.2/nova/tests/functional/regressions/test_bug_1951656.py --- nova-26.1.0/nova/tests/functional/regressions/test_bug_1951656.py 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/regressions/test_bug_1951656.py 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,73 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from oslo_utils import uuidutils + + +from nova.tests.fixtures import libvirt as fakelibvirt +from nova.tests.functional.libvirt import test_vgpu +from nova.virt.libvirt import utils as libvirt_utils + + +class VGPUTestsLibvirt7_7(test_vgpu.VGPUTestBase): + + def _create_mdev(self, physical_device, mdev_type, uuid=None): + # We need to fake the newly created sysfs object by adding a new + # FakeMdevDevice in the existing persisted Connection object so + # when asking to get the existing mdevs, we would see it. + if not uuid: + uuid = uuidutils.generate_uuid() + mdev_name = libvirt_utils.mdev_uuid2name(uuid) + libvirt_parent = self.pci2libvirt_address(physical_device) + + # Libvirt 7.7 now creates mdevs with a parent_addr suffix. + new_mdev_name = '_'.join([mdev_name, libvirt_parent]) + + # Here, we get the right compute thanks by the self.current_host that + # was modified just before + connection = self.computes[ + self._current_host].driver._host.get_connection() + connection.mdev_info.devices.update( + {mdev_name: fakelibvirt.FakeMdevDevice(dev_name=new_mdev_name, + type_id=mdev_type, + parent=libvirt_parent)}) + return uuid + + def setUp(self): + super(VGPUTestsLibvirt7_7, self).setUp() + extra_spec = {"resources:VGPU": "1"} + self.flavor = self._create_flavor(extra_spec=extra_spec) + + # Start compute1 supporting only nvidia-11 + self.flags( + enabled_mdev_types=fakelibvirt.NVIDIA_11_VGPU_TYPE, + group='devices') + + self.compute1 = self.start_compute_with_vgpu('host1') + + def test_create_servers_with_vgpu(self): + + # Create a single instance against a specific compute node. + self._create_server( + image_uuid='155d900f-4e14-4e4c-a73d-069cbf4541e6', + flavor_id=self.flavor, host=self.compute1.host, + networks='auto', expected_state='ACTIVE') + + self.assert_mdev_usage(self.compute1, expected_amount=1) + + self._create_server( + image_uuid='155d900f-4e14-4e4c-a73d-069cbf4541e6', + flavor_id=self.flavor, host=self.compute1.host, + networks='auto', expected_state='ACTIVE') + + self.assert_mdev_usage(self.compute1, expected_amount=2) diff -Nru nova-26.1.0/nova/tests/functional/regressions/test_bug_1995153.py nova-26.2.2/nova/tests/functional/regressions/test_bug_1995153.py --- nova-26.1.0/nova/tests/functional/regressions/test_bug_1995153.py 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/regressions/test_bug_1995153.py 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,107 @@ +# Copyright (C) 2023 Red Hat, Inc +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + + +import fixtures +from unittest import mock + +from oslo_serialization import jsonutils +from oslo_utils import units + +from nova.objects import fields +from nova.tests.fixtures import libvirt as fakelibvirt +from nova.tests.functional import integrated_helpers +from nova.tests.functional.libvirt import base + + +class Bug1995153RegressionTest( + base.ServersTestBase, + integrated_helpers.InstanceHelperMixin +): + + ADDITIONAL_FILTERS = ['NUMATopologyFilter', 'PciPassthroughFilter'] + + ALIAS_NAME = 'a1' + PCI_DEVICE_SPEC = [jsonutils.dumps( + { + 'vendor_id': fakelibvirt.PCI_VEND_ID, + 'product_id': fakelibvirt.PCI_PROD_ID, + } + )] + # we set the numa_affinity policy to required to ensure strict affinity + # between pci devices and the guest cpu and memory will be enforced. + PCI_ALIAS = [jsonutils.dumps( + { + 'vendor_id': fakelibvirt.PCI_VEND_ID, + 'product_id': fakelibvirt.PCI_PROD_ID, + 'name': ALIAS_NAME, + 'device_type': fields.PciDeviceType.STANDARD, + 'numa_policy': fields.PCINUMAAffinityPolicy.REQUIRED, + } + )] + + def setUp(self): + super(Bug1995153RegressionTest, self).setUp() + self.flags( + device_spec=self.PCI_DEVICE_SPEC, + alias=self.PCI_ALIAS, + group='pci' + ) + host_manager = self.scheduler.manager.host_manager + pci_filter_class = host_manager.filter_cls_map['PciPassthroughFilter'] + host_pass_mock = mock.Mock(wraps=pci_filter_class().host_passes) + self.mock_filter = self.useFixture(fixtures.MockPatch( + 'nova.scheduler.filters.pci_passthrough_filter' + '.PciPassthroughFilter.host_passes', + side_effect=host_pass_mock)).mock + + def test_socket_policy_bug_1995153(self): + """Previously, the numa_usage_from_instance_numa() method in + hardware.py saved the host NUMAToplogy object with NUMACells that have + no `socket` set. This was an omission in the original implementation of + the `socket` PCI NUMA affinity policy. The consequence was that any + code path that called into numa_usage_from_instance_numa() would + clobber the host NUMA topology in the database with a socket-less + version. Booting an instance with NUMA toplogy would do that, for + example. If then a second instance was booted with the `socket` PCI + NUMA affinity policy, it would read the socket-less host NUMATopology + from the database, and error out with a NotImplementedError. This was + bug 1995153. Demonstrate that this is fixed. + """ + host_info = fakelibvirt.HostInfo( + cpu_nodes=2, cpu_sockets=1, cpu_cores=2, cpu_threads=2, + kB_mem=(16 * units.Gi) // units.Ki) + self.flags(cpu_dedicated_set='0-3', group='compute') + pci_info = fakelibvirt.HostPCIDevicesInfo(num_pci=1, numa_node=1) + + self.start_compute(host_info=host_info, pci_info=pci_info) + + extra_spec = { + 'hw:cpu_policy': 'dedicated', + 'pci_passthrough:alias': '%s:1' % self.ALIAS_NAME, + 'hw:pci_numa_affinity_policy': 'socket' + } + # Boot a first instance with a guest NUMA topology to run the + # numa_usage_from_instance_numa() and update the host NUMATopology in + # the database. + self._create_server( + flavor_id=self._create_flavor( + extra_spec={'hw:cpu_policy': 'dedicated'})) + + # Boot an instance with the `socket` PCI NUMA affinity policy and + # assert that it boots correctly now. + flavor_id = self._create_flavor(extra_spec=extra_spec) + self._create_server(flavor_id=flavor_id) + self.assertTrue(self.mock_filter.called) diff -Nru nova-26.1.0/nova/tests/functional/regressions/test_bug_2025480.py nova-26.2.2/nova/tests/functional/regressions/test_bug_2025480.py --- nova-26.1.0/nova/tests/functional/regressions/test_bug_2025480.py 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/regressions/test_bug_2025480.py 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,86 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +from unittest import mock + +from nova import context +from nova.objects import compute_node +from nova import test +from nova.tests import fixtures as nova_fixtures +from nova.tests.functional import fixtures as func_fixtures +from nova.tests.functional import integrated_helpers + + +class UnshelveUpdateAvailableResourcesPeriodicRace( + test.TestCase, integrated_helpers.InstanceHelperMixin): + def setUp(self): + super(UnshelveUpdateAvailableResourcesPeriodicRace, self).setUp() + + placement = func_fixtures.PlacementFixture() + self.useFixture(placement) + self.placement = placement.api + self.neutron = nova_fixtures.NeutronFixture(self) + self.useFixture(self.neutron) + self.useFixture(nova_fixtures.GlanceFixture(self)) + # Start nova services. + self.api = self.useFixture(nova_fixtures.OSAPIFixture( + api_version='v2.1')).admin_api + self.api.microversion = 'latest' + self.notifier = self.useFixture( + nova_fixtures.NotificationFixture(self)) + + self.start_service('conductor') + self.start_service('scheduler') + + def test_unshelve_spawning_update_available_resources(self): + compute = self._start_compute('compute1') + + server = self._create_server( + networks=[{'port': self.neutron.port_1['id']}]) + + node = compute_node.ComputeNode.get_by_nodename( + context.get_admin_context(), 'compute1') + self.assertEqual(1, node.vcpus_used) + + # with default config shelve means immediate offload as well + req = { + 'shelve': {} + } + self.api.post_server_action(server['id'], req) + self._wait_for_server_parameter( + server, {'status': 'SHELVED_OFFLOADED', + 'OS-EXT-SRV-ATTR:host': None}) + + node = compute_node.ComputeNode.get_by_nodename( + context.get_admin_context(), 'compute1') + self.assertEqual(0, node.vcpus_used) + + def fake_spawn(*args, **kwargs): + self._run_periodics() + + with mock.patch.object( + compute.driver, 'spawn', side_effect=fake_spawn): + req = {'unshelve': None} + self.api.post_server_action(server['id'], req) + self.notifier.wait_for_versioned_notifications( + 'instance.unshelve.start') + self._wait_for_server_parameter( + server, + { + 'status': 'ACTIVE', + 'OS-EXT-STS:task_state': None, + 'OS-EXT-SRV-ATTR:host': 'compute1', + }) + + node = compute_node.ComputeNode.get_by_nodename( + context.get_admin_context(), 'compute1') + # After the fix, the instance should have resources claimed + self.assertEqual(1, node.vcpus_used) diff -Nru nova-26.1.0/nova/tests/functional/regressions/test_bug_2040264.py nova-26.2.2/nova/tests/functional/regressions/test_bug_2040264.py --- nova-26.1.0/nova/tests/functional/regressions/test_bug_2040264.py 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/regressions/test_bug_2040264.py 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,56 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from nova.tests import fixtures as nova_fixtures +from nova.tests.functional import integrated_helpers + + +class ComputeVersion6xPinnedRpcTests(integrated_helpers._IntegratedTestBase): + + compute_driver = 'fake.MediumFakeDriver' + ADMIN_API = True + api_major_version = 'v2.1' + microversion = 'latest' + + def setUp(self): + super(ComputeVersion6xPinnedRpcTests, self).setUp() + self.useFixture(nova_fixtures.CastAsCallFixture(self)) + + self.compute1 = self._start_compute(host='host1') + + def _test_rebuild_instance_with_compute_rpc_pin(self, version_cap): + # Since passing the latest microversion (>= 2.93) passes + # the 'reimage_boot_volume' parameter as True and it is + # not acceptable with compute RPC version (required 6.1) + # These tests fail, so assigning microversion to 2.92 + self.api.microversion = '2.92' + self.flags(compute=version_cap, group='upgrade_levels') + + server_req = self._build_server(networks='none') + server = self.api.post_server({'server': server_req}) + server = self._wait_for_state_change(server, 'ACTIVE') + + self.api.post_server_action(server['id'], {'rebuild': { + 'imageRef': '155d900f-4e14-4e4c-a73d-069cbf4541e6' + }}) + + # We automatically pin to 6.0 if old computes are Yoga or older. + def test_rebuild_instance_6_0(self): + self._test_rebuild_instance_with_compute_rpc_pin('6.0') + + # We automatically pin to 6.1 if old computes are Zed. + def test_rebuild_instance_6_1(self): + self._test_rebuild_instance_with_compute_rpc_pin('6.1') + + # We automatically pin to 6.2 if old computes are 2023.1. + def test_rebuild_instance_6_2(self): + self._test_rebuild_instance_with_compute_rpc_pin('6.2') diff -Nru nova-26.1.0/nova/tests/functional/test_server_group.py nova-26.2.2/nova/tests/functional/test_server_group.py --- nova-26.1.0/nova/tests/functional/test_server_group.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/test_server_group.py 2024-01-16 10:19:41.000000000 +0000 @@ -20,6 +20,7 @@ from nova.compute import instance_actions from nova import context from nova.db.main import api as db +from nova import objects from nova import test from nova.tests import fixtures as nova_fixtures from nova.tests.functional.api import client @@ -495,6 +496,85 @@ self.assertIn('Invalid input', ex.response.text) self.assertIn('soft-affinity', ex.response.text) + @mock.patch('nova.scheduler.filters.affinity_filter.' + 'ServerGroupAffinityFilter.host_passes', return_value=True) + def test_failed_count_with_affinity_violation(self, mock_host_passes): + """Check failed count not incremented after violation of the late + affinity check. https://bugs.launchpad.net/nova/+bug/1996732 + """ + + created_group = self.api.post_server_groups(self.affinity) + flavor = self.api.get_flavors()[2] + + # Ensure the first instance is on compute1 + with utils.temporary_mutation(self.admin_api, microversion='2.53'): + compute2_service_id = self.admin_api.get_services( + host=self.compute2.host, binary='nova-compute')[0]['id'] + self.admin_api.put_service(compute2_service_id, + {'status': 'disabled'}) + + self._boot_a_server_to_group(created_group, flavor=flavor) + + # Ensure the second instance is on compute2 + with utils.temporary_mutation(self.admin_api, microversion='2.53'): + self.admin_api.put_service(compute2_service_id, + {'status': 'enabled'}) + compute1_service_id = self.admin_api.get_services( + host=self.compute.host, binary='nova-compute')[0]['id'] + self.admin_api.put_service(compute1_service_id, + {'status': 'disabled'}) + + # Expects GroupAffinityViolation exception + failed_server = self._boot_a_server_to_group(created_group, + flavor=flavor, + expected_status='ERROR') + + self.assertEqual('Exceeded maximum number of retries. Exhausted all ' + 'hosts available for retrying build failures for ' + 'instance %s.' % failed_server['id'], + failed_server['fault']['message']) + + ctxt = context.get_admin_context() + computes = objects.ComputeNodeList.get_all(ctxt) + + for node in computes: + self.assertEqual(node.stats.get('failed_builds'), '0') + + @mock.patch('nova.scheduler.filters.affinity_filter.' + 'ServerGroupAntiAffinityFilter.host_passes', return_value=True) + def test_failed_count_with_anti_affinity_violation(self, mock_host_passes): + """Check failed count after violation of the late affinity check. + https://bugs.launchpad.net/nova/+bug/1996732 + """ + + created_group = self.api.post_server_groups(self.anti_affinity) + flavor = self.api.get_flavors()[2] + + # Ensure two instances are scheduled on the same host + with utils.temporary_mutation(self.admin_api, microversion='2.53'): + compute2_service_id = self.admin_api.get_services( + host=self.compute2.host, binary='nova-compute')[0]['id'] + self.admin_api.put_service(compute2_service_id, + {'status': 'disabled'}) + + self._boot_a_server_to_group(created_group, flavor=flavor) + + # Expects GroupAffinityViolation exception + failed_server = self._boot_a_server_to_group(created_group, + flavor=flavor, + expected_status='ERROR') + + self.assertEqual('Exceeded maximum number of retries. Exhausted all ' + 'hosts available for retrying build failures for ' + 'instance %s.' % failed_server['id'], + failed_server['fault']['message']) + + ctxt = context.get_admin_context() + computes = objects.ComputeNodeList.get_all(ctxt) + + for node in computes: + self.assertEqual(node.stats.get('failed_builds'), '0') + class ServerGroupAffinityConfTest(ServerGroupTestBase): api_major_version = 'v2.1' diff -Nru nova-26.1.0/nova/tests/functional/test_server_rescue.py nova-26.2.2/nova/tests/functional/test_server_rescue.py --- nova-26.1.0/nova/tests/functional/test_server_rescue.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/functional/test_server_rescue.py 2024-01-16 10:19:41.000000000 +0000 @@ -10,6 +10,10 @@ # License for the specific language governing permissions and limitations # under the License. +import datetime + +from oslo_utils.fixture import uuidsentinel as uuids + from nova.tests import fixtures as nova_fixtures from nova.tests.functional.api import client from nova.tests.functional import integrated_helpers @@ -23,7 +27,37 @@ self.useFixture(nova_fixtures.CinderFixture(self)) self._start_compute(host='host1') - def _create_bfv_server(self): + def _create_image(self, metadata=None): + image = { + 'id': uuids.stable_rescue_image, + 'name': 'fake-image-rescue-property', + 'created_at': datetime.datetime(2011, 1, 1, 1, 2, 3), + 'updated_at': datetime.datetime(2011, 1, 1, 1, 2, 3), + 'deleted_at': None, + 'deleted': False, + 'status': 'active', + 'is_public': False, + 'container_format': 'raw', + 'disk_format': 'raw', + 'size': '25165824', + 'min_ram': 0, + 'min_disk': 0, + 'protected': False, + 'visibility': 'public', + 'tags': ['tag1', 'tag2'], + 'properties': { + 'kernel_id': 'nokernel', + 'ramdisk_id': 'nokernel', + 'hw_rescue_device': 'disk', + 'hw_rescue_bus': 'scsi', + }, + } + if metadata: + image['properties'].update(metadata) + return self.glance.create(None, image) + + def _create_bfv_server(self, metadata=None): + image = self._create_image(metadata=metadata) server_request = self._build_server(networks=[]) server_request.pop('imageRef') server_request['block_device_mapping_v2'] = [{ @@ -33,7 +67,7 @@ 'destination_type': 'volume'}] server = self.api.post_server({'server': server_request}) self._wait_for_state_change(server, 'ACTIVE') - return server + return server, image class DisallowBFVRescuev286(BFVRescue): @@ -43,10 +77,10 @@ microversion = '2.86' def test_bfv_rescue_not_supported(self): - server = self._create_bfv_server() + server, image = self._create_bfv_server() ex = self.assertRaises(client.OpenStackApiException, self.api.post_server_action, server['id'], {'rescue': { - 'rescue_image_ref': '155d900f-4e14-4e4c-a73d-069cbf4541e6'}}) + 'rescue_image_ref': image['id']}}) self.assertEqual(400, ex.response.status_code) self.assertIn('Cannot rescue a volume-backed instance', ex.response.text) @@ -60,10 +94,10 @@ microversion = '2.86' def test_bfv_rescue_not_supported(self): - server = self._create_bfv_server() + server, image = self._create_bfv_server() ex = self.assertRaises(client.OpenStackApiException, self.api.post_server_action, server['id'], {'rescue': { - 'rescue_image_ref': '155d900f-4e14-4e4c-a73d-069cbf4541e6'}}) + 'rescue_image_ref': image['id']}}) self.assertEqual(400, ex.response.status_code) self.assertIn('Cannot rescue a volume-backed instance', ex.response.text) @@ -77,10 +111,10 @@ microversion = '2.87' def test_bfv_rescue_not_supported(self): - server = self._create_bfv_server() + server, image = self._create_bfv_server() ex = self.assertRaises(client.OpenStackApiException, self.api.post_server_action, server['id'], {'rescue': { - 'rescue_image_ref': '155d900f-4e14-4e4c-a73d-069cbf4541e6'}}) + 'rescue_image_ref': image['id']}}) self.assertEqual(400, ex.response.status_code) self.assertIn('Host unable to rescue a volume-backed instance', ex.response.text) @@ -94,7 +128,41 @@ microversion = '2.87' def test_bfv_rescue_supported(self): - server = self._create_bfv_server() + server, image = self._create_bfv_server() self.api.post_server_action(server['id'], {'rescue': { + 'rescue_image_ref': image['id']}}) + self._wait_for_state_change(server, 'RESCUE') + + +class DisallowBFVRescuev287WithoutRescueImageProperties(BFVRescue): + """Asserts that BFV rescue requests fail with microversion 2.87 (or later) + when the required hw_rescue_device and hw_rescue_bus image properties + are not set on the image. + """ + compute_driver = 'fake.MediumFakeDriver' + microversion = '2.87' + + def test_bfv_rescue_failed(self): + server, image = self._create_bfv_server() + # try rescue without hw_rescue_device and hw_rescue_bus properties set + ex = self.assertRaises(client.OpenStackApiException, + self.api.post_server_action, server['id'], {'rescue': { 'rescue_image_ref': '155d900f-4e14-4e4c-a73d-069cbf4541e6'}}) + self.assertEqual(400, ex.response.status_code) + self.assertIn('Cannot rescue a volume-backed instance', + ex.response.text) + + +class AllowBFVRescuev287WithRescueImageProperties(BFVRescue): + """Asserts that BFV rescue requests pass with microversion 2.87 (or later) + when the required hw_rescue_device and hw_rescue_bus image properties + are set on the image. + """ + compute_driver = 'fake.RescueBFVDriver' + microversion = '2.87' + + def test_bfv_rescue_done(self): + server, image = self._create_bfv_server() + self.api.post_server_action(server['id'], {'rescue': { + 'rescue_image_ref': image['id']}}) self._wait_for_state_change(server, 'RESCUE') diff -Nru nova-26.1.0/nova/tests/unit/api/openstack/compute/test_flavor_access.py nova-26.2.2/nova/tests/unit/api/openstack/compute/test_flavor_access.py --- nova-26.1.0/nova/tests/unit/api/openstack/compute/test_flavor_access.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/api/openstack/compute/test_flavor_access.py 2024-01-16 10:19:41.000000000 +0000 @@ -353,14 +353,37 @@ mock_verify.assert_called_once_with( req.environ['nova.context'], 'proj2') + @mock.patch('nova.objects.Flavor.remove_access') @mock.patch('nova.api.openstack.identity.verify_project_id', side_effect=exc.HTTPBadRequest( explanation="Project ID proj2 is not a valid project.")) - def test_remove_tenant_access_with_invalid_tenant(self, mock_verify): + def test_remove_tenant_access_with_invalid_tenant(self, + mock_verify, + mock_remove_access): """Tests the case that the tenant does not exist in Keystone.""" req = fakes.HTTPRequest.blank(self._prefix + '/flavors/2/action', use_admin_context=True) body = {'removeTenantAccess': {'tenant': 'proj2'}} + + self.flavor_action_controller._remove_tenant_access( + req, '2', body=body) + mock_verify.assert_called_once_with( + req.environ['nova.context'], 'proj2') + mock_remove_access.assert_called_once_with('proj2') + + @mock.patch('nova.api.openstack.identity.verify_project_id', + side_effect=exc.HTTPBadRequest( + explanation="Nova was unable to find Keystone " + "service endpoint.")) + def test_remove_tenant_access_missing_keystone_endpoint(self, + mock_verify): + """Tests the case that Keystone identity service endpoint + version 3.0 was not found. + """ + req = fakes.HTTPRequest.blank(self._prefix + '/flavors/2/action', + use_admin_context=True) + body = {'removeTenantAccess': {'tenant': 'proj2'}} + self.assertRaises(exc.HTTPBadRequest, self.flavor_action_controller._remove_tenant_access, req, '2', body=body) diff -Nru nova-26.1.0/nova/tests/unit/api/openstack/compute/test_remote_consoles.py nova-26.2.2/nova/tests/unit/api/openstack/compute/test_remote_consoles.py --- nova-26.1.0/nova/tests/unit/api/openstack/compute/test_remote_consoles.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/api/openstack/compute/test_remote_consoles.py 2024-01-16 10:19:41.000000000 +0000 @@ -104,6 +104,18 @@ 'get_vnc_console', exception.InstanceNotFound(instance_id=fakes.FAKE_UUID)) + def test_get_vnc_console_instance_invalid_state(self): + body = {'os-getVNCConsole': {'type': 'novnc'}} + self._check_console_failure( + self.controller.get_vnc_console, + webob.exc.HTTPConflict, + body, + 'get_vnc_console', + exception.InstanceInvalidState( + attr='fake-attr', state='fake-state', method='fake-method', + instance_uuid=fakes.FAKE_UUID) + ) + def test_get_vnc_console_invalid_type(self): body = {'os-getVNCConsole': {'type': 'invalid'}} self._check_console_failure( diff -Nru nova-26.1.0/nova/tests/unit/cmd/test_status.py nova-26.2.2/nova/tests/unit/cmd/test_status.py --- nova-26.1.0/nova/tests/unit/cmd/test_status.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/cmd/test_status.py 2024-01-16 10:19:41.000000000 +0000 @@ -446,3 +446,19 @@ upgradecheck.Code.SUCCESS, result.code ) + + +class TestUpgradeCheckServiceUserToken(test.NoDBTestCase): + + def setUp(self): + super().setUp() + self.cmd = status.UpgradeCommands() + + def test_service_user_token_not_configured(self): + result = self.cmd._check_service_user_token() + self.assertEqual(upgradecheck.Code.FAILURE, result.code) + + def test_service_user_token_configured(self): + self.flags(send_service_user_token=True, group='service_user') + result = self.cmd._check_service_user_token() + self.assertEqual(upgradecheck.Code.SUCCESS, result.code) diff -Nru nova-26.1.0/nova/tests/unit/compute/test_api.py nova-26.2.2/nova/tests/unit/compute/test_api.py --- nova-26.1.0/nova/tests/unit/compute/test_api.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/compute/test_api.py 2024-01-16 10:19:41.000000000 +0000 @@ -5790,7 +5790,10 @@ destination_type='volume', volume_type=None, snapshot_id=None, volume_id=uuids.volume_id, volume_size=None)]) - rescue_image_meta_obj = image_meta_obj.ImageMeta.from_dict({}) + rescue_image_meta_obj = image_meta_obj.ImageMeta.from_dict({ + 'properties': {'hw_rescue_device': 'disk', + 'hw_rescue_bus': 'scsi'} + }) with test.nested( mock.patch.object(self.compute_api.placementclient, @@ -5842,6 +5845,7 @@ # Assert that the instance task state as set in the compute API self.assertEqual(task_states.RESCUING, instance.task_state) + @mock.patch('nova.objects.instance.Instance.image_meta') @mock.patch('nova.objects.compute_node.ComputeNode' '.get_by_host_and_nodename') @mock.patch('nova.compute.utils.is_volume_backed_instance', @@ -5850,7 +5854,8 @@ '.get_by_instance_uuid') def test_rescue_bfv_without_required_trait(self, mock_get_bdms, mock_is_volume_backed, - mock_get_cn): + mock_get_cn, + mock_image_meta): instance = self._create_instance_obj() bdms = objects.BlockDeviceMappingList(objects=[ objects.BlockDeviceMapping( @@ -5858,6 +5863,12 @@ destination_type='volume', volume_type=None, snapshot_id=None, volume_id=uuids.volume_id, volume_size=None)]) + + instance.image_meta = image_meta_obj.ImageMeta.from_dict({ + 'properties': {'hw_rescue_device': 'disk', + 'hw_rescue_bus': 'scsi'} + }) + with test.nested( mock.patch.object(self.compute_api.placementclient, 'get_provider_traits'), @@ -5895,6 +5906,124 @@ mock_get_traits.assert_called_once_with( self.context, uuids.cn) + @mock.patch('nova.objects.image_meta.ImageMeta.from_image_ref') + @mock.patch('nova.objects.compute_node.ComputeNode' + '.get_by_host_and_nodename') + @mock.patch('nova.compute.utils.is_volume_backed_instance', + return_value=True) + @mock.patch('nova.objects.block_device.BlockDeviceMappingList' + '.get_by_instance_uuid') + def test_rescue_bfv_with_required_image_properties( + self, mock_get_bdms, mock_is_volume_backed, mock_get_cn, + mock_image_meta_obj_from_ref): + instance = self._create_instance_obj() + bdms = objects.BlockDeviceMappingList(objects=[ + objects.BlockDeviceMapping( + boot_index=0, image_id=uuids.image_id, source_type='image', + destination_type='volume', volume_type=None, + snapshot_id=None, volume_id=uuids.volume_id, + volume_size=None)]) + rescue_image_meta_obj = image_meta_obj.ImageMeta.from_dict({ + 'properties': {'hw_rescue_device': 'disk', + 'hw_rescue_bus': 'scsi'} + }) + + with test.nested( + mock.patch.object(self.compute_api.placementclient, + 'get_provider_traits'), + mock.patch.object(self.compute_api.volume_api, 'get'), + mock.patch.object(self.compute_api.volume_api, 'check_attached'), + mock.patch.object(instance, 'save'), + mock.patch.object(self.compute_api, '_record_action_start'), + mock.patch.object(self.compute_api.compute_rpcapi, + 'rescue_instance') + ) as ( + mock_get_traits, mock_get_volume, mock_check_attached, + mock_instance_save, mock_record_start, mock_rpcapi_rescue + ): + # Mock out the returned compute node, image_meta, bdms and volume + mock_image_meta_obj_from_ref.return_value = rescue_image_meta_obj + mock_get_bdms.return_value = bdms + mock_get_volume.return_value = mock.sentinel.volume + mock_get_cn.return_value = mock.Mock(uuid=uuids.cn) + + # Ensure the required trait is returned, allowing BFV rescue + mock_trait_info = mock.Mock(traits=[ot.COMPUTE_RESCUE_BFV]) + mock_get_traits.return_value = mock_trait_info + + # Try to rescue the instance + self.compute_api.rescue(self.context, instance, + rescue_image_ref=uuids.rescue_image_id, + allow_bfv_rescue=True) + + # Assert all of the calls made in the compute API + mock_get_bdms.assert_called_once_with(self.context, instance.uuid) + mock_get_volume.assert_called_once_with( + self.context, uuids.volume_id) + mock_check_attached.assert_called_once_with( + self.context, mock.sentinel.volume) + mock_is_volume_backed.assert_called_once_with( + self.context, instance, bdms) + mock_get_cn.assert_called_once_with( + self.context, instance.host, instance.node) + mock_get_traits.assert_called_once_with(self.context, uuids.cn) + mock_instance_save.assert_called_once_with( + expected_task_state=[None]) + mock_record_start.assert_called_once_with( + self.context, instance, instance_actions.RESCUE) + mock_rpcapi_rescue.assert_called_once_with( + self.context, instance=instance, rescue_password=None, + rescue_image_ref=uuids.rescue_image_id, clean_shutdown=True) + + # Assert that the instance task state as set in the compute API + self.assertEqual(task_states.RESCUING, instance.task_state) + + @mock.patch('nova.objects.image_meta.ImageMeta.from_image_ref') + @mock.patch('nova.compute.utils.is_volume_backed_instance', + return_value=True) + @mock.patch('nova.objects.block_device.BlockDeviceMappingList' + '.get_by_instance_uuid') + def test_rescue_bfv_without_required_image_properties( + self, mock_get_bdms, mock_is_volume_backed, + mock_image_meta_obj_from_ref): + instance = self._create_instance_obj() + bdms = objects.BlockDeviceMappingList(objects=[ + objects.BlockDeviceMapping( + boot_index=0, image_id=uuids.image_id, source_type='image', + destination_type='volume', volume_type=None, + snapshot_id=None, volume_id=uuids.volume_id, + volume_size=None)]) + rescue_image_meta_obj = image_meta_obj.ImageMeta.from_dict({ + 'properties': {} + }) + + with test.nested( + mock.patch.object(self.compute_api.volume_api, 'get'), + mock.patch.object(self.compute_api.volume_api, 'check_attached'), + ) as ( + mock_get_volume, mock_check_attached + ): + # Mock out the returned bdms, volume and image_meta + mock_get_bdms.return_value = bdms + mock_get_volume.return_value = mock.sentinel.volume + mock_image_meta_obj_from_ref.return_value = rescue_image_meta_obj + + # Assert that any attempt to rescue a bfv instance on a compute + # node that does not report the COMPUTE_RESCUE_BFV trait fails and + # raises InstanceNotRescuable + self.assertRaises(exception.InstanceNotRescuable, + self.compute_api.rescue, self.context, instance, + rescue_image_ref=None, allow_bfv_rescue=True) + + # Assert the calls made in the compute API prior to the failure + mock_get_bdms.assert_called_once_with(self.context, instance.uuid) + mock_get_volume.assert_called_once_with( + self.context, uuids.volume_id) + mock_check_attached.assert_called_once_with( + self.context, mock.sentinel.volume) + mock_is_volume_backed.assert_called_once_with( + self.context, instance, bdms) + @mock.patch('nova.compute.utils.is_volume_backed_instance', return_value=True) @mock.patch('nova.objects.block_device.BlockDeviceMappingList' diff -Nru nova-26.1.0/nova/tests/unit/compute/test_compute_mgr.py nova-26.2.2/nova/tests/unit/compute/test_compute_mgr.py --- nova-26.1.0/nova/tests/unit/compute/test_compute_mgr.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/compute/test_compute_mgr.py 2024-01-16 10:19:41.000000000 +0000 @@ -6753,13 +6753,14 @@ self.compute = manager.ComputeManager() self._test_build_and_run_instance() + @mock.patch.object(manager.ComputeManager, '_build_succeeded') @mock.patch.object(objects.InstanceActionEvent, 'event_finish_with_failure') @mock.patch.object(objects.InstanceActionEvent, 'event_start') @mock.patch.object(objects.Instance, 'save') @mock.patch.object(manager.ComputeManager, '_build_and_run_instance') def _test_build_and_run_instance(self, mock_build, mock_save, - mock_start, mock_finish): + mock_start, mock_finish, mock_succeeded): self._do_build_instance_update(mock_save) orig_do_build_and_run = self.compute._do_build_and_run_instance @@ -6792,6 +6793,7 @@ self.requested_networks, self.security_groups, self.block_device_mapping, self.node, self.limits, self.filter_properties, {}, self.accel_uuids) + mock_succeeded.assert_called_once_with(self.node) # This test when sending an icehouse compatible rpc call to juno compute # node, NetworkRequest object can load from three items tuple. @@ -6819,6 +6821,7 @@ self.assertEqual('10.0.0.1', str(requested_network.address)) self.assertEqual(uuids.port_instance, requested_network.port_id) + @mock.patch.object(manager.ComputeManager, '_build_failed') @mock.patch.object(objects.InstanceActionEvent, 'event_finish_with_failure') @mock.patch.object(objects.InstanceActionEvent, 'event_start') @@ -6834,7 +6837,7 @@ def test_build_abort_exception(self, mock_build_run, mock_build, mock_set, mock_nil, mock_add, mock_clean_vol, mock_clean_net, mock_save, - mock_start, mock_finish): + mock_start, mock_finish, mock_failed): self._do_build_instance_update(mock_save) mock_build_run.side_effect = exception.BuildAbortException(reason='', instance_uuid=self.instance.uuid) @@ -6877,7 +6880,9 @@ mock.ANY, mock.ANY) mock_nil.assert_called_once_with(self.instance) mock_set.assert_called_once_with(self.instance, clean_task_state=True) + mock_failed.assert_called_once_with(self.node) + @mock.patch.object(manager.ComputeManager, '_build_failed') @mock.patch.object(objects.InstanceActionEvent, 'event_finish_with_failure') @mock.patch.object(objects.InstanceActionEvent, 'event_start') @@ -6888,8 +6893,8 @@ @mock.patch.object(conductor_api.ComputeTaskAPI, 'build_instances') @mock.patch.object(manager.ComputeManager, '_build_and_run_instance') def test_rescheduled_exception(self, mock_build_run, - mock_build, mock_set, mock_nil, - mock_save, mock_start, mock_finish): + mock_build, mock_set, mock_nil, mock_save, + mock_start, mock_finish, mock_failed): self._do_build_instance_update(mock_save, reschedule_update=True) mock_build_run.side_effect = exception.RescheduledException(reason='', instance_uuid=self.instance.uuid) @@ -6936,6 +6941,7 @@ self.admin_pass, self.injected_files, self.requested_networks, self.security_groups, self.block_device_mapping, request_spec={}, host_lists=[fake_host_list]) + mock_failed.assert_called_once_with(self.node) @mock.patch.object(manager.ComputeManager, '_shutdown_instance') @mock.patch.object(manager.ComputeManager, '_build_networks_for_instance') @@ -7289,6 +7295,139 @@ self.security_groups, self.block_device_mapping, request_spec={}, host_lists=[fake_host_list]) + @mock.patch('nova.compute.resource_tracker.ResourceTracker.instance_claim', + new=mock.MagicMock()) + @mock.patch.object(objects.InstanceActionEvent, + 'event_finish_with_failure') + @mock.patch.object(objects.InstanceActionEvent, 'event_start') + @mock.patch.object(objects.Instance, 'save') + @mock.patch.object(manager.ComputeManager, + '_nil_out_instance_obj_host_and_node') + @mock.patch.object(conductor_api.ComputeTaskAPI, 'build_instances') + @mock.patch.object(manager.ComputeManager, '_build_failed') + @mock.patch.object(manager.ComputeManager, '_build_succeeded') + @mock.patch.object(manager.ComputeManager, + '_validate_instance_group_policy') + def test_group_affinity_violation_exception_with_retry( + self, mock_validate_policy, mock_succeeded, mock_failed, mock_build, + mock_nil, mock_save, mock_start, mock_finish, + ): + """Test retry by affinity or anti-affinity validation check doesn't + increase failed build + """ + + self._do_build_instance_update(mock_save, reschedule_update=True) + mock_validate_policy.side_effect = \ + exception.GroupAffinityViolation( + instance_uuid=self.instance.uuid, policy="Affinity") + + orig_do_build_and_run = self.compute._do_build_and_run_instance + + def _wrapped_do_build_and_run_instance(*args, **kwargs): + ret = orig_do_build_and_run(*args, **kwargs) + self.assertEqual(build_results.RESCHEDULED_BY_POLICY, ret) + return ret + + with test.nested( + mock.patch.object( + self.compute, '_do_build_and_run_instance', + side_effect=_wrapped_do_build_and_run_instance, + ), + mock.patch.object( + self.compute.network_api, 'get_instance_nw_info', + ), + ): + self.compute.build_and_run_instance( + self.context, self.instance, + self.image, request_spec={}, + filter_properties=self.filter_properties, + accel_uuids=self.accel_uuids, + injected_files=self.injected_files, + admin_password=self.admin_pass, + requested_networks=self.requested_networks, + security_groups=self.security_groups, + block_device_mapping=self.block_device_mapping, node=self.node, + limits=self.limits, host_list=fake_host_list) + + mock_succeeded.assert_not_called() + mock_failed.assert_not_called() + + self._instance_action_events(mock_start, mock_finish) + self._assert_build_instance_update(mock_save, reschedule_update=True) + mock_nil.assert_called_once_with(self.instance) + mock_build.assert_called_once_with(self.context, + [self.instance], self.image, self.filter_properties, + self.admin_pass, self.injected_files, self.requested_networks, + self.security_groups, self.block_device_mapping, + request_spec={}, host_lists=[fake_host_list]) + + @mock.patch('nova.compute.resource_tracker.ResourceTracker.instance_claim', + new=mock.MagicMock()) + @mock.patch.object(objects.InstanceActionEvent, + 'event_finish_with_failure') + @mock.patch.object(objects.InstanceActionEvent, 'event_start') + @mock.patch.object(objects.Instance, 'save') + @mock.patch.object(manager.ComputeManager, + '_nil_out_instance_obj_host_and_node') + @mock.patch.object(manager.ComputeManager, '_cleanup_allocated_networks') + @mock.patch.object(manager.ComputeManager, '_set_instance_obj_error_state') + @mock.patch.object(compute_utils, 'add_instance_fault_from_exc') + @mock.patch.object(conductor_api.ComputeTaskAPI, 'build_instances') + @mock.patch.object(manager.ComputeManager, '_build_failed') + @mock.patch.object(manager.ComputeManager, '_build_succeeded') + @mock.patch.object(manager.ComputeManager, + '_validate_instance_group_policy') + def test_group_affinity_violation_exception_without_retry( + self, mock_validate_policy, mock_succeeded, mock_failed, mock_build, + mock_add, mock_set_state, mock_clean_net, mock_nil, mock_save, + mock_start, mock_finish, + ): + """Test failure by affinity or anti-affinity validation check doesn't + increase failed build + """ + + self._do_build_instance_update(mock_save) + mock_validate_policy.side_effect = \ + exception.GroupAffinityViolation( + instance_uuid=self.instance.uuid, policy="Affinity") + + orig_do_build_and_run = self.compute._do_build_and_run_instance + + def _wrapped_do_build_and_run_instance(*args, **kwargs): + ret = orig_do_build_and_run(*args, **kwargs) + self.assertEqual(build_results.FAILED_BY_POLICY, ret) + return ret + + with mock.patch.object( + self.compute, '_do_build_and_run_instance', + side_effect=_wrapped_do_build_and_run_instance, + ): + self.compute.build_and_run_instance( + self.context, self.instance, + self.image, request_spec={}, + filter_properties={}, + accel_uuids=[], + injected_files=self.injected_files, + admin_password=self.admin_pass, + requested_networks=self.requested_networks, + security_groups=self.security_groups, + block_device_mapping=self.block_device_mapping, node=self.node, + limits=self.limits, host_list=fake_host_list) + + mock_succeeded.assert_not_called() + mock_failed.assert_not_called() + + self._instance_action_events(mock_start, mock_finish) + self._assert_build_instance_update(mock_save) + mock_clean_net.assert_called_once_with(self.context, self.instance, + self.requested_networks) + mock_add.assert_called_once_with(self.context, self.instance, + mock.ANY, mock.ANY, fault_message=mock.ANY) + mock_nil.assert_called_once_with(self.instance) + mock_build.assert_not_called() + mock_set_state.assert_called_once_with(self.instance, + clean_task_state=True) + @mock.patch.object(objects.InstanceActionEvent, 'event_finish_with_failure') @mock.patch.object(objects.InstanceActionEvent, 'event_start') @@ -7868,7 +8007,7 @@ nodes.return_value = ['nodename'] migration_list.return_value = [objects.Migration( uuid=uuids.migration, instance_uuid=uuids.instance)] - self.assertRaises(exception.RescheduledException, + self.assertRaises(exception.GroupAffinityViolation, self.compute._validate_instance_group_policy, self.context, instance, hints) @@ -7924,6 +8063,42 @@ mock_clean.assert_called_once_with(self.instance, self.network_info) mock_prepspawn.assert_called_once_with(self.instance) mock_failedspawn.assert_called_once_with(self.instance) + + @mock.patch.object(virt_driver.ComputeDriver, 'failed_spawn_cleanup') + @mock.patch.object(virt_driver.ComputeDriver, 'prepare_for_spawn') + @mock.patch.object(virt_driver.ComputeDriver, + 'prepare_networks_before_block_device_mapping') + @mock.patch.object(virt_driver.ComputeDriver, + 'clean_networks_preparation') + def test_failed_prepare_for_spawn(self, mock_clean, mock_prepnet, + mock_prepspawn, mock_failedspawn): + mock_prepspawn.side_effect = exception.ComputeResourcesUnavailable( + reason="asdf") + with mock.patch.object(self.compute, + '_build_networks_for_instance', + return_value=self.network_info + ) as _build_networks_for_instance: + + try: + with self.compute._build_resources(self.context, self.instance, + self.requested_networks, self.security_groups, + self.image, self.block_device_mapping, + self.resource_provider_mapping, self.accel_uuids): + pass + except Exception as e: + self.assertIsInstance(e, + exception.ComputeResourcesUnavailable) + + _build_networks_for_instance.assert_has_calls( + [mock.call(self.context, self.instance, + self.requested_networks, self.security_groups, + self.resource_provider_mapping, + self.network_arqs)]) + + mock_prepnet.assert_not_called() + mock_clean.assert_called_once_with(self.instance, self.network_info) + mock_prepspawn.assert_called_once_with(self.instance) + mock_failedspawn.assert_called_once_with(self.instance) @mock.patch.object(virt_driver.ComputeDriver, 'failed_spawn_cleanup') @mock.patch.object(virt_driver.ComputeDriver, 'prepare_for_spawn') diff -Nru nova-26.1.0/nova/tests/unit/compute/test_compute.py nova-26.2.2/nova/tests/unit/compute/test_compute.py --- nova-26.1.0/nova/tests/unit/compute/test_compute.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/compute/test_compute.py 2024-01-16 10:19:41.000000000 +0000 @@ -5670,6 +5670,7 @@ pagesize=2048, cpu_usage=2, memory_usage=0, + socket=0, pinned_cpus=set([1, 2]), siblings=[set([1]), set([2])], mempages=[objects.NUMAPagesTopology( @@ -5685,6 +5686,7 @@ pagesize=2048, memory_usage=0, cpu_usage=0, + socket=0, siblings=[set([3]), set([4])], mempages=[objects.NUMAPagesTopology( size_kb=2048, total=256, used=0)]) diff -Nru nova-26.1.0/nova/tests/unit/compute/test_resource_tracker.py nova-26.2.2/nova/tests/unit/compute/test_resource_tracker.py --- nova-26.1.0/nova/tests/unit/compute/test_resource_tracker.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/compute/test_resource_tracker.py 2024-01-16 10:19:41.000000000 +0000 @@ -179,6 +179,7 @@ memory=_2MB, cpu_usage=0, memory_usage=0, + socket=0, mempages=[_NUMA_PAGE_TOPOLOGIES['2mb*1024']], siblings=[set([1]), set([2])], pinned_cpus=set()), @@ -189,6 +190,7 @@ memory=_2MB, cpu_usage=0, memory_usage=0, + socket=0, mempages=[_NUMA_PAGE_TOPOLOGIES['2mb*1024']], siblings=[set([3]), set([4])], pinned_cpus=set())]), diff -Nru nova-26.1.0/nova/tests/unit/compute/test_stats.py nova-26.2.2/nova/tests/unit/compute/test_stats.py --- nova-26.1.0/nova/tests/unit/compute/test_stats.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/compute/test_stats.py 2024-01-16 10:19:41.000000000 +0000 @@ -208,6 +208,22 @@ self.assertEqual(0, self.stats.num_os_type("Linux")) self.assertEqual(0, self.stats["num_vm_" + vm_states.BUILDING]) + def test_update_stats_for_instance_being_unshelved(self): + instance = self._create_instance() + self.stats.update_stats_for_instance(instance) + self.assertEqual(1, self.stats.num_instances_for_project("1234")) + + instance["vm_state"] = vm_states.SHELVED_OFFLOADED + instance["task_state"] = task_states.SPAWNING + self.stats.update_stats_for_instance(instance) + + self.assertEqual(1, self.stats.num_instances) + self.assertEqual(1, self.stats.num_instances_for_project(1234)) + self.assertEqual(1, self.stats["num_os_type_Linux"]) + self.assertEqual(1, self.stats["num_vm_%s" % + vm_states.SHELVED_OFFLOADED]) + self.assertEqual(1, self.stats["num_task_%s" % task_states.SPAWNING]) + def test_io_workload(self): vms = [vm_states.ACTIVE, vm_states.BUILDING, vm_states.PAUSED] tasks = [task_states.RESIZE_MIGRATING, task_states.REBUILDING, diff -Nru nova-26.1.0/nova/tests/unit/network/test_neutron.py nova-26.2.2/nova/tests/unit/network/test_neutron.py --- nova-26.1.0/nova/tests/unit/network/test_neutron.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/network/test_neutron.py 2024-01-16 10:19:41.000000000 +0000 @@ -142,6 +142,22 @@ self.assertIsInstance(cl.httpclient.auth, service_token.ServiceTokenAuthWrapper) + @mock.patch('nova.service_auth._SERVICE_AUTH') + @mock.patch('nova.network.neutron._ADMIN_AUTH') + @mock.patch.object(ks_loading, 'load_auth_from_conf_options') + def test_admin_with_service_token( + self, mock_load, mock_admin_auth, mock_service_auth + ): + self.flags(send_service_user_token=True, group='service_user') + + admin_context = context.get_admin_context() + + cl = neutronapi.get_client(admin_context) + self.assertIsInstance(cl.httpclient.auth, + service_token.ServiceTokenAuthWrapper) + self.assertEqual(mock_admin_auth, cl.httpclient.auth.user_auth) + self.assertEqual(mock_service_auth, cl.httpclient.auth.service_auth) + @mock.patch.object(client.Client, "list_networks", side_effect=exceptions.Unauthorized()) def test_Unauthorized_user(self, mock_list_networks): @@ -7415,7 +7431,7 @@ network_id=uuids.network_id, fields='segment_id') @mock.patch.object(neutronapi, 'get_client') - def test_get_segment_ids_for_network_with_no_segments(self, mock_client): + def test_get_segment_ids_for_network_with_segments_none(self, mock_client): subnets = {'subnets': [{'segment_id': None}]} mocked_client = mock.create_autospec(client.Client) mock_client.return_value = mocked_client @@ -7431,6 +7447,22 @@ network_id=uuids.network_id, fields='segment_id') @mock.patch.object(neutronapi, 'get_client') + def test_get_segment_ids_for_network_with_no_segments(self, mock_client): + subnets = {'subnets': [{}]} + mocked_client = mock.create_autospec(client.Client) + mock_client.return_value = mocked_client + mocked_client.list_subnets.return_value = subnets + with mock.patch.object( + self.api, 'has_segment_extension', return_value=True, + ): + res = self.api.get_segment_ids_for_network( + self.context, uuids.network_id) + self.assertEqual([], res) + mock_client.assert_called_once_with(self.context, admin=True) + mocked_client.list_subnets.assert_called_once_with( + network_id=uuids.network_id, fields='segment_id') + + @mock.patch.object(neutronapi, 'get_client') def test_get_segment_ids_for_network_fails(self, mock_client): mocked_client = mock.create_autospec(client.Client) mock_client.return_value = mocked_client @@ -8112,17 +8144,20 @@ 'pf_mac_address': '52:54:00:1e:59:c6', 'vf_num': 1, }, + 'network_caps': ['gso', 'sg', 'tso', 'tx'], 'dev_type': obj_fields.PciDeviceType.SRIOV_VF, } PciDevice = collections.namedtuple('PciDevice', ['vendor_id', 'product_id', 'address', 'card_serial_number', 'sriov_cap', - 'dev_type', 'parent_addr']) + 'dev_type', 'parent_addr', + 'network_caps']) mydev = PciDevice(**pci_dev) self.assertEqual(self.api._get_vf_pci_device_profile(mydev), {'pf_mac_address': '52:54:00:1e:59:c6', 'vf_num': 1, - 'card_serial_number': 'MT2113X00000'}) + 'card_serial_number': 'MT2113X00000', + 'capabilities': ['gso', 'sg', 'tso', 'tx']}) @mock.patch.object( neutronapi.API, '_get_vf_pci_device_profile', diff -Nru nova-26.1.0/nova/tests/unit/objects/test_instance.py nova-26.2.2/nova/tests/unit/objects/test_instance.py --- nova-26.1.0/nova/tests/unit/objects/test_instance.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/objects/test_instance.py 2024-01-16 10:19:41.000000000 +0000 @@ -1633,6 +1633,21 @@ self._test_save_objectfield_fk_constraint_fails( 'other_foreign_key', db_exc.DBReferenceError) + @mock.patch('nova.objects.instance.LOG.debug') + def test_obj_load_attr_log(self, mock_log_debug): + # Instance with no UUID should not log. + instance = objects.Instance() + self.assertRaises( + exception.OrphanedObjectError, instance.obj_load_attr, 'foo') + mock_log_debug.assert_not_called() + # Instance with UUID should log. + instance = objects.Instance( + uuid='127a0d59-b88c-422b-b9a1-2dc7cc51fb9a') + self.assertRaises( + exception.OrphanedObjectError, instance.obj_load_attr, 'foo') + msg = "Lazy-load of '%s' attempted by orphaned instance" + mock_log_debug.assert_called_once_with(msg, 'foo', instance=instance) + class TestRemoteInstanceObject(test_objects._RemoteTest, _TestInstanceObject): diff -Nru nova-26.1.0/nova/tests/unit/objects/test_pci_device.py nova-26.2.2/nova/tests/unit/objects/test_pci_device.py --- nova-26.1.0/nova/tests/unit/objects/test_pci_device.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/objects/test_pci_device.py 2024-01-16 10:19:41.000000000 +0000 @@ -171,6 +171,16 @@ self.pci_device = pci_device.PciDevice.create(None, self.dev_dict) self.assertEqual(self.pci_device.card_serial_number, '42') + def test_pci_device_extra_info_network_capabilities(self): + self.dev_dict = copy.copy(dev_dict) + self.pci_device = pci_device.PciDevice.create(None, self.dev_dict) + self.assertEqual(self.pci_device.network_caps, []) + + self.dev_dict = copy.copy(dev_dict) + self.dev_dict['capabilities'] = {'network': ['sg', 'tso', 'tx']} + self.pci_device = pci_device.PciDevice.create(None, self.dev_dict) + self.assertEqual(self.pci_device.network_caps, ['sg', 'tso', 'tx']) + def test_update_device(self): self.pci_device = pci_device.PciDevice.create(None, dev_dict) self.pci_device.obj_reset_changes() diff -Nru nova-26.1.0/nova/tests/unit/scheduler/fakes.py nova-26.2.2/nova/tests/unit/scheduler/fakes.py --- nova-26.1.0/nova/tests/unit/scheduler/fakes.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/scheduler/fakes.py 2024-01-16 10:19:41.000000000 +0000 @@ -34,6 +34,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=16, total=387184, used=0), @@ -46,6 +47,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=1548736, used=0), diff -Nru nova-26.1.0/nova/tests/unit/test_service_auth.py nova-26.2.2/nova/tests/unit/test_service_auth.py --- nova-26.1.0/nova/tests/unit/test_service_auth.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/test_service_auth.py 2024-01-16 10:19:41.000000000 +0000 @@ -56,3 +56,13 @@ result = service_auth.get_auth_plugin(self.ctx) self.assertEqual(1, mock_load.call_count) self.assertNotIsInstance(result, service_token.ServiceTokenAuthWrapper) + + @mock.patch.object(ks_loading, 'load_auth_from_conf_options', + new=mock.Mock()) + def test_get_auth_plugin_user_auth(self): + self.flags(send_service_user_token=True, group='service_user') + user_auth = mock.Mock() + + result = service_auth.get_auth_plugin(self.ctx, user_auth=user_auth) + + self.assertEqual(user_auth, result.user_auth) diff -Nru nova-26.1.0/nova/tests/unit/virt/hyperv/test_vmops.py nova-26.2.2/nova/tests/unit/virt/hyperv/test_vmops.py --- nova-26.1.0/nova/tests/unit/virt/hyperv/test_vmops.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/hyperv/test_vmops.py 2024-01-16 10:19:41.000000000 +0000 @@ -1129,7 +1129,7 @@ mock_unplug_vifs.assert_called_once_with( mock_instance, mock.sentinel.fake_network_info) mock_disconnect_volumes.assert_called_once_with( - mock.sentinel.FAKE_BD_INFO) + mock.sentinel.FAKE_BD_INFO, force=True) mock_delete_disk_files.assert_called_once_with( mock_instance.name) diff -Nru nova-26.1.0/nova/tests/unit/virt/hyperv/test_volumeops.py nova-26.2.2/nova/tests/unit/virt/hyperv/test_volumeops.py --- nova-26.1.0/nova/tests/unit/virt/hyperv/test_volumeops.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/hyperv/test_volumeops.py 2024-01-16 10:19:41.000000000 +0000 @@ -141,7 +141,13 @@ self._volumeops.disconnect_volumes(block_device_info) fake_volume_driver.disconnect_volume.assert_called_once_with( - block_device_mapping[0]['connection_info']) + block_device_mapping[0]['connection_info'], force=False) + + # Verify force=True + fake_volume_driver.disconnect_volume.reset_mock() + self._volumeops.disconnect_volumes(block_device_info, force=True) + fake_volume_driver.disconnect_volume.assert_called_once_with( + block_device_mapping[0]['connection_info'], force=True) @mock.patch('time.sleep') @mock.patch.object(volumeops.VolumeOps, '_get_volume_driver') @@ -181,7 +187,7 @@ if attach_failed: fake_volume_driver.disconnect_volume.assert_called_once_with( - fake_conn_info) + fake_conn_info, force=False) mock_sleep.assert_has_calls( [mock.call(CONF.hyperv.volume_attach_retry_interval)] * CONF.hyperv.volume_attach_retry_count) @@ -203,7 +209,13 @@ mock_get_volume_driver.assert_called_once_with( mock.sentinel.conn_info) fake_volume_driver.disconnect_volume.assert_called_once_with( - mock.sentinel.conn_info) + mock.sentinel.conn_info, force=False) + + # Verify force=True + fake_volume_driver.disconnect_volume.reset_mock() + self._volumeops.disconnect_volume(mock.sentinel.conn_info, force=True) + fake_volume_driver.disconnect_volume.assert_called_once_with( + mock.sentinel.conn_info, force=True) @mock.patch.object(volumeops.VolumeOps, '_get_volume_driver') def test_detach_volume(self, mock_get_volume_driver): @@ -347,7 +359,13 @@ self._base_vol_driver.disconnect_volume(conn_info) self._conn.disconnect_volume.assert_called_once_with( - conn_info['data']) + conn_info['data'], force=False) + + # Verify force=True + self._conn.disconnect_volume.reset_mock() + self._base_vol_driver.disconnect_volume(conn_info, force=True) + self._conn.disconnect_volume.assert_called_once_with( + conn_info['data'], force=True) @mock.patch.object(volumeops.BaseVolumeDriver, '_get_disk_res_path') def _test_get_disk_resource_path_by_conn_info(self, diff -Nru nova-26.1.0/nova/tests/unit/virt/ironic/test_driver.py nova-26.2.2/nova/tests/unit/virt/ironic/test_driver.py --- nova-26.1.0/nova/tests/unit/virt/ironic/test_driver.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/ironic/test_driver.py 2024-01-16 10:19:41.000000000 +0000 @@ -2542,7 +2542,10 @@ @mock.patch.object(cw.IronicClientWrapper, 'call') def test_prepare_for_spawn(self, mock_call): - node = ironic_utils.get_test_node(driver='fake') + node = ironic_utils.get_test_node( + driver='fake', instance_uuid=None, + provision_state=ironic_states.AVAILABLE, + power_state=ironic_states.POWER_OFF) self.mock_conn.get_node.return_value = node instance = fake_instance.fake_instance_obj(self.ctx, node=node.uuid) @@ -2574,7 +2577,10 @@ instance) def test_prepare_for_spawn_conflict(self): - node = ironic_utils.get_test_node(driver='fake') + node = ironic_utils.get_test_node( + driver='fake', instance_uuid=None, + provision_state=ironic_states.AVAILABLE, + power_state=ironic_states.POWER_OFF) self.mock_conn.get_node.return_value = node self.mock_conn.update_node.side_effect = sdk_exc.ConflictException instance = fake_instance.fake_instance_obj(self.ctx, node=node.id) @@ -2582,6 +2588,18 @@ self.driver.prepare_for_spawn, instance) + def test_prepare_for_spawn_not_available(self): + node = ironic_utils.get_test_node( + driver='fake', instance_uuid=None, + provision_state=ironic_states.CLEANWAIT, + power_state=ironic_states.POWER_OFF) + self.mock_conn.get_node.return_value = node + self.mock_conn.update_node.side_effect = sdk_exc.ConflictException + instance = fake_instance.fake_instance_obj(self.ctx, node=node.id) + self.assertRaises(exception.ComputeResourcesUnavailable, + self.driver.prepare_for_spawn, + instance) + @mock.patch.object(ironic_driver.IronicDriver, '_cleanup_deploy') def test_failed_spawn_cleanup(self, mock_cleanup): node = ironic_utils.get_test_node(driver='fake') diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/test_config.py nova-26.2.2/nova/tests/unit/virt/libvirt/test_config.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/test_config.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/test_config.py 2024-01-16 10:19:41.000000000 +0000 @@ -3181,6 +3181,32 @@ config.LibvirtConfigNodeDeviceMdevInformation) self.assertEqual("nvidia-11", obj.mdev_information.type) self.assertEqual(12, obj.mdev_information.iommu_group) + self.assertIsNone(obj.mdev_information.uuid) + + def test_config_mdev_device_uuid(self): + xmlin = """ + + mdev_b2107403_110c_45b0_af87_32cc91597b8a_0000_41_00_0 + /sys/devices/pci0000:40/0000:40:03.1/0000:41:00.0/b2107403-110c-45b0-af87-32cc91597b8a + pci_0000_41_00_0 + + vfio_mdev + + + + b2107403-110c-45b0-af87-32cc91597b8a + + + """ + + obj = config.LibvirtConfigNodeDevice() + obj.parse_str(xmlin) + self.assertIsInstance(obj.mdev_information, + config.LibvirtConfigNodeDeviceMdevInformation) + self.assertEqual("nvidia-442", obj.mdev_information.type) + self.assertEqual(57, obj.mdev_information.iommu_group) + self.assertEqual("b2107403-110c-45b0-af87-32cc91597b8a", + obj.mdev_information.uuid) def test_config_vdpa_device(self): xmlin = """ diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/test_driver.py nova-26.2.2/nova/tests/unit/virt/libvirt/test_driver.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/test_driver.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/test_driver.py 2024-01-16 10:19:41.000000000 +0000 @@ -694,6 +694,7 @@ 'numa_topology': None, 'config_drive': None, 'vm_mode': None, + 'vm_state': None, 'kernel_id': None, 'ramdisk_id': None, 'os_type': 'linux', @@ -740,6 +741,7 @@ imagebackend.Image._get_driver_format) self.libvirt = self.useFixture(nova_fixtures.LibvirtFixture()) + self.cgroups = self.useFixture(nova_fixtures.CGroupsFixture()) # ensure tests perform the same on all host architectures; this is # already done by the fakelibvirt fixture but we want to change the @@ -1333,6 +1335,22 @@ @mock.patch.object(libvirt_driver.LibvirtDriver, '_register_all_undefined_instance_details', new=mock.Mock()) + @mock.patch('nova.virt.libvirt.host.libvirt.Connection.compareCPU') + def test__check_cpu_compatibility_skip_compare_at_init( + self, mocked_compare + ): + self.flags(group='workarounds', skip_cpu_compare_at_startup=True) + self.flags(cpu_mode="custom", + cpu_models=["Icelake-Server-noTSX"], + cpu_model_extra_flags = ["-mpx"], + group="libvirt") + drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True) + drvr.init_host("dummyhost") + mocked_compare.assert_not_called() + + @mock.patch.object(libvirt_driver.LibvirtDriver, + '_register_all_undefined_instance_details', + new=mock.Mock()) def test__check_cpu_compatibility_with_flag(self): self.flags(cpu_mode="custom", cpu_models=["Penryn"], @@ -1343,7 +1361,7 @@ @mock.patch('nova.virt.libvirt.host.libvirt.Connection.compareCPU') def test__check_cpu_compatibility_advance_flag(self, mocked_compare): - mocked_compare.side_effect = (2, 0) + mocked_compare.side_effect = (-1, 0) self.flags(cpu_mode="custom", cpu_models=["qemu64"], cpu_model_extra_flags = ["avx", "avx2"], @@ -1356,7 +1374,7 @@ def test__check_cpu_compatibility_wrong_flag(self, mocked_compare): # here, and in the surrounding similar tests, the non-zero error # code in the compareCPU() side effect indicates error - mocked_compare.side_effect = (2, 0) + mocked_compare.side_effect = (-1, 0) self.flags(cpu_mode="custom", cpu_models=["Broadwell-noTSX"], cpu_model_extra_flags = ["a v x"], @@ -1369,7 +1387,7 @@ def test__check_cpu_compatibility_enabled_and_disabled_flags( self, mocked_compare ): - mocked_compare.side_effect = (2, 0) + mocked_compare.side_effect = (-1, 0) self.flags( cpu_mode="custom", cpu_models=["Cascadelake-Server"], @@ -3047,9 +3065,7 @@ 'fake-flavor', 'fake-image-meta').obj_to_primitive()) @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_fits(self, is_able): + def test_get_guest_config_numa_host_instance_fits(self): self.flags(cpu_shared_set=None, cpu_dedicated_set=None, group='compute') instance_ref = objects.Instance(**self.test_instance) @@ -3087,9 +3103,7 @@ @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) @mock.patch('nova.privsep.utils.supports_direct_io', new=mock.Mock(return_value=True)) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_no_fit(self, is_able): + def test_get_guest_config_numa_host_instance_no_fit(self): instance_ref = objects.Instance(**self.test_instance) image_meta = objects.ImageMeta.from_dict(self.test_image_meta) flavor = objects.Flavor(memory_mb=4096, vcpus=4, root_gb=496, @@ -3516,10 +3530,7 @@ host_topology, inst_topology, numa_tune) @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_pci_no_numa_info( - self, is_able): + def test_get_guest_config_numa_host_instance_pci_no_numa_info(self): self.flags(cpu_shared_set='3', cpu_dedicated_set=None, group='compute') @@ -3573,10 +3584,7 @@ @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) @mock.patch('nova.privsep.utils.supports_direct_io', new=mock.Mock(return_value=True)) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_2pci_no_fit( - self, is_able): + def test_get_guest_config_numa_host_instance_2pci_no_fit(self): self.flags(cpu_shared_set='3', cpu_dedicated_set=None, group='compute') instance_ref = objects.Instance(**self.test_instance) @@ -3693,10 +3701,7 @@ None) @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_fit_w_cpu_pinset( - self, is_able): + def test_get_guest_config_numa_host_instance_fit_w_cpu_pinset(self): self.flags(cpu_shared_set='2-3', cpu_dedicated_set=None, group='compute') @@ -3735,10 +3740,7 @@ self.assertIsNone(cfg.cpu.numa) @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_non_numa_host_instance_topo( - self, is_able): + def test_get_guest_config_non_numa_host_instance_topo(self): instance_topology = objects.InstanceNUMATopology(cells=[ objects.InstanceNUMACell( id=0, cpuset=set([0]), pcpuset=set(), memory=1024), @@ -3786,10 +3788,7 @@ numa_cfg_cell.memory) @mock.patch.object(host.Host, "_check_machine_type", new=mock.Mock()) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_numa_host_instance_topo( - self, is_able): + def test_get_guest_config_numa_host_instance_topo(self): self.flags(cpu_shared_set='0-5', cpu_dedicated_set=None, group='compute') @@ -7199,9 +7198,7 @@ [], image_meta, disk_info) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_with_cpu_quota(self, is_able): + def test_get_guest_config_with_cpu_quota(self): self.flags(virt_type='kvm', group='libvirt') drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True) @@ -7537,9 +7534,7 @@ self.flags(images_type='rbd', group='libvirt') self._test_get_guest_config_disk_cachemodes('rbd') - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=True) - def test_get_guest_config_with_bogus_cpu_quota(self, is_able): + def test_get_guest_config_with_bogus_cpu_quota(self): self.flags(virt_type='kvm', group='libvirt') drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True) @@ -7557,9 +7552,10 @@ drvr._get_guest_config, instance_ref, [], image_meta, disk_info) - @mock.patch.object( - host.Host, "is_cpu_control_policy_capable", return_value=False) - def test_get_update_guest_cputune(self, is_able): + def test_get_update_guest_cputune(self): + # No CPU controller on the host + self.cgroups.version = 0 + drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True) instance_ref = objects.Instance(**self.test_instance) instance_ref.flavor.extra_specs = {'quota:cpu_shares': '10000', @@ -9584,7 +9580,7 @@ drvr._disconnect_volume( self.context, fake_connection_info, fake_instance_1) mock_volume_driver.disconnect_volume.assert_called_once_with( - fake_connection_info, fake_instance_1) + fake_connection_info, fake_instance_1, force=False) @mock.patch.object(libvirt_driver.LibvirtDriver, '_detach_encryptor') @mock.patch('nova.objects.InstanceList.get_uuids_by_host') @@ -9958,7 +9954,12 @@ device_name='vdc', ), mock.call.detach_encryptor(**encryption), - mock.call.disconnect_volume(connection_info, instance)]) + mock.call.disconnect_volume( + connection_info, + instance, + force=False, + ) + ]) get_device_conf_func = mock_detach_with_retry.mock_calls[0][1][2] self.assertEqual(mock_guest.get_disk, get_device_conf_func.func) self.assertEqual(('vdc',), get_device_conf_func.args) @@ -12170,7 +12171,7 @@ mock_updated_guest_xml, mock_migrateToURI3): self.compute = manager.ComputeManager() - instance_ref = self.test_instance + instance_ref = objects.Instance(**self.test_instance) target_connection = '127.0.0.2' target_xml = self.device_xml_tmpl.format( @@ -12339,7 +12340,7 @@ mock_migrateToURI3, mock_min_version): self.compute = manager.ComputeManager() - instance_ref = self.test_instance + instance_ref = objects.Instance(**self.test_instance) target_connection = '127.0.0.2' target_xml = self.device_xml_tmpl.format( @@ -12958,12 +12959,38 @@ @mock.patch.object(fakelibvirt.virDomain, "migrateToURI3") @mock.patch('nova.virt.libvirt.migration.get_updated_guest_xml', return_value='') + def test_live_migration_paused_instance_postcopy(self, mock_new_xml, + mock_migrateToURI3, + mock_min_version): + disk_paths = [] + params = {'bandwidth': CONF.libvirt.live_migration_bandwidth} + migrate_data = objects.LibvirtLiveMigrateData(block_migration=False, + serial_listen_addr=False) + dom = fakelibvirt.virDomain + guest = libvirt_guest.Guest(dom) + drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), False) + drvr._parse_migration_flags() + instance = objects.Instance(**self.test_instance) + instance.vm_state = vm_states.PAUSED + + drvr._live_migration_operation(self.context, instance, 'dest', + True, migrate_data, guest, + disk_paths) + + # Verify VIR_MIGRATE_POSTCOPY flag was not set + self.assertEqual(drvr._live_migration_flags, 27) + mock_migrateToURI3.assert_called_once_with( + drvr._live_migration_uri('dest'), params=params, flags=27) + + @mock.patch.object(host.Host, 'has_min_version', return_value=True) + @mock.patch.object(fakelibvirt.virDomain, "migrateToURI3") + @mock.patch('nova.virt.libvirt.migration.get_updated_guest_xml', + return_value='') @mock.patch('nova.virt.libvirt.guest.Guest.get_xml_desc', return_value='') def test_block_live_migration_native_tls( self, mock_old_xml, mock_new_xml, mock_migrateToURI3, mock_min_version): self.flags(live_migration_with_native_tls=True, group='libvirt') - target_connection = None disk_paths = ['vda', 'vdb'] @@ -20257,16 +20284,64 @@ self.context, mock.sentinel.connection_info, instance, - destroy_secrets=False + destroy_secrets=False, + force=True ), mock.call( self.context, mock.sentinel.connection_info, instance, - destroy_secrets=True + destroy_secrets=True, + force=True ) ]) + @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._get_volume_driver') + @mock.patch( + 'nova.virt.libvirt.driver.LibvirtDriver._should_disconnect_target', + new=mock.Mock(return_value=True)) + @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._detach_encryptor', + new=mock.Mock()) + @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._undefine_domain', + new=mock.Mock()) + @mock.patch('nova.virt.libvirt.driver.LibvirtDriver._get_vpmems', + new=mock.Mock(return_value=None)) + def test_cleanup_disconnect_volume(self, mock_vol_driver): + """Verify that we call disconnect_volume() with force=True + + cleanup() is called by destroy() when an instance is being deleted and + force=True should be passed down to os-brick's disconnect_volume() + call, which will ensure removal of devices regardless of errors. + + We need to ensure that devices are removed when an instance is being + deleted to avoid leaving leftover devices that could later be + erroneously connected by external entities (example: multipathd) to + instances that should not have access to the volumes. + + See https://bugs.launchpad.net/nova/+bug/2004555 for details. + """ + connection_info = mock.MagicMock() + block_device_info = { + 'block_device_mapping': [ + { + 'connection_info': connection_info + } + ] + } + instance = objects.Instance(self.context, **self.test_instance) + drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI()) + + drvr.cleanup( + self.context, + instance, + network_info={}, + block_device_info=block_device_info, + destroy_vifs=False, + destroy_disks=False, + ) + mock_vol_driver.return_value.disconnect_volume.assert_called_once_with( + connection_info, instance, force=True) + @mock.patch.object(libvirt_driver.LibvirtDriver, '_get_volume_encryption') @mock.patch.object(libvirt_driver.LibvirtDriver, '_allow_native_luksv1') def test_swap_volume_native_luks_blocked(self, mock_allow_native_luksv1, @@ -22110,6 +22185,7 @@ self.flags(sysinfo_serial="none", group="libvirt") self.flags(instances_path=self.useFixture(fixtures.TempDir()).path) self.useFixture(nova_fixtures.LibvirtFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) os_vif.initialize() self.drvr = libvirt_driver.LibvirtDriver( diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/test_host.py nova-26.2.2/nova/tests/unit/virt/libvirt/test_host.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/test_host.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/test_host.py 2024-01-16 10:19:41.000000000 +0000 @@ -1337,7 +1337,7 @@ "parent_ifname": "ens1", "capabilities": { "network": ["rx", "tx", "sg", "tso", "gso", "gro", "rxvlan", - "txvlan", "rxhash"], + "txvlan", "rxhash", "switchdev"], "sriov": {"pf_mac_address": "52:54:00:1e:59:c6", "vf_num": 1}, # Should be obtained from the parent PF in this case. @@ -1613,25 +1613,59 @@ self.host.compare_cpu("cpuxml") mock_compareCPU.assert_called_once_with("cpuxml", 0) - def test_is_cpu_control_policy_capable_ok(self): + def test_is_cpu_control_policy_capable_via_neither(self): + self.useFixture(nova_fixtures.CGroupsFixture(version=0)) + self.assertFalse(self.host.is_cpu_control_policy_capable()) + + def test_is_cpu_control_policy_capable_via_cgroupsv1(self): + self.useFixture(nova_fixtures.CGroupsFixture(version=1)) + self.assertTrue(self.host.is_cpu_control_policy_capable()) + + def test_is_cpu_control_policy_capable_via_cgroupsv2(self): + self.useFixture(nova_fixtures.CGroupsFixture(version=2)) + self.assertTrue(self.host.is_cpu_control_policy_capable()) + + def test_has_cgroupsv1_cpu_controller_ok(self): m = mock.mock_open( - read_data="""cg /cgroup/cpu,cpuacct cg opt1,cpu,opt3 0 0 -cg /cgroup/memory cg opt1,opt2 0 0 -""") - with mock.patch('builtins.open', m, create=True): - self.assertTrue(self.host.is_cpu_control_policy_capable()) + read_data=( + "cg /cgroup/cpu,cpuacct cg opt1,cpu,opt3 0 0" + "cg /cgroup/memory cg opt1,opt2 0 0" + ) + ) + with mock.patch("builtins.open", m, create=True): + self.assertTrue(self.host._has_cgroupsv1_cpu_controller()) - def test_is_cpu_control_policy_capable_ko(self): + def test_has_cgroupsv1_cpu_controller_ko(self): m = mock.mock_open( - read_data="""cg /cgroup/cpu,cpuacct cg opt1,opt2,opt3 0 0 -cg /cgroup/memory cg opt1,opt2 0 0 -""") - with mock.patch('builtins.open', m, create=True): - self.assertFalse(self.host.is_cpu_control_policy_capable()) + read_data=( + "cg /cgroup/cpu,cpuacct cg opt1,opt2,opt3 0 0" + "cg /cgroup/memory cg opt1,opt2 0 0" + ) + ) + with mock.patch("builtins.open", m, create=True): + self.assertFalse(self.host._has_cgroupsv1_cpu_controller()) + + @mock.patch("builtins.open", side_effect=IOError) + def test_has_cgroupsv1_cpu_controller_ioerror(self, _): + self.assertFalse(self.host._has_cgroupsv1_cpu_controller()) - @mock.patch('builtins.open', side_effect=IOError) - def test_is_cpu_control_policy_capable_ioerror(self, mock_open): - self.assertFalse(self.host.is_cpu_control_policy_capable()) + def test_has_cgroupsv2_cpu_controller_ok(self): + m = mock.mock_open( + read_data="cpuset cpu io memory hugetlb pids rdma misc" + ) + with mock.patch("builtins.open", m, create=True): + self.assertTrue(self.host._has_cgroupsv2_cpu_controller()) + + def test_has_cgroupsv2_cpu_controller_ko(self): + m = mock.mock_open( + read_data="memory pids" + ) + with mock.patch("builtins.open", m, create=True): + self.assertFalse(self.host._has_cgroupsv2_cpu_controller()) + + @mock.patch("builtins.open", side_effect=IOError) + def test_has_cgroupsv2_cpu_controller_ioerror(self, _): + self.assertFalse(self.host._has_cgroupsv2_cpu_controller()) def test_get_canonical_machine_type(self): # this test relies on configuration from the FakeLibvirtFixture diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_fibrechannel.py 2024-01-16 10:19:41.000000000 +0000 @@ -81,3 +81,23 @@ self.assertEqual(requested_size, new_size) libvirt_driver.connector.extend_volume.assert_called_once_with( connection_info['data']) + + def test_disconnect_volume(self): + device_path = '/dev/fake-dev' + connection_info = {'data': {'device_path': device_path}} + + libvirt_driver = fibrechannel.LibvirtFibreChannelVolumeDriver( + self.fake_host) + libvirt_driver.connector.disconnect_volume = mock.MagicMock() + libvirt_driver.disconnect_volume( + connection_info, mock.sentinel.instance) + + libvirt_driver.connector.disconnect_volume.assert_called_once_with( + connection_info['data'], connection_info['data'], force=False) + + # Verify force=True + libvirt_driver.connector.disconnect_volume.reset_mock() + libvirt_driver.disconnect_volume( + connection_info, mock.sentinel.instance, force=True) + libvirt_driver.connector.disconnect_volume.assert_called_once_with( + connection_info['data'], connection_info['data'], force=True) diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_iscsi.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_iscsi.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_iscsi.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_iscsi.py 2024-01-16 10:19:41.000000000 +0000 @@ -57,10 +57,19 @@ device=device_path)) libvirt_driver.disconnect_volume(connection_info, mock.sentinel.instance) + libvirt_driver.connector.disconnect_volume.assert_called_once_with( + connection_info['data'], None, force=False) msg = mock_LOG_warning.call_args_list[0] self.assertIn('Ignoring VolumeDeviceNotFound', msg[0][0]) + # Verify force=True + libvirt_driver.connector.disconnect_volume.reset_mock() + libvirt_driver.disconnect_volume( + connection_info, mock.sentinel.instance, force=True) + libvirt_driver.connector.disconnect_volume.assert_called_once_with( + connection_info['data'], None, force=True) + def test_extend_volume(self): device_path = '/dev/fake-dev' connection_info = {'data': {'device_path': device_path}} diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_lightos.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_lightos.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_lightos.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_lightos.py 2024-01-16 10:19:41.000000000 +0000 @@ -62,7 +62,13 @@ connection_info = {'data': disk_info} lightos_driver.disconnect_volume(connection_info, None) lightos_driver.connector.disconnect_volume.assert_called_once_with( - disk_info, None) + disk_info, None, force=False) + + # Verify force=True + lightos_driver.connector.disconnect_volume.reset_mock() + lightos_driver.disconnect_volume(connection_info, None, force=True) + lightos_driver.connector.disconnect_volume.assert_called_once_with( + disk_info, None, force=True) @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', new=mock.Mock(return_value=mock.Mock())) diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_nvme.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_nvme.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_nvme.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_nvme.py 2024-01-16 10:19:41.000000000 +0000 @@ -77,7 +77,13 @@ connection_info = {'data': disk_info} nvme_driver.disconnect_volume(connection_info, None) nvme_driver.connector.disconnect_volume.assert_called_once_with( - disk_info, None) + disk_info, None, force=False) + + # Verify force=True + nvme_driver.connector.disconnect_volume.reset_mock() + nvme_driver.disconnect_volume(connection_info, None, force=True) + nvme_driver.connector.disconnect_volume.assert_called_once_with( + disk_info, None, force=True) @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', new=mock.Mock(return_value=mock.Mock())) diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_scaleio.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_scaleio.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_scaleio.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_scaleio.py 2024-01-16 10:19:41.000000000 +0000 @@ -49,7 +49,13 @@ conn = {'data': mock.sentinel.conn_data} sio.disconnect_volume(conn, mock.sentinel.instance) sio.connector.disconnect_volume.assert_called_once_with( - mock.sentinel.conn_data, None) + mock.sentinel.conn_data, None, force=False) + + # Verify force=True + sio.connector.disconnect_volume.reset_mock() + sio.disconnect_volume(conn, mock.sentinel.instance, force=True) + sio.connector.disconnect_volume.assert_called_once_with( + mock.sentinel.conn_data, None, force=True) @mock.patch('os_brick.initiator.connector.InitiatorConnector.factory', new=mock.Mock(return_value=mock.Mock())) diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_storpool.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_storpool.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_storpool.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_storpool.py 2024-01-16 10:19:41.000000000 +0000 @@ -53,9 +53,11 @@ } return {'type': 'block', 'path': test_attached[v]['path']} - def disconnect_volume(self, connection_info, device_info): + def disconnect_volume(self, connection_info, device_info, **kwargs): self.inst.assertIn('client_id', connection_info) self.inst.assertIn('volume', connection_info) + self.inst.assertIn('force', kwargs) + self.inst.assertEqual(self.inst.force, kwargs.get('force')) v = connection_info['volume'] if v not in test_attached: @@ -86,6 +88,11 @@ class LibvirtStorPoolVolumeDriverTestCase( test_volume.LibvirtVolumeBaseTestCase): + def setUp(self): + super().setUp() + # This is for testing the force flag of disconnect_volume() + self.force = False + def mock_storpool(f): def _config_inner_inner1(inst, *args, **kwargs): @mock.patch( @@ -175,3 +182,10 @@ libvirt_driver.disconnect_volume(ci_2, mock.sentinel.instance) self.assertDictEqual({}, test_attached) + + # Connect the volume again so we can detach it again + libvirt_driver.connect_volume(ci_2, mock.sentinel.instance) + # Verify force=True + self.force = True + libvirt_driver.disconnect_volume( + ci_2, mock.sentinel.instance, force=True) diff -Nru nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py --- nova-26.1.0/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/libvirt/volume/test_vzstorage.py 2024-01-16 10:19:41.000000000 +0000 @@ -95,7 +95,13 @@ conn = {'data': mock.sentinel.conn_data} drv.disconnect_volume(conn, mock.sentinel.instance) drv.connector.disconnect_volume.assert_called_once_with( - mock.sentinel.conn_data, None) + mock.sentinel.conn_data, None, force=False) + + # Verify force=True + drv.connector.disconnect_volume.reset_mock() + drv.disconnect_volume(conn, mock.sentinel.instance, force=True) + drv.connector.disconnect_volume.assert_called_once_with( + mock.sentinel.conn_data, None, force=True) def test_libvirt_vzstorage_driver_get_config(self): libvirt_driver = vzstorage.LibvirtVZStorageVolumeDriver(self.fake_host) diff -Nru nova-26.1.0/nova/tests/unit/virt/test_hardware.py nova-26.2.2/nova/tests/unit/virt/test_hardware.py --- nova-26.1.0/nova/tests/unit/virt/test_hardware.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/test_hardware.py 2024-01-16 10:19:41.000000000 +0000 @@ -2023,6 +2023,7 @@ memory=256, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=32768, used=0), @@ -2036,6 +2037,7 @@ memory=256, cpu_usage=0, memory_usage=0, + socket=1, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=32768, used=64), @@ -2049,6 +2051,7 @@ memory=2, cpu_usage=0, memory_usage=0, + socket=2, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=16)], @@ -2130,6 +2133,7 @@ memory=160, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=32768, used=32), @@ -2170,6 +2174,7 @@ memory=1024, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], @@ -2181,6 +2186,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], @@ -2192,6 +2198,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], @@ -2258,6 +2265,7 @@ memory=1024, cpu_usage=2, memory_usage=512, + socket=0, mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], siblings=[set([0]), set([1]), set([2]), set([3])], @@ -2269,6 +2277,7 @@ memory=512, cpu_usage=1, memory_usage=512, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], @@ -2280,6 +2289,7 @@ memory=256, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[ objects.NUMAPagesTopology(size_kb=4, total=512, used=0)], @@ -2330,6 +2340,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[objects.NUMAPagesTopology( size_kb=2048, total=512, used=128, @@ -2342,6 +2353,7 @@ memory=512, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), mempages=[objects.NUMAPagesTopology( size_kb=1048576, total=5, used=2, @@ -2606,6 +2618,7 @@ memory=2048, cpu_usage=2, memory_usage=2048, + socket=0, pinned_cpus=set(), mempages=[objects.NUMAPagesTopology( size_kb=4, total=524288, used=0)], @@ -2616,6 +2629,7 @@ memory=2048, cpu_usage=2, memory_usage=2048, + socket=0, pinned_cpus=set(), mempages=[objects.NUMAPagesTopology( size_kb=4, total=524288, used=0)], @@ -4160,6 +4174,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0]), set([1]), set([2]), set([3])], mempages=[objects.NUMAPagesTopology( @@ -4189,6 +4204,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set([0, 1, 3]), mempages=[objects.NUMAPagesTopology( size_kb=4, total=524288, used=0)], @@ -4218,6 +4234,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0]), set([1]), set([2]), set([3])], mempages=[objects.NUMAPagesTopology( @@ -4246,6 +4263,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0, 2]), set([1, 3])], mempages=[objects.NUMAPagesTopology( @@ -4272,6 +4290,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set([0, 1, 2, 3]), siblings=[set([0, 2]), set([1, 3])], mempages=[objects.NUMAPagesTopology( @@ -4298,6 +4317,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0]), set([1]), set([2]), set([3])], mempages=[objects.NUMAPagesTopology( @@ -4324,6 +4344,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set([0, 1, 2, 3]), siblings=[set([0]), set([1]), set([2]), set([3])], mempages=[objects.NUMAPagesTopology( @@ -4353,6 +4374,7 @@ memory=4096, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set([2]), siblings=[set([0, 4]), set([1, 5]), set([2, 6]), set([3, 7])], mempages=[objects.NUMAPagesTopology( @@ -4383,6 +4405,7 @@ memory=4096, cpu_usage=2, memory_usage=0, + socket=0, pinned_cpus=set([2, 6, 7]), siblings=[set([0, 4]), set([1, 5]), set([2, 6]), set([3, 7])], mempages=[objects.NUMAPagesTopology( @@ -4415,6 +4438,7 @@ cpu_usage=2, memory_usage=0, pinned_cpus=set(), + socket=0, siblings=[{cpu} for cpu in range(8)], mempages=[objects.NUMAPagesTopology( size_kb=4, total=524288, used=0)] @@ -4448,6 +4472,7 @@ memory=4096, cpu_usage=2, memory_usage=0, + socket=0, pinned_cpus=set([0, 1, 2, 3]), siblings=[{cpu} for cpu in range(8)], mempages=[objects.NUMAPagesTopology( @@ -4490,6 +4515,7 @@ memory=4096, cpu_usage=2, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0, 5]), set([1, 6]), set([2, 7]), set([3, 8]), set([4, 9])], @@ -4529,6 +4555,7 @@ memory=4096, cpu_usage=2, memory_usage=0, + socket=0, pinned_cpus=set([0, 1, 2, 5, 6, 7]), siblings=[set([0, 5]), set([1, 6]), set([2, 7]), set([3, 8]), set([4, 9])], @@ -4764,6 +4791,7 @@ memory=2048, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([0]), set([1])], mempages=[objects.NUMAPagesTopology( @@ -4775,6 +4803,7 @@ memory=2048, cpu_usage=0, memory_usage=0, + socket=0, pinned_cpus=set(), siblings=[set([2]), set([3])], mempages=[objects.NUMAPagesTopology( diff -Nru nova-26.1.0/nova/tests/unit/virt/test_netutils.py nova-26.2.2/nova/tests/unit/virt/test_netutils.py --- nova-26.1.0/nova/tests/unit/virt/test_netutils.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/test_netutils.py 2024-01-16 10:19:41.000000000 +0000 @@ -17,6 +17,17 @@ class TestNetUtilsTestCase(test.NoDBTestCase): + + def _get_fake_instance_nw_info(self, num_networks, dhcp_server, mtu): + network_info = fake_network.fake_get_instance_nw_info(self, + num_networks) + for vif in network_info: + for subnet in vif['network']['subnets']: + subnet['meta']['dhcp_server'] = dhcp_server + vif['network']['meta']['mtu'] = mtu + + return network_info + def test_get_cached_vifs_with_vlan_no_nw_info(self): # Make sure that an empty dictionary will be returned when # nw_info is None @@ -39,3 +50,15 @@ expected = {'fa:16:3e:d1:28:e4': '2145'} self.assertEqual(expected, netutils.get_cached_vifs_with_vlan(network_info)) + + def test__get_link_mtu(self): + network_info_dhcp = self._get_fake_instance_nw_info( + 1, '192.168.0.100', 9000) + network_info_no_dhcp = self._get_fake_instance_nw_info( + 1, None, 9000) + + for vif in network_info_dhcp: + self.assertIsNone(netutils._get_link_mtu(vif)) + + for vif in network_info_no_dhcp: + self.assertEqual(9000, netutils._get_link_mtu(vif)) diff -Nru nova-26.1.0/nova/tests/unit/virt/test_virt_drivers.py nova-26.2.2/nova/tests/unit/virt/test_virt_drivers.py --- nova-26.1.0/nova/tests/unit/virt/test_virt_drivers.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/virt/test_virt_drivers.py 2024-01-16 10:19:41.000000000 +0000 @@ -832,6 +832,7 @@ # This is needed for the live migration tests which spawn off the # operation for monitoring. self.useFixture(nova_fixtures.SpawnIsSynchronousFixture()) + self.useFixture(nova_fixtures.CGroupsFixture()) # When destroying an instance, os-vif will try to execute some commands # which hang tests so let's just stub out the unplug call to os-vif # since we don't care about it. diff -Nru nova-26.1.0/nova/tests/unit/volume/test_cinder.py nova-26.2.2/nova/tests/unit/volume/test_cinder.py --- nova-26.1.0/nova/tests/unit/volume/test_cinder.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/tests/unit/volume/test_cinder.py 2024-01-16 10:19:41.000000000 +0000 @@ -1276,3 +1276,14 @@ admin_ctx = context.get_admin_context() params = cinder._get_cinderclient_parameters(admin_ctx) self.assertEqual(params[0], mock_admin_auth) + + @mock.patch('nova.service_auth._SERVICE_AUTH') + @mock.patch('nova.volume.cinder._ADMIN_AUTH') + def test_admin_context_without_user_token_but_with_service_token( + self, mock_admin_auth, mock_service_auth + ): + self.flags(send_service_user_token=True, group='service_user') + admin_ctx = context.get_admin_context() + params = cinder._get_cinderclient_parameters(admin_ctx) + self.assertEqual(mock_admin_auth, params[0].user_auth) + self.assertEqual(mock_service_auth, params[0].service_auth) diff -Nru nova-26.1.0/nova/virt/hardware.py nova-26.2.2/nova/virt/hardware.py --- nova-26.1.0/nova/virt/hardware.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/hardware.py 2024-01-16 10:19:41.000000000 +0000 @@ -2566,6 +2566,7 @@ cpuset=host_cell.cpuset, pcpuset=host_cell.pcpuset, memory=host_cell.memory, + socket=host_cell.socket, cpu_usage=0, memory_usage=0, mempages=host_cell.mempages, diff -Nru nova-26.1.0/nova/virt/hyperv/vmops.py nova-26.2.2/nova/virt/hyperv/vmops.py --- nova-26.1.0/nova/virt/hyperv/vmops.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/hyperv/vmops.py 2024-01-16 10:19:41.000000000 +0000 @@ -747,7 +747,7 @@ # should be disconnected even if the VM doesn't exist anymore, # so they are not leaked. self.unplug_vifs(instance, network_info) - self._volumeops.disconnect_volumes(block_device_info) + self._volumeops.disconnect_volumes(block_device_info, force=True) if destroy_disks: self._delete_disk_files(instance_name) diff -Nru nova-26.1.0/nova/virt/hyperv/volumeops.py nova-26.2.2/nova/virt/hyperv/volumeops.py --- nova-26.1.0/nova/virt/hyperv/volumeops.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/hyperv/volumeops.py 2024-01-16 10:19:41.000000000 +0000 @@ -59,10 +59,10 @@ for vol in volumes: self.attach_volume(vol['connection_info'], instance_name) - def disconnect_volumes(self, block_device_info): + def disconnect_volumes(self, block_device_info, force=False): mapping = driver.block_device_info_get_mapping(block_device_info) for vol in mapping: - self.disconnect_volume(vol['connection_info']) + self.disconnect_volume(vol['connection_info'], force=force) def attach_volume(self, connection_info, instance_name, disk_bus=constants.CTRL_TYPE_SCSI): @@ -116,9 +116,9 @@ volume_driver.set_disk_qos_specs(connection_info, qos_specs) - def disconnect_volume(self, connection_info): + def disconnect_volume(self, connection_info, force=False): volume_driver = self._get_volume_driver(connection_info) - volume_driver.disconnect_volume(connection_info) + volume_driver.disconnect_volume(connection_info, force=force) def detach_volume(self, connection_info, instance_name): LOG.debug("Detaching volume: %(connection_info)s " @@ -231,8 +231,8 @@ def connect_volume(self, connection_info): return self._connector.connect_volume(connection_info['data']) - def disconnect_volume(self, connection_info): - self._connector.disconnect_volume(connection_info['data']) + def disconnect_volume(self, connection_info, force=False): + self._connector.disconnect_volume(connection_info['data'], force=force) def get_disk_resource_path(self, connection_info): disk_paths = self._connector.get_volume_paths(connection_info['data']) diff -Nru nova-26.1.0/nova/virt/ironic/driver.py nova-26.2.2/nova/virt/ironic/driver.py --- nova-26.1.0/nova/virt/ironic/driver.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/ironic/driver.py 2024-01-16 10:19:41.000000000 +0000 @@ -397,6 +397,18 @@ _("Ironic node uuid not supplied to " "driver for instance %s.") % instance.uuid) node = self._get_node(node_uuid) + + # Its possible this node has just moved from deleting + # to cleaning. Placement will update the inventory + # as all reserved, but this instance might have got here + # before that happened, but after the previous allocation + # got deleted. We trigger a re-schedule to another node. + if (self._node_resources_used(node) or + self._node_resources_unavailable(node)): + msg = "Chosen ironic node %s is not available" % node_uuid + LOG.info(msg, instance=instance) + raise exception.ComputeResourcesUnavailable(reason=msg) + self._set_instance_id(node, instance) def failed_spawn_cleanup(self, instance): diff -Nru nova-26.1.0/nova/virt/libvirt/config.py nova-26.2.2/nova/virt/libvirt/config.py --- nova-26.1.0/nova/virt/libvirt/config.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/config.py 2024-01-16 10:19:41.000000000 +0000 @@ -3382,6 +3382,7 @@ root_name="capability", **kwargs) self.type = None self.iommu_group = None + self.uuid = None def parse_dom(self, xmldoc): super(LibvirtConfigNodeDeviceMdevInformation, @@ -3391,6 +3392,8 @@ self.type = c.get('id') if c.tag == "iommuGroup": self.iommu_group = int(c.get('number')) + if c.tag == "uuid": + self.uuid = c.text class LibvirtConfigNodeDeviceVpdCap(LibvirtConfigObject): diff -Nru nova-26.1.0/nova/virt/libvirt/driver.py nova-26.2.2/nova/virt/libvirt/driver.py --- nova-26.1.0/nova/virt/libvirt/driver.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/driver.py 2024-01-16 10:19:41.000000000 +0000 @@ -984,33 +984,26 @@ msg = _("The cpu_models option is required when cpu_mode=custom") raise exception.Invalid(msg) - cpu = vconfig.LibvirtConfigGuestCPU() - for model in models: - cpu.model = self._get_cpu_model_mapping(model) - try: - self._compare_cpu(cpu, self._get_cpu_info(), None) - except exception.InvalidCPUInfo as e: - msg = (_("Configured CPU model: %(model)s is not " - "compatible with host CPU. Please correct your " - "config and try again. %(e)s") % { - 'model': model, 'e': e}) - raise exception.InvalidCPUInfo(msg) - - # Use guest CPU model to check the compatibility between guest CPU and - # configured extra_flags - cpu = vconfig.LibvirtConfigGuestCPU() - cpu.model = self._host.get_capabilities().host.cpu.model - for flag in set(x.lower() for x in CONF.libvirt.cpu_model_extra_flags): - cpu_feature = self._prepare_cpu_flag(flag) - cpu.add_feature(cpu_feature) - try: - self._compare_cpu(cpu, self._get_cpu_info(), None) - except exception.InvalidCPUInfo as e: - msg = (_("Configured extra flag: %(flag)s it not correct, or " - "the host CPU does not support this flag. Please " - "correct the config and try again. %(e)s") % { - 'flag': flag, 'e': e}) - raise exception.InvalidCPUInfo(msg) + if not CONF.workarounds.skip_cpu_compare_at_startup: + # Use guest CPU model to check the compatibility between + # guest CPU and configured extra_flags + for model in models: + cpu = vconfig.LibvirtConfigGuestCPU() + cpu.model = self._get_cpu_model_mapping(model) + for flag in set(x.lower() for + x in CONF.libvirt.cpu_model_extra_flags): + cpu_feature = self._prepare_cpu_flag(flag) + cpu.add_feature(cpu_feature) + try: + self._compare_cpu(cpu, self._get_cpu_info(), None) + except exception.InvalidCPUInfo as e: + msg = (_("Configured CPU model: %(model)s " + "and CPU Flags %(flags)s ar not " + "compatible with host CPU. Please correct your " + "config and try again. %(e)s") % { + 'model': model, 'e': e, + 'flags': CONF.libvirt.cpu_model_extra_flags}) + raise exception.InvalidCPUInfo(msg) def _check_vtpm_support(self) -> None: # TODO(efried): A key manager must be configured to create/retrieve @@ -1644,7 +1637,7 @@ try: self._disconnect_volume( context, connection_info, instance, - destroy_secrets=destroy_secrets) + destroy_secrets=destroy_secrets, force=True) except Exception as exc: with excutils.save_and_reraise_exception() as ctxt: if cleanup_instance_disks: @@ -1961,7 +1954,7 @@ return (False if connection_count > 1 else True) def _disconnect_volume(self, context, connection_info, instance, - encryption=None, destroy_secrets=True): + encryption=None, destroy_secrets=True, force=False): self._detach_encryptor( context, connection_info, @@ -1973,7 +1966,8 @@ multiattach = connection_info.get('multiattach', False) if self._should_disconnect_target( context, instance, multiattach, vol_driver, volume_id): - vol_driver.disconnect_volume(connection_info, instance) + vol_driver.disconnect_volume( + connection_info, instance, force=force) else: LOG.info('Detected multiple connections on this host for ' 'volume: %(volume)s, skipping target disconnect.', @@ -8227,15 +8221,52 @@ def _get_mediated_device_information(self, devname): """Returns a dict of a mediated device.""" - virtdev = self._host.device_lookup_by_name(devname) + # LP #1951656 - In Libvirt 7.7, the mdev name now includes the PCI + # address of the parent device (e.g. mdev__) due to + # the mdevctl allowing for multiple mediated devs having the same UUID + # defined (only one can be active at a time). Since the guest + # information doesn't have the parent ID, try to lookup which + # mediated device is available that matches the UUID. If multiple + # devices are found that match the UUID, then this is an error + # condition. + try: + virtdev = self._host.device_lookup_by_name(devname) + except libvirt.libvirtError as ex: + if ex.get_error_code() != libvirt.VIR_ERR_NO_NODE_DEVICE: + raise + mdevs = [dev for dev in self._host.list_mediated_devices() + if dev.startswith(devname)] + # If no matching devices are found, simply raise the original + # exception indicating that no devices are found. + if not mdevs: + raise + elif len(mdevs) > 1: + msg = ("The mediated device name %(devname)s refers to a UUID " + "that is present in multiple libvirt mediated devices. " + "Matching libvirt mediated devices are %(devices)s. " + "Mediated device UUIDs must be unique for Nova." % + {'devname': devname, + 'devices': ', '.join(mdevs)}) + raise exception.InvalidLibvirtMdevConfig(reason=msg) + + LOG.debug('Found requested device %s as %s. Using that.', + devname, mdevs[0]) + virtdev = self._host.device_lookup_by_name(mdevs[0]) xmlstr = virtdev.XMLDesc(0) cfgdev = vconfig.LibvirtConfigNodeDevice() cfgdev.parse_str(xmlstr) + # Starting with Libvirt 7.3, the uuid information is available in the + # node device information. If its there, use that. Otherwise, + # fall back to the previous behavior of parsing the uuid from the + # devname. + if cfgdev.mdev_information.uuid: + mdev_uuid = cfgdev.mdev_information.uuid + else: + mdev_uuid = libvirt_utils.mdev_name2uuid(cfgdev.name) device = { "dev_id": cfgdev.name, - # name is like mdev_00ead764_fdc0_46b6_8db9_2963f5c815b4 - "uuid": libvirt_utils.mdev_name2uuid(cfgdev.name), + "uuid": mdev_uuid, # the physical GPU PCI device "parent": cfgdev.parent, "type": cfgdev.mdev_information.type, @@ -8323,6 +8354,7 @@ :param requested_types: Filter out the result for only mediated devices having those types. """ + LOG.debug('Searching for available mdevs...') allocated_mdevs = self._get_all_assigned_mediated_devices() mdevs = self._get_mediated_devices(requested_types) available_mdevs = set() @@ -8338,6 +8370,7 @@ available_mdevs.add(mdev["uuid"]) available_mdevs -= set(allocated_mdevs) + LOG.info('Available mdevs at: %s.', available_mdevs) return available_mdevs def _create_new_mediated_device(self, parent, uuid=None): @@ -8349,6 +8382,7 @@ :returns: the newly created mdev UUID or None if not possible """ + LOG.debug('Attempting to create new mdev...') supported_types = self.supported_vgpu_types # Try to see if we can still create a new mediated device devices = self._get_mdev_capable_devices(supported_types) @@ -8360,6 +8394,7 @@ # The device is not the one that was called, not creating # the mdev continue + LOG.debug('Trying on: %s.', dev_name) dev_supported_type = self._get_vgpu_type_per_pgpu(dev_name) if dev_supported_type and device['types'][ dev_supported_type]['availableInstances'] > 0: @@ -8369,7 +8404,13 @@ pci_addr = "{}:{}:{}.{}".format(*dev_name[4:].split('_')) chosen_mdev = nova.privsep.libvirt.create_mdev( pci_addr, dev_supported_type, uuid=uuid) + LOG.info('Created mdev: %s on pGPU: %s.', + chosen_mdev, pci_addr) return chosen_mdev + LOG.debug('Failed: No available instances on device.') + LOG.info('Failed to create mdev. ' + 'No free space found among the following devices: %s.', + [dev['dev_id'] for dev in devices]) @utils.synchronized(VGPU_RESOURCE_SEMAPHORE) def _allocate_mdevs(self, allocations): @@ -8452,6 +8493,8 @@ # Take the first available mdev chosen_mdev = mdevs_available.pop() else: + LOG.debug('No available mdevs where found. ' + 'Creating an new one...') chosen_mdev = self._create_new_mediated_device(parent_device) if not chosen_mdev: # If we can't find devices having available VGPUs, just raise @@ -8459,6 +8502,7 @@ reason='mdev-capable resource is not available') else: chosen_mdevs.append(chosen_mdev) + LOG.info('Allocated mdev: %s.', chosen_mdev) return chosen_mdevs def _detach_mediated_devices(self, guest): @@ -10071,6 +10115,17 @@ else: migration_flags = self._live_migration_flags + # Note(siva_krishnan): live migrating paused instance fails + # when VIR_MIGRATE_POSTCOPY flag is set. It is unset here + # to permit live migration of paused instance. + if ( + instance.vm_state == vm_states.PAUSED and + self._is_post_copy_enabled(migration_flags) + ): + LOG.debug('Post-copy flag unset because instance is paused.', + instance=instance) + migration_flags ^= libvirt.VIR_MIGRATE_POSTCOPY + if not migrate_data.serial_listen_addr: # In this context we want to ensure that serial console is # disabled on source node. This is because nova couldn't diff -Nru nova-26.1.0/nova/virt/libvirt/host.py nova-26.2.2/nova/virt/libvirt/host.py --- nova-26.1.0/nova/virt/libvirt/host.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/host.py 2024-01-16 10:19:41.000000000 +0000 @@ -1566,7 +1566,7 @@ def list_mediated_devices(self, flags=0): """Lookup mediated devices. - :returns: a list of virNodeDevice instance + :returns: a list of strings with the name of the instance """ return self._list_devices("mdev", flags=flags) @@ -1611,15 +1611,44 @@ CONFIG_CGROUP_SCHED may be disabled in some kernel configs to improve scheduler latency. """ + return self._has_cgroupsv1_cpu_controller() or \ + self._has_cgroupsv2_cpu_controller() + + def _has_cgroupsv1_cpu_controller(self): + LOG.debug(f"Searching host: '{self.get_hostname()}' " + "for CPU controller through CGroups V1...") try: with open("/proc/self/mounts", "r") as fd: for line in fd.readlines(): # mount options and split options bits = line.split()[3].split(",") if "cpu" in bits: + LOG.debug("CPU controller found on host.") + return True + LOG.debug("CPU controller missing on host.") + return False + except IOError as ex: + LOG.debug(f"Search failed due to: '{ex}'. " + "Maybe the host is not running under CGroups V1. " + "Deemed host to be missing controller by this approach.") + return False + + def _has_cgroupsv2_cpu_controller(self): + LOG.debug(f"Searching host: '{self.get_hostname()}' " + "for CPU controller through CGroups V2...") + try: + with open("/sys/fs/cgroup/cgroup.controllers", "r") as fd: + for line in fd.readlines(): + bits = line.split() + if "cpu" in bits: + LOG.debug("CPU controller found on host.") return True + LOG.debug("CPU controller missing on host.") return False - except IOError: + except IOError as ex: + LOG.debug(f"Search failed due to: '{ex}'. " + "Maybe the host is not running under CGroups V2. " + "Deemed host to be missing controller by this approach.") return False def get_canonical_machine_type(self, arch, machine) -> str: diff -Nru nova-26.1.0/nova/virt/libvirt/utils.py nova-26.2.2/nova/virt/libvirt/utils.py --- nova-26.1.0/nova/virt/libvirt/utils.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/utils.py 2024-01-16 10:19:41.000000000 +0000 @@ -575,17 +575,31 @@ def mdev_name2uuid(mdev_name: str) -> str: - """Convert an mdev name (of the form mdev_) to a - uuid (of the form 8-4-4-4-12). + """Convert an mdev name (of the form mdev_ or + mdev__) to a uuid + (of the form 8-4-4-4-12). + + :param mdev_name: the name of the mdev to parse the UUID from + :returns: string containing the uuid """ - return str(uuid.UUID(mdev_name[5:].replace('_', '-'))) + mdev_uuid = mdev_name[5:].replace('_', '-') + # Unconditionnally remove the PCI address from the name + mdev_uuid = mdev_uuid[:36] + return str(uuid.UUID(mdev_uuid)) -def mdev_uuid2name(mdev_uuid: str) -> str: - """Convert an mdev uuid (of the form 8-4-4-4-12) to a name (of the form - mdev_). +def mdev_uuid2name(mdev_uuid: str, parent: str = None) -> str: + """Convert an mdev uuid (of the form 8-4-4-4-12) and optionally its parent + device to a name (of the form mdev_[_]). + + :param mdev_uuid: the uuid of the mediated device + :param parent: the parent device id for the mediated device + :returns: name of the mdev to reference in libvirt """ - return "mdev_" + mdev_uuid.replace('-', '_') + name = "mdev_" + mdev_uuid.replace('-', '_') + if parent and parent.startswith('pci_'): + name = name + parent[4:] + return name def get_flags_by_flavor_specs(flavor: 'objects.Flavor') -> ty.Set[str]: diff -Nru nova-26.1.0/nova/virt/libvirt/volume/fibrechannel.py nova-26.2.2/nova/virt/libvirt/volume/fibrechannel.py --- nova-26.1.0/nova/virt/libvirt/volume/fibrechannel.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/fibrechannel.py 2024-01-16 10:19:41.000000000 +0000 @@ -59,7 +59,7 @@ connection_info['data']['multipath_id'] = \ device_info['multipath_id'] - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Detach the volume from instance_name.""" LOG.debug("calling os-brick to detach FC Volume", instance=instance) @@ -69,11 +69,12 @@ # the 2nd param of disconnect_volume and be consistent # with the rest of the connectors. self.connector.disconnect_volume(connection_info['data'], - connection_info['data']) + connection_info['data'], + force=force) LOG.debug("Disconnected FC Volume", instance=instance) super(LibvirtFibreChannelVolumeDriver, - self).disconnect_volume(connection_info, instance) + self).disconnect_volume(connection_info, instance, force=force) def extend_volume(self, connection_info, instance, requested_size): """Extend the volume.""" diff -Nru nova-26.1.0/nova/virt/libvirt/volume/fs.py nova-26.2.2/nova/virt/libvirt/volume/fs.py --- nova-26.1.0/nova/virt/libvirt/volume/fs.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/fs.py 2024-01-16 10:19:41.000000000 +0000 @@ -116,7 +116,7 @@ connection_info['data']['device_path'] = \ self._get_device_path(connection_info) - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Disconnect the volume.""" vol_name = connection_info['data']['name'] mountpoint = self._get_mount_path(connection_info) diff -Nru nova-26.1.0/nova/virt/libvirt/volume/iscsi.py nova-26.2.2/nova/virt/libvirt/volume/iscsi.py --- nova-26.1.0/nova/virt/libvirt/volume/iscsi.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/iscsi.py 2024-01-16 10:19:41.000000000 +0000 @@ -66,19 +66,20 @@ connection_info['data']['device_path'] = device_info['path'] - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Detach the volume from instance_name.""" LOG.debug("calling os-brick to detach iSCSI Volume", instance=instance) try: - self.connector.disconnect_volume(connection_info['data'], None) + self.connector.disconnect_volume( + connection_info['data'], None, force=force) except os_brick_exception.VolumeDeviceNotFound as exc: LOG.warning('Ignoring VolumeDeviceNotFound: %s', exc) return LOG.debug("Disconnected iSCSI Volume", instance=instance) super(LibvirtISCSIVolumeDriver, - self).disconnect_volume(connection_info, instance) + self).disconnect_volume(connection_info, instance, force=force) def extend_volume(self, connection_info, instance, requested_size): """Extend the volume.""" diff -Nru nova-26.1.0/nova/virt/libvirt/volume/lightos.py nova-26.2.2/nova/virt/libvirt/volume/lightos.py --- nova-26.1.0/nova/virt/libvirt/volume/lightos.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/lightos.py 2024-01-16 10:19:41.000000000 +0000 @@ -42,14 +42,15 @@ LOG.debug("Connecting NVMe volume with device_info %s", device_info) connection_info['data']['device_path'] = device_info['path'] - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Detach the volume from the instance.""" LOG.debug("Disconnecting NVMe disk. instance:%s, volume_id:%s", connection_info.get("instance", ""), connection_info.get("volume_id", "")) - self.connector.disconnect_volume(connection_info['data'], None) + self.connector.disconnect_volume( + connection_info['data'], None, force=force) super(LibvirtLightOSVolumeDriver, self).disconnect_volume( - connection_info, instance) + connection_info, instance, force=force) def extend_volume(self, connection_info, instance, requested_size=None): """Extend the volume.""" diff -Nru nova-26.1.0/nova/virt/libvirt/volume/nvme.py nova-26.2.2/nova/virt/libvirt/volume/nvme.py --- nova-26.1.0/nova/virt/libvirt/volume/nvme.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/nvme.py 2024-01-16 10:19:41.000000000 +0000 @@ -45,13 +45,13 @@ connection_info['data']['device_path'] = device_info['path'] - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Detach the volume from the instance.""" LOG.debug("Disconnecting NVMe disk", instance=instance) self.connector.disconnect_volume( - connection_info['data'], None) + connection_info['data'], None, force=force) super(LibvirtNVMEVolumeDriver, - self).disconnect_volume(connection_info, instance) + self).disconnect_volume(connection_info, instance, force=force) def extend_volume(self, connection_info, instance, requested_size): """Extend the volume.""" diff -Nru nova-26.1.0/nova/virt/libvirt/volume/quobyte.py nova-26.2.2/nova/virt/libvirt/volume/quobyte.py --- nova-26.1.0/nova/virt/libvirt/volume/quobyte.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/quobyte.py 2024-01-16 10:19:41.000000000 +0000 @@ -189,7 +189,7 @@ instance=instance) @utils.synchronized('connect_qb_volume') - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Disconnect the volume.""" mount_path = self._get_mount_path(connection_info) diff -Nru nova-26.1.0/nova/virt/libvirt/volume/scaleio.py nova-26.2.2/nova/virt/libvirt/volume/scaleio.py --- nova-26.1.0/nova/virt/libvirt/volume/scaleio.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/scaleio.py 2024-01-16 10:19:41.000000000 +0000 @@ -57,12 +57,13 @@ instance=instance) connection_info['data']['device_path'] = device_info['path'] - def disconnect_volume(self, connection_info, instance): - self.connector.disconnect_volume(connection_info['data'], None) + def disconnect_volume(self, connection_info, instance, force=False): + self.connector.disconnect_volume( + connection_info['data'], None, force=force) LOG.debug("Disconnected volume", instance=instance) super(LibvirtScaleIOVolumeDriver, self).disconnect_volume( - connection_info, instance) + connection_info, instance, force=force) def extend_volume(self, connection_info, instance, requested_size): LOG.debug("calling os-brick to extend ScaleIO Volume", diff -Nru nova-26.1.0/nova/virt/libvirt/volume/smbfs.py nova-26.2.2/nova/virt/libvirt/volume/smbfs.py --- nova-26.1.0/nova/virt/libvirt/volume/smbfs.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/smbfs.py 2024-01-16 10:19:41.000000000 +0000 @@ -52,7 +52,7 @@ device_path = self._get_device_path(connection_info) connection_info['data']['device_path'] = device_path - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Disconnect the volume.""" smbfs_share = connection_info['data']['export'] mount_path = self._get_mount_path(connection_info) diff -Nru nova-26.1.0/nova/virt/libvirt/volume/storpool.py nova-26.2.2/nova/virt/libvirt/volume/storpool.py --- nova-26.1.0/nova/virt/libvirt/volume/storpool.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/storpool.py 2024-01-16 10:19:41.000000000 +0000 @@ -47,10 +47,11 @@ device_info, instance=instance) connection_info['data']['device_path'] = device_info['path'] - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): LOG.debug("Detaching StorPool volume %s", connection_info['data']['volume'], instance=instance) - self.connector.disconnect_volume(connection_info['data'], None) + self.connector.disconnect_volume( + connection_info['data'], None, force=force) LOG.debug("Detached StorPool volume", instance=instance) def extend_volume(self, connection_info, instance, requested_size): diff -Nru nova-26.1.0/nova/virt/libvirt/volume/volume.py nova-26.2.2/nova/virt/libvirt/volume/volume.py --- nova-26.1.0/nova/virt/libvirt/volume/volume.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/volume.py 2024-01-16 10:19:41.000000000 +0000 @@ -135,7 +135,7 @@ """Connect the volume.""" pass - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Disconnect the volume.""" pass diff -Nru nova-26.1.0/nova/virt/libvirt/volume/vzstorage.py nova-26.2.2/nova/virt/libvirt/volume/vzstorage.py --- nova-26.1.0/nova/virt/libvirt/volume/vzstorage.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/libvirt/volume/vzstorage.py 2024-01-16 10:19:41.000000000 +0000 @@ -126,9 +126,10 @@ return _connect_volume(connection_info, instance) - def disconnect_volume(self, connection_info, instance): + def disconnect_volume(self, connection_info, instance, force=False): """Detach the volume from instance_name.""" LOG.debug("calling os-brick to detach Vzstorage Volume", instance=instance) - self.connector.disconnect_volume(connection_info['data'], None) + self.connector.disconnect_volume( + connection_info['data'], None, force=force) LOG.debug("Disconnected Vzstorage Volume", instance=instance) diff -Nru nova-26.1.0/nova/virt/netutils.py nova-26.2.2/nova/virt/netutils.py --- nova-26.1.0/nova/virt/netutils.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/virt/netutils.py 2024-01-16 10:19:41.000000000 +0000 @@ -263,12 +263,19 @@ 'id': link_id, 'vif_id': vif['id'], 'type': nic_type, - 'mtu': vif['network']['meta'].get('mtu'), + 'mtu': _get_link_mtu(vif), 'ethernet_mac_address': vif.get('address'), } return link +def _get_link_mtu(vif): + for subnet in vif['network']['subnets']: + if subnet['meta'].get('dhcp_server'): + return None + return vif['network']['meta'].get('mtu') + + def _get_nets(vif, subnet, version, net_num, link_id): """Get networks for the given VIF and subnet diff -Nru nova-26.1.0/nova/volume/cinder.py nova-26.2.2/nova/volume/cinder.py --- nova-26.1.0/nova/volume/cinder.py 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/nova/volume/cinder.py 2024-01-16 10:19:41.000000000 +0000 @@ -91,12 +91,14 @@ # from them generated from 'context.get_admin_context' # which only set is_admin=True but is without token. # So add load_auth_plugin when this condition appear. + user_auth = None if context.is_admin and not context.auth_token: if not _ADMIN_AUTH: _ADMIN_AUTH = _load_auth_plugin(CONF) - return _ADMIN_AUTH - else: - return service_auth.get_auth_plugin(context) + user_auth = _ADMIN_AUTH + + # When user_auth = None, user_auth will be extracted from the context. + return service_auth.get_auth_plugin(context, user_auth=user_auth) # NOTE(efried): Bug #1752152 diff -Nru nova-26.1.0/releasenotes/notes/Do-not-send-mtu-value-in-metadata-for-networks-with-enabled-dhcp-641506f2a13b540f.yaml nova-26.2.2/releasenotes/notes/Do-not-send-mtu-value-in-metadata-for-networks-with-enabled-dhcp-641506f2a13b540f.yaml --- nova-26.1.0/releasenotes/notes/Do-not-send-mtu-value-in-metadata-for-networks-with-enabled-dhcp-641506f2a13b540f.yaml 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/releasenotes/notes/Do-not-send-mtu-value-in-metadata-for-networks-with-enabled-dhcp-641506f2a13b540f.yaml 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,5 @@ +--- +other: + - | + For networks which have any subnets with enabled DHCP, MTU value is not send + in the metadata. In such case MTU is configured through the DHCP server. diff -Nru nova-26.1.0/releasenotes/notes/rescue-volume-based-instance-c6e3fba236d90be7.yaml nova-26.2.2/releasenotes/notes/rescue-volume-based-instance-c6e3fba236d90be7.yaml --- nova-26.1.0/releasenotes/notes/rescue-volume-based-instance-c6e3fba236d90be7.yaml 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/releasenotes/notes/rescue-volume-based-instance-c6e3fba236d90be7.yaml 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,6 @@ +--- +fixes: + - | + Fix rescuing volume based instance by adding a check for 'hw_rescue_disk' + and 'hw_rescue_device' properties in image metadata before attempting + to rescue instance. diff -Nru nova-26.1.0/releasenotes/notes/service-user-token-421d067c16257782.yaml nova-26.2.2/releasenotes/notes/service-user-token-421d067c16257782.yaml --- nova-26.1.0/releasenotes/notes/service-user-token-421d067c16257782.yaml 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/releasenotes/notes/service-user-token-421d067c16257782.yaml 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,11 @@ +upgrade: + - | + Configuration of service user tokens is now **required** for all Nova services + to ensure security of block-storage volume data. + + All Nova configuration files must configure the ``[service_user]`` section as + described in the `documentation`__. + + See https://bugs.launchpad.net/nova/+bug/2004555 for more details. + + __ https://docs.openstack.org/nova/latest/admin/configuration/service-user-token.html diff -Nru nova-26.1.0/releasenotes/notes/translate_vf_network_capabilities_to_port_binding-48abbfe0ce2923cf.yaml nova-26.2.2/releasenotes/notes/translate_vf_network_capabilities_to_port_binding-48abbfe0ce2923cf.yaml --- nova-26.1.0/releasenotes/notes/translate_vf_network_capabilities_to_port_binding-48abbfe0ce2923cf.yaml 1970-01-01 00:00:00.000000000 +0000 +++ nova-26.2.2/releasenotes/notes/translate_vf_network_capabilities_to_port_binding-48abbfe0ce2923cf.yaml 2024-01-16 10:19:41.000000000 +0000 @@ -0,0 +1,16 @@ +--- +fixes: + - | + Previously ``switchdev`` capabilities should be configured manually by a + user with admin privileges using port's binding profile. This blocked + regular users from managing ports with Open vSwitch hardware offloading + as providing write access to a port's binding profile to non-admin users + introduces security risks. For example, a binding profile may contain a + ``pci_slot`` definition, which denotes the host PCI address of the + device attached to the VM. A malicious user can use this parameter to + passthrough any host device to a guest, so it is impossible to provide + write access to a binding profile to regular users in many scenarios. + + This patch fixes this situation by translating VF capabilities reported + by Libvirt to Neutron port binding profiles. Other VF capabilities are + translated as well for possible future use. diff -Nru nova-26.1.0/tools/check-cherry-picks.sh nova-26.2.2/tools/check-cherry-picks.sh --- nova-26.1.0/tools/check-cherry-picks.sh 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/tools/check-cherry-picks.sh 2024-01-16 10:19:41.000000000 +0000 @@ -23,7 +23,7 @@ checked=0 branches+="" for hash in $hashes; do - branch=$(git branch -a --contains "$hash" 2>/dev/null| grep -oE '(master|stable/[a-z]+)') + branch=$(git branch -a --contains "$hash" 2>/dev/null| grep -oE '(master|stable/[a-z0-9.]+)') if [ $? -ne 0 ]; then echo "Cherry pick hash $hash not on any master or stable branches" exit 1 diff -Nru nova-26.1.0/.zuul.yaml nova-26.2.2/.zuul.yaml --- nova-26.1.0/.zuul.yaml 2023-01-25 16:07:06.000000000 +0000 +++ nova-26.2.2/.zuul.yaml 2024-01-16 10:19:41.000000000 +0000 @@ -262,8 +262,6 @@ NOVA_BACKEND: LVM # Do not waste time clearing volumes. LVM_VOLUME_CLEAR: none - # Disable SSH validation in tests to save time. - TEMPEST_RUN_VALIDATION: false # Increase the size of the swift loopback device to accommodate RAW # snapshots from the LV based instance disks. # See bug #1913451 for more details.