Sorry we missed you! Our docs have had a" diff -Nru cloud-init-23.3.3/doc/rtd/development/code_review.rst cloud-init-23.4.3/doc/rtd/development/code_review.rst --- cloud-init-23.3.3/doc/rtd/development/code_review.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/code_review.rst 2024-02-02 21:21:52.000000000 +0000 @@ -3,259 +3,237 @@ Code review process ******************* -In order to manage incoming pull requests effectively, and provide -timely feedback and/or acceptance, this document serves as a guideline -for the review process. It outlines the expectations for those -submitting code to the project as well as those reviewing the code. - Code is reviewed for acceptance by at least one core team member (later referred to as committers), but comments and suggestions from others are encouraged and welcome. -The process is intended to provide timely and actionable feedback for -any submission. +Goals +===== + +This process aims to: + +* provide timely and actionable feedback on every submission, +* make sure incoming PRs are handled efficiently, and +* get PRs accepted within a reasonable time frame. + +.. _asking-for-help: Asking for help =============== -``cloud-init`` contributors, potential contributors, community members and -users are encouraged to ask for any help that they need. If you have -questions about the code review process, or at any point during the -code review process, these are the available avenues: - -* if you have an open Pull Request, comment on that pull request, -* join the ``#cloud-init`` `channel on the Libera IRC`_ network, +Cloud-init contributors, community members and users are encouraged to ask for +help if they need it. If you have questions about the code review process, or +need advice on an open PR, these are the available avenues: + +* Open a PR, add "WIP:" to the title, and leave a comment on that PR +* join the ``#cloud-init`` `channel on the Libera IRC `_ network +* post on the ``#cloud-init`` `Discourse topic `_ * send an email to the cloud-init mailing list: :: cloud-init@lists.launchpad.net -These are listed in order of preference, but please use whichever of them you -are most comfortable with. +These are listed in order of our preference, but please use whichever of them +you are most comfortable with. -Goals -===== +Role definitions +================ -This process has the following goals: +There are three roles involved in code reviews: -* To ensure code reviews occur in a timely fashion and provide actionable - feedback if changes are desired. -* To ensure the minimisation of ancillary problems to increase the - efficiency for those reviewing the submitted code. +* **Proposer** -Role definitions -================ + The person(s) submitting the PR + +* **Reviewer** -Any code review process will have (at least) two involved parties. For -our purposes, these parties are referred to as **Proposer** and -**Reviewer**. We also have the **Committer** role which is a special -case of the **Reviewer** role. - -The terms are defined here (and the use of the singular form is not meant to -imply that they refer to a single person): - -**Proposer** - The person proposing a pull request (hereafter known as a PR). - -**Reviewer** - A person who is reviewing a PR. - -**Committer** - A cloud-init core developer (i.e., a person who has permission to - merge PRs into ``main``). - -Prerequisites for landing pull requests -======================================= - -Before a PR can be landed into ``main``, the following conditions *must* -be met: - -* the CLA has been signed by the **Proposer** (or is covered by an - entity-level CLA signature), -* all required status checks are passing, -* at least one "Approve" review has been received from a **Committer**, and -* no "Request changes" reviews from any **Committer** are outstanding. - -The following conditions *should* be met: - -* any Python functions/methods/classes have docstrings added/updated, -* any changes to config module behaviour are captured in the documentation of - the config module, -* any Python code added has corresponding unit tests, and -* no "Request changes" reviews from any **Reviewer** are outstanding. - -These conditions can be relaxed at the discretion of the -**Committers** on a case-by-case basis. Generally, for accountability, -this should not be the decision of a single **Committer**, and the -decision should be documented in comments on the PR. - -(To take a specific example, the ``cc_phone_home`` module had no tests -at the time `PR #237`_ was submitted, so the **Proposer** was not expected to -write a full set of tests for their minor modification, but they were expected -to update the config module docs.) + A person who is reviewing the PR -Non-Committer reviews +* **Committer** + + A cloud-init core developer (i.e., someone with permission to merge PRs + into ``main``) + +.. _PR-acceptance: + +PR acceptance conditions +======================== + +Before a PR can be accepted and merged into ``main`` ("landed"), the following +conditions **must** be met: + +* The `CLA`_ must be signed **by the proposer** (unless the proposer is + covered by an entity-level CLA signature), +* All required status checks must be passing, +* At least one "Approve" review must be given **by a committer**, and +* No "Request changes" reviews from a committer can be outstanding. + +The following conditions **should** be met: + +* Any Python functions/methods/classes have docstrings added/updated, +* Any changes to config module behaviour are captured in that module's + documentation, +* Any Python code added has corresponding :ref:`unit tests`, and +* No "Request changes" reviews from any **reviewer** are outstanding. + +These conditions can be relaxed at the discretion of the committers on a +case-by-case basis. For accountability, this should not be the decision of a +single committer, and the decision should be documented in comments on the +PR. + +To take a specific example, the ``cc_phone_home`` module had no tests +at the time `PR #237`_ was submitted, so the **proposer** was not expected to +write a full set of tests for their minor modification, but they *were* +expected to update the config module docs. + +Non-committer reviews ===================== -Reviews from non-**Committers** are *always* welcome. Please feel empowered to -review PRs and leave your thoughts and comments on any submitted PRs, -regardless of the **Proposer**. - -Much of the below process is written in terms of the **Committers**. This is -not intended to reflect that reviews should only come from that group, but -rather an acknowledgement that we are ultimately responsible for maintaining -the standards of the codebase. It would be entirely reasonable (and very -welcome) for a **Reviewer** to only examine part of a PR, but it would not be -appropriate for a **Committer** to merge a PR without full scrutiny. +Reviews from non-committer reviewers are *always* welcome. Please feel +empowered to review PRs and leave your thoughts and comments on any submitted +PR, regardless of the proposer. + +Much of the below process is written in terms of the **committers**. This does +not mean that reviews should only come from that group, but rather acknowledges +that we are ultimately responsible for maintaining the standards of the +codebase. It is reasonable (and very welcome) for a reviewer to only examine +part of a PR, but a committer must not merge a PR without full scrutiny. Opening phase ============= -In this phase, the **Proposer** is responsible for opening a pull request and -meeting the prerequisites laid out above. +Proposer opens a PR +------------------- -If they need help understanding the prerequisites, or help meeting the -prerequisites, then they can (and should!) ask for help. See the -`Asking For Help`_ section above for the ways to do that. +In this phase, the proposer opens a pull request and needs to ensure they meet +the criteria laid out above. If they need help understanding or meeting these +criteria, then they can (and should!) ask for help. -These are the steps that comprise the opening phase: +CI runs automatically +--------------------- -1. The **Proposer** opens a PR +* If CI fails: -2. CI runs automatically, and if: + The **proposer** is expected to fix CI failures. If they don't understand the + failures, they should comment on the PR to ask for help (or use another way + of :ref:`asking-for-help`). If they don't ask for help, the + committers will assume the proposer is working on addressing the failures. - CI fails: - The **Proposer** is expected to fix CI failures. If the - **Proposer** doesn't understand the nature of the failures they - are seeing, they should comment in the PR to request assistance, - or use another way of `Asking For Help`_. +* If CI passes: - (Note that if assistance is not requested, the **Committers** - will assume that the **Proposer** is working on addressing the - failures themselves. If you require assistance, please do ask - for help!) - - CI passes: - Move on to the `Review phase`_. + Move on to the **review phase**. Review phase ============ -In this phase, the **Proposer** and the **Reviewers** will iterate -together to, hopefully, get the PR merged into the ``cloud-init`` codebase. -There are three potential outcomes: merged, rejected permanently, and -temporarily closed. The first two are covered in this section; see -`Inactive Pull Requests`_ for details about temporary closure. - -In this section, when the verbs "merge" or "squash merge" are used, they -should be understood to mean "squash merged using the GitHub UI", which -is the only way that changes can land in ``cloud-init``'s ``main`` branch. - -These are the steps that comprise the review phase: - -1. **The Committers** assign a **Committer** to the PR: - - This **Committer** is expected to shepherd the PR to completion (and to - merge it, if that is the outcome reached). This means that they - will perform an initial review, and monitor the PR to ensure that - the **Proposer** is receiving any assistance that they require. The - **Committers** will perform this assignment on a daily basis. - - This assignment is intended to ensure that the **Proposer** has a - clear point of contact with a ``cloud-init`` core developer, and that - they get timely feedback after submitting a PR. It *is not* - intended to preclude reviews from any other **Reviewers**, nor to - imply that the **Committer** has ownership over the review process. - - The assigned **Committer** may choose to delegate the code review of - a PR to another **Reviewer** if they think that they would be better - suited. - - (Note that, in GitHub terms, this is setting an `Assignee`, not - requesting a review.) - -2. That **Committer** performs an initial review of the PR, resulting - in one of the following: - - Approve: - If the submitted PR meets all of the `Prerequisites for - Landing Pull Requests`_ and passes code review, then the - **Committer** will squash merge immediately. - - There may be circumstances where a PR should not be merged - immediately. The :guilabel:`wip` label will be applied to PRs for which - this is true. Only **Committers** are able to apply labels to - PRs, so anyone who believes that this label should be applied to a - PR should request its application in a comment on the PR. - - The review process is **DONE**. - - Approve (with nits): - If the **Proposer** submits their PR with :guilabel:`"Allow edits from - maintainer"` enabled, and the only changes the **Committer** - requests are minor "nits", the **Committer** can push fixes for - those nits and *immediately* squash merge. If the **Committer** - does not wish to fix these nits but believes they should block a - straightforward :guilabel:`Approve`, then their review should be "Needs - Changes" instead. - - A nit is understood to be something like a minor style issue or a - spelling error, generally confined to a single line of code. - - If a **Committer** is unsure as to whether their requested change - is a nit, they should not treat it as a nit. - - (If a **Proposer** wants to opt-out of this, then they should - uncheck :guilabel:`"Allow edits from maintainer"` when submitting their - PR.) - - The review process is **DONE**. - - Outright rejection: - The **Committer** will close the PR, with useful messaging for the - **Proposer** as to why this has happened. - - This is reserved for cases where the proposed change is completely - unfit for landing, and there is no reasonable path forward. This - should only be used sparingly, as there are very few cases where - proposals are completely unfit. - - If a different approach to the same problem is planned, it should - be submitted as a separate PR. The **Committer** should include - this information in their message when the PR is closed. - - The review process is **DONE**. - - Needs Changes: - The **Committer** will give the **Proposer** a clear idea of what - is required for an :guilabel:`Approve` vote or, for more complex PRs, - what the next steps towards an :guilabel:`Approve` vote are. - - The **Proposer** will ask questions if they don't understand, or - disagree with, the **Committer**'s review comments. - - Once consensus has been reached, the **Proposer** will address the - review comments. - - Once the review comments are addressed (as well as, potentially, - in the interim), CI will run. If CI fails, the **Proposer** is - expected to fix CI failures. If CI passes, the **Proposer** - should indicate that the PR is ready for re-review (by @ing the - assigned reviewer), effectively moving back to the start of this - section. +In this phase, the **proposer** and the **reviewers** will work iteratively +together to get the PR merged into the cloud-init codebase. + +There are three potential outcomes: **merged**, **rejected permanently**, and +**temporarily closed**. The first two are covered in this section; see +the :ref:`inactive pull requests` section for details about +temporary closure. + +A committer is assigned +----------------------- + +The committers assign a committer to the PR. This committer is +expected to shepherd the PR to completion (and to merge it, if that is the +outcome reached). + +They perform an initial review, and monitor the PR to ensure the proposer is +receiving help if they need it. The committers perform this assignment on a +regular basis for any new PRs submitted. + +Committer's initial review +-------------------------- + +The assigned committer performs an initial review of the PR, resulting in one +of the following. + +Approve +~~~~~~~ + +If the submitted PR meets all of the +:ref:`PR acceptance conditions` and passes code review, then the +committer will squash merge immediately. + +Sometimes, a PR should not be merged immediately. The :guilabel:`wip` label +will be applied to PRs for which this is true. Only committers are able to +apply labels to PRs, so anyone who thinks this label should be applied to a +PR should request it in a comment on the PR. + +- The review process is **DONE**. + +Approve (with nits) +~~~~~~~~~~~~~~~~~~~ + +A "nit" is understood to be something like a minor style issue or a spelling +error, generally confined to a single line of code. + +If the proposer submits their PR with :guilabel:`"Allow edits from maintainer"` +enabled, and the only changes the committer requests are minor nits, the +committer can push fixes for those nits and immediately squash merge. + +If the committer does not wish to fix these nits but believes they should +block a straightforward `Approve`, then their review should be +`Needs Changes` instead. + +If a committer is unsure whether their requested change is a nit, they should +not treat it as a nit. + +If a proposer wants to opt-out of this, they should uncheck +:guilabel:`"Allow edits from maintainer"` when submitting their PR. + +- The review process is **DONE**. + +Outright rejection +~~~~~~~~~~~~~~~~~~ + +The committer will close the PR with a message for the proposer to explain why. + +This is reserved for cases where the proposed change is unfit for landing and +there is no reasonable path forward. This should only be used sparingly, as +there are very few cases where proposals are *completely* unfit. + +If a different approach to the same problem is planned, it should be +submitted as a separate PR. The committer should include this information in +their message when the PR is closed. + +- The review process is **DONE**. + +Needs Changes +~~~~~~~~~~~~~ + +The committer will give the proposer clear feedback on what is needed for an +`Approve` vote or, for more complex PRs, what the next steps towards an +`Approve` vote are. + +The proposer can ask questions if they don't understand, or disagree with, the +committer's review comments. + +Once agreement has been reached, the proposer will address the review comments. + +Once the review comments are addressed, CI will run. If CI fails, the proposer +is expected to fix any CI failures. If CI passes, the proposer should indicate +that the PR is ready for re-review (by `@` mentioning the assigned reviewer), +effectively moving back to the start of the `Review phase`. + +.. _inactive-PRs: Inactive pull requests ====================== -PRs will be temporarily closed if they have been waiting on -**Proposer** action for a certain amount of time without activity. A -PR will be marked as stale (with an explanatory comment) after 14 days -of inactivity. It will be closed after a further 7 days of inactivity. - -These closes are not considered permanent, and the closing message -should reflect this for the **Proposer**. However, if a PR is reopened, -it should effectively enter the `Opening phase`_ again, as it may -need some work done to get CI passing again. +PRs will be temporarily closed if they have been waiting on proposer action for +a certain amount of time without activity. A PR will be marked as **stale** +(with an explanatory comment) after 14 days of inactivity. + +It will be closed after a further 7 days of inactivity. + +These closes are not considered permanent, and the closing message should +reflect this for the proposer. However, if a PR is re-opened, it should +effectively re-enter the `Opening phase`, as it may need some work done +to get CI passing again. .. LINKS: -.. _channel on the Libera IRC: https://kiwiirc.com/nextclient/irc.libera.chat/cloud-init +.. include:: ../links.txt .. _PR #237: https://github.com/canonical/cloud-init/pull/237 diff -Nru cloud-init-23.3.3/doc/rtd/development/contribute_code.rst cloud-init-23.4.3/doc/rtd/development/contribute_code.rst --- cloud-init-23.3.3/doc/rtd/development/contribute_code.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/contribute_code.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,114 @@ +Contribute to the code +********************** + +For a run-through of the entire process, the following pages will be your best +starting point: + +* :doc:`Find issues to work on` +* :doc:`Build your first pull request` +* :doc:`Our code review process` + +On the rest of this page you'll find the key resources you'll need to start +contributing to the cloud-init codebase. + +Testing +======= + +Submissions to cloud-init must include testing. Unit testing and integration +testing are integral parts of contributing code. + +.. toctree:: + :maxdepth: 1 + :hidden: + + testing.rst + integration_tests.rst + +* :doc:`Unit testing overview and design principles` +* :doc:`Integration testing` + +Popular contributions +===================== + +.. toctree:: + :maxdepth: 1 + :hidden: + + module_creation.rst + datasource_creation.rst + +The two most popular contributions we receive are new cloud config +:doc:`modules ` and new +:doc:`datasources `; these pages will provide instructions +on how to create them. + +Note that any new modules should use underscores in any new config options and +not hyphens (e.g. ``new_option`` and *not* ``new-option``). + +Code style and design +===================== + +We generally adhere to `PEP 8`_, and this is enforced by our use of ``black``, +``isort`` and ``ruff``. + +Python support +-------------- + +Cloud-init upstream currently supports Python 3.6 and above. + +Cloud-init upstream will stay compatible with a particular Python version for 6 +years after release. After 6 years, we will stop testing upstream changes +against the unsupported version of Python and may introduce breaking changes. +This policy may change as needed. + +The following table lists the cloud-init versions in which the minimum Python +version changed: + +.. list-table:: + :header-rows: 1 + :align: center + + * - Cloud-init version + - Python version + * - 22.1 + - 3.6+ + * - 20.3 + - 3.5+ + * - 19.4 + - 2.7+ + +Type annotations +---------------- + +The cloud-init codebase uses Python's annotation support for storing type +annotations in the style specified by `PEP-484`_ and `PEP-526`_. Their use in +the codebase is encouraged. + +Other resources +=============== + +.. toctree:: + :maxdepth: 1 + :hidden: + + dir_layout.rst + +* :doc:`Explanation of the directory structure` + +Feature flags +------------- + +.. automodule:: cloudinit.features + :members: + +.. LINKS: +.. include:: ../links.txt +.. _quickstart documentation: https://docs.github.com/en/get-started/quickstart +.. _tools/.github-cla-signers: https://github.com/canonical/cloud-init/blob/main/tools/.github-cla-signers +.. _repository: https://github.com/canonical/cloud-init +.. _contributor-agreement-canonical: https://launchpad.net/%7Econtributor-agreement-canonical/+members +.. _PR #344: https://github.com/canonical/cloud-init/pull/344 +.. _PR #345: https://github.com/canonical/cloud-init/pull/345 +.. _PEP-484: https://www.python.org/dev/peps/pep-0484/ +.. _PEP-526: https://www.python.org/dev/peps/pep-0526/ +.. _PEP 8: https://peps.python.org/pep-0008/ diff -Nru cloud-init-23.3.3/doc/rtd/development/contribute_docs.rst cloud-init-23.4.3/doc/rtd/development/contribute_docs.rst --- cloud-init-23.3.3/doc/rtd/development/contribute_docs.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/contribute_docs.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,70 @@ +.. _docs: + +Contribute to our docs +********************** + +.. toctree:: + :maxdepth: 1 + :hidden: + + Style guide + Directory layout + +The documentation for cloud-init is hosted in GitHub and rendered on +`Read the Docs`_. It is mostly written in reStructuredText. + +The process for contributing to the docs is largely the same as for code, +except that for cosmetic changes to the documentation (spelling, grammar, etc) +you can also use the GitHub web interface to submit changes as quick PRs. + +Previewing the docs +=================== + +The documentation for submitted/active PRs is automatically built by Read the +Docs and served from the PR's "conversation" tab as an automatic check. + +However, while you are working on docs for a feature you are adding, you will +most likely want to build the docs locally. There is a Makefile target to build +the documentation for you: + +.. code-block:: shell-session + + $ tox -e doc + +This will do two things: + +- Build the documentation using Sphinx. +- Run doc8 against the documentation source code. + +Once built, the HTML files will be viewable in `doc/rtd_html`. Use your +web browser to open `index.html` to view and navigate the site. + +How are the docs structured? +============================ + +We use `Diataxis`_ to organise our documentation. There is more detail on the +layout of the ``doc`` directory in the :doc:`docs_layout` article. + +We also have a :doc:`style_docs` that will help you if you need to edit or +write any content. + +In your first PR +================= + +You will need to add your GitHub username (alphabetically) to the in-repository +list that we use to track :ref:`CLA signatures `: +`tools/.github-cla-signers`_. + +Please include this in the same PR alongside your first contribution. Do +not create a separate PR to add your name to the CLA signatures. + +If you need some help with your contribution, you can contact us on our +`IRC channel `_. If you have already submitted a work-in-progress PR, you +can also ask for guidance from our technical author by `tagging s-makin`_ as a +reviewer. + +.. LINKS +.. include:: ../links.txt +.. _Read the Docs: https://readthedocs.com/ +.. _tools/.github-cla-signers: https://github.com/canonical/cloud-init/blob/main/tools/.github-cla-signers +.. _tagging s-makin: https://github.com/s-makin diff -Nru cloud-init-23.3.3/doc/rtd/development/contributing.rst cloud-init-23.4.3/doc/rtd/development/contributing.rst --- cloud-init-23.3.3/doc/rtd/development/contributing.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/contributing.rst 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ -.. include:: ../../../CONTRIBUTING.rst diff -Nru cloud-init-23.3.3/doc/rtd/development/debugging.rst cloud-init-23.4.3/doc/rtd/development/debugging.rst --- cloud-init-23.3.3/doc/rtd/development/debugging.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/debugging.rst 2024-02-02 21:21:52.000000000 +0000 @@ -1,13 +1,13 @@ .. _debugging: -Debugging ``cloud-init`` -************************ +Debugging cloud-init +******************** Overview ======== -This topic will discuss general approaches for testing and debugging -``cloud-init`` on deployed instances. +This topic will discuss general approaches for testing and debugging cloud-init +on deployed instances. .. _boot_time_analysis: @@ -18,13 +18,13 @@ ----------------------------- Occasionally, instances don't appear as performant as we would like and -``cloud-init`` packages a simple facility to inspect which operations took the +cloud-init packages a simple facility to inspect which operations took the longest during boot and setup. The script :file:`/usr/bin/cloud-init` has an analysis sub-command, :command:`analyze`, which parses any :file:`cloud-init.log` file into formatted and sorted events. It allows for detailed analysis of the most costly -``cloud-init`` operations, and to determine the long-pole in ``cloud-init`` +cloud-init operations, and to determine the long-pole in cloud-init configuration and setup. These subcommands default to reading :file:`/var/log/cloud-init.log`. @@ -81,7 +81,7 @@ ^^^^^^^^^^^^^^^^^^^^^^^^ Parse :file:`cloud-init.log` into event records and sort them based on the -highest time cost for a quick assessment of areas of ``cloud-init`` that may +highest time cost for a quick assessment of areas of cloud-init that may need improvement. .. code-block:: shell-session @@ -101,8 +101,8 @@ :command:`analyze boot` ^^^^^^^^^^^^^^^^^^^^^^^ -Make subprocess calls to the kernel in order to get relevant pre-``cloud-init`` -timestamps, such as the kernel start, kernel finish boot, and ``cloud-init`` +Make subprocess calls to the kernel in order to get relevant pre-cloud-init +timestamps, such as the kernel start, kernel finish boot, and cloud-init start. .. code-block:: shell-session @@ -123,7 +123,7 @@ Analyze quickstart - LXC ------------------------ -To quickly obtain a ``cloud-init`` log, try using :command:``lxc`` on any +To quickly obtain a cloud-init log, try using :command:`lxc` on any Ubuntu system: .. code-block:: shell-session @@ -138,7 +138,7 @@ Analyze quickstart - KVM ------------------------ -To quickly analyze a KVM ``cloud-init`` log: +To quickly analyze a KVM cloud-init log: 1. Download the current cloud image @@ -188,7 +188,7 @@ the cached user data and metadata after the instance has booted. Each cloud-config module has a module ``FREQUENCY`` configured: ``PER_INSTANCE``, ``PER_BOOT``, ``PER_ONCE`` or ``PER_ALWAYS``. When a module is run by -``cloud-init``, it stores a semaphore file in +cloud-init, it stores a semaphore file in :file:`/var/lib/cloud/instance/sem/config_.` which marks when the module last successfully ran. Presence of this semaphore file prevents a module from running again if it has already been run. To ensure that @@ -212,37 +212,37 @@ .. _proposed_sru_testing: -Stable Release Updates (SRU) testing for ``cloud-init`` -======================================================= +Stable Release Updates (SRU) testing for cloud-init +=================================================== Once an Ubuntu release is stable (i.e. after it is released), updates for it must follow a special procedure called a "Stable Release Update" (`SRU`_). -The ``cloud-init`` project has a specific process it follows when validating -a ``cloud-init`` SRU, documented in the `CloudinitUpdates`_ wiki page. +The cloud-init project has a specific process it follows when validating +a cloud-init SRU, documented in the `CloudinitUpdates`_ wiki page. -Generally an SRU test of ``cloud-init`` performs the following: +Generally an SRU test of cloud-init performs the following: - * Install a pre-release version of ``cloud-init`` from the **-proposed** APT +* Install a pre-release version of cloud-init from the **-proposed** APT pocket (e.g., **bionic-proposed**). - * Upgrade ``cloud-init`` and attempt a clean run of ``cloud-init`` to assert +* Upgrade cloud-init and attempt a clean run of cloud-init to assert that the new version works properly on the specific platform and Ubuntu series. - * Check for tracebacks or errors in behaviour. +* Check for tracebacks or errors in behaviour. Manual SRU verification procedure --------------------------------- -Below are steps to manually test a pre-release version of ``cloud-init`` +Below are steps to manually test a pre-release version of cloud-init from **-proposed** .. note:: For each Ubuntu SRU, the Ubuntu Server team manually validates the new - version of ``cloud-init`` on these platforms: **Amazon EC2, Azure, GCE, + version of cloud-init on these platforms: **Amazon EC2, Azure, GCE, OpenStack, Oracle, Softlayer (IBM), LXD, KVM** 1. Launch a VM on your favorite platform, providing this cloud-config - user data and replacing `` with your username: + user data and replacing ```` with your username: .. code-block:: yaml @@ -251,9 +251,9 @@ ssh_import_id: [] hostname: SRU-worked-{{v1.cloud_name}} -2. Wait for current ``cloud-init`` to complete, replace ```` with +2. Wait for current cloud-init to complete, replace ```` with the IP address of the VM that you launched in step 1. Be sure to make a - note of the datasource ``cloud-init`` detected in ``--long`` output. You + note of the datasource cloud-init detected in ``--long`` output. You will need this during step 5, where you will use it to confirm the same datasource is detected after the upgrade: @@ -263,8 +263,8 @@ $ ssh ubuntu@$CI_VM_IP -- cloud-init status --wait --long 3. Set up the **-proposed** pocket on your VM and upgrade to the **-proposed** - ``cloud-init``. To do this, create the following bash script, which will - add the **-proposed** pocket to APT's sources and install ``cloud-init`` + cloud-init. To do this, create the following bash script, which will + add the **-proposed** pocket to APT's sources and install cloud-init from that pocket: .. code-block:: bash @@ -283,16 +283,16 @@ $ scp setup_proposed.sh ubuntu@$CI_VM_IP:. $ ssh ubuntu@$CI_VM_IP -- sudo bash setup_proposed.sh -4. Change hostname, clean ``cloud-init``'s state, and reboot to run - ``cloud-init`` from scratch: +4. Change hostname, clean cloud-init's state, and reboot to run cloud-init from + scratch: .. code-block:: shell-session $ ssh ubuntu@$CI_VM_IP -- sudo hostname something-else $ ssh ubuntu@$CI_VM_IP -- sudo cloud-init clean --logs --reboot -5. Validate **-proposed** ``cloud-init`` came up without error. First, we block - until ``cloud-init`` completes, then verify from ``--long`` that the +5. Validate **-proposed** cloud-init came up without error. First, we block + until cloud-init completes, then verify from ``--long`` that the datasource is the same as the one picked up from step 1. Errors will show up in ``--long``: @@ -306,7 +306,7 @@ $ ssh ubuntu@$CI_VM_IP -- hostname -Then, check for any errors or warnings in ``cloud-init`` logs. If successful, +Then, check for any errors or warnings in cloud-init logs. If successful, this will produce no output: .. code-block:: shell-session @@ -315,10 +315,12 @@ 6. If you encounter an error during SRU testing: - * Create a `new cloud-init bug`_ reporting the version of ``cloud-init`` + * Create a `new cloud-init bug`_ reporting the version of cloud-init affected - * Ping upstream ``cloud-init`` on Libera's `#cloud-init IRC channel`_ + * Ping upstream cloud-init on Libera's `#cloud-init IRC channel `_ +.. LINKS +.. include:: ../links.txt .. _SRU: https://wiki.ubuntu.com/StableReleaseUpdates .. _CloudinitUpdates: https://wiki.ubuntu.com/CloudinitUpdates .. _new cloud-init bug: https://github.com/canonical/cloud-init/issues diff -Nru cloud-init-23.3.3/doc/rtd/development/dir_layout.rst cloud-init-23.4.3/doc/rtd/development/dir_layout.rst --- cloud-init-23.3.3/doc/rtd/development/dir_layout.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/dir_layout.rst 2024-02-02 21:21:52.000000000 +0000 @@ -3,85 +3,62 @@ Directory layout **************** -``Cloud-init``'s directory structure is somewhat different from a regular -application: :: - - /var/lib/cloud/ - - data/ - - instance-id - - previous-instance-id - - datasource - - previous-datasource - - previous-hostname - - handlers/ - - instance - - instances/ - i-00000XYZ/ - - boot-finished - - cloud-config.txt - - datasource - - handlers/ - - obj.pkl - - scripts/ - - sem/ - - user-data.txt - - user-data.txt.i - - scripts/ - - per-boot/ - - per-instance/ - - per-once/ - - seed/ - - sem/ - ``/var/lib/cloud`` +================== - The main directory containing the ``cloud-init``-specific subdirectories. - It is typically located at :file:`/var/lib` but there are certain - configuration scenarios where this can be altered. +The main directory containing the cloud-init-specific subdirectories. It is +typically located at :file:`/var/lib` but there are certain configuration +scenarios where this can be changed. .. TODO: expand this section -``data/`` - - Contains information related to instance IDs, datasources and hostnames of - the previous and current instance if they are different. These can be - examined as needed to determine any information related to a previous boot - (if applicable). +``.../data/`` +------------- -``handlers/`` +This directory contains information about instance IDs, datasources and +hostnames of the previous and current instance if they are different. These can +be examined as needed to determine any information related to a previous boot +(if applicable). - Custom ``part-handlers`` code is written out here. Files that end up here are - written out within the scheme of ``part-handler-XYZ`` where ``XYZ`` is the - handler number (the first handler found starts at ``0``). +``.../handlers/`` +----------------- +Custom ``part-handlers`` code is written out here. Files that end up here are +written out within the scheme of ``part-handler-XYZ`` where ``XYZ`` is the +handler number (the first handler found starts at ``0``). -``instance`` +``.../instance`` +---------------- - A symlink to the current ``instances/`` subdirectory that points to the - currently active instance (the active instance is dependent on the datasource - loaded). +A symlink to the current ``instances/`` subdirectory, which points to the +currently active instance. Note that the active instance depends on the loaded +datasource. -``instances/`` +``.../instances/`` +------------------ - All instances that were created using this image end up with instance - identifier subdirectories (and corresponding data for each instance). The - currently active instance will be symlinked to the ``instance`` symlink file - defined previously. +All instances that were created using this image end up with instance +identifier subdirectories (with corresponding data for each instance). The +currently active instance will be symlinked to the ``instance`` symlink file +defined previously. -``scripts/`` +``.../scripts/`` +---------------- - Scripts that are downloaded/created by the corresponding ``part-handler`` - will end up in one of these subdirectories. +Scripts in one of these subdirectories are downloaded/created by the +corresponding ``part-handler``. -``seed/`` +``.../seed/`` +------------- - Contains seeded data files: :file:`meta-data`, :file:`network-config`, - :file:`user-data`, :file:`vendor-data`. +Contains seeded data files: :file:`meta-data`, :file:`network-config`, +:file:`user-data`, :file:`vendor-data`. -``sem/`` +``.../sem/`` +------------ - ``Cloud-init`` has a concept of a module semaphore, which basically consists - of the module name and its frequency. These files are used to ensure a module - is only run "per-once", "per-instance", or "per-always". This folder contains - semaphore :file:`files` which are only supposed to run "per-once" (not tied - to the instance ID). +Cloud-init has a concept of a module semaphore, which consists of the module +name and its frequency. These files are used to ensure a module is only run +"per-once", "per-instance", or "per-always". This folder contains +semaphore :file:`files` which are only supposed to run "per-once" (not tied +to the instance ID). diff -Nru cloud-init-23.3.3/doc/rtd/development/docs.rst cloud-init-23.4.3/doc/rtd/development/docs.rst --- cloud-init-23.3.3/doc/rtd/development/docs.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/docs.rst 1970-01-01 00:00:00.000000000 +0000 @@ -1,136 +0,0 @@ -.. _docs: - -Documentation -************* - -These docs are hosted on `Read the Docs`_. The following will explain how to -contribute to, and build, these docs locally. - -The documentation is primarily written in reStructuredText, with some pages -written in standard Markdown. - -Building -======== - -There is a makefile target to build the documentation for you: - -.. code-block:: shell-session - - $ tox -e doc - -This will do two things: - -- Build the documentation using sphinx. -- Run doc8 against the documentation source code. - -Once built, the HTML files will be viewable in `doc/rtd_html`. Use your -web browser to open `index.html` to view and navigate the site. - -Style guide -=========== - -Language --------- - -Where possible, text should be written in UK English. However, discretion and -common sense can both be applied. For example, where text refers to code -elements that exist in US English, the spelling of these elements should not -be changed to UK English. - -Headings --------- - -In reStructuredText, headings are denoted using symbols to underline the text. -The headings used across the documentation use the following hierarchy: - -- ``#####``: Top level header (reserved for the main index page) -- ``*****``: Title header (used once at the top of a new page) -- ``=====``: Section headers -- ``-----``: Subsection headers -- ``^^^^^``: Sub-subsection headers -- ``"""""``: Paragraphs - -The length of the underline must be at least as long as the title itself. - -Ensure that you do not skip header levels when creating your document -structure, i.e., that a section is followed by a subsection, and not a -sub-subsection. - -Line length ------------ - -Please keep the line lengths to a maximum of **79** characters. This ensures -that the pages and tables do not get so wide that side scrolling is required. - -Anchor labels -------------- - -Adding an anchor label at the top of the page allows for the page to be -referenced by other pages. For example for the FAQ page this would be: - -.. code-block:: rst - - .. _faq: - - FAQ - *** - -When the reference is used in a document, the displayed text will be that of -the next heading immediately following the label (so, FAQ in this example), -unless specifically overridden. - -If you use labels within a page to refer, for example, to a subsection, use a -label that follows the format: ``[pagelabel]-[Section]`` e.g., for this -"Anchor labels" section, something like ``_docs-Anchor:`` or ``_docs-Label:``. -Using a consistent style will aid greatly when referencing from other pages. - -Links ------ - -To aid in documentation maintenance and keeping links up-to-date, links should -be presented in a single block at the end of the page. - -Where possible, use contextual text in your links to aid users with screen -readers and other accessibility tools. For example, "check out our -:ref:`documentation style guide`" is preferable to "click -:ref:`here` for more". - -Code blocks ------------ - -Our documentation uses the Sphinx extension "sphinx-copybutton", which creates -a small button on the right-hand side of code blocks for users to copy the -code snippets we provide. - -The copied code will strip out the prompt symbol (``$``) so that users can -paste commands directly into their terminal. For user convenience, please -ensure that code output is presented in a separate code block to the commands. - -Vertical whitespace -------------------- - -One newline between each section helps ensure readability of the documentation -source code. - -Common words ------------- - -There are some common words that should follow specific usage: - -- ``cloud-init``: Always hyphenated. Follows sentence case, so only - capitalised at the start of a sentence (e.g., ``Cloud-init``). -- ``metadata``, ``datasource``: One word. -- ``user data``, ``vendor data``: Two words, not to be combined or hyphenated. - -Acronyms --------- - -Acronyms are always capitalised (e.g., JSON, YAML, QEMU, LXD) in text. - -The first time an acronym is used on a page, it is best practice to introduce -it by showing the expanded name followed by the acronym in parentheses. E.g., -Quick EMUlator (QEMU). If the acronym is very common, or you provide a link to -a documentation page that provides such details, you will not need to do this. - - -.. _Read the Docs: https://readthedocs.com/ diff -Nru cloud-init-23.3.3/doc/rtd/development/docs_layout.rst cloud-init-23.4.3/doc/rtd/development/docs_layout.rst --- cloud-init-23.3.3/doc/rtd/development/docs_layout.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/docs_layout.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,79 @@ +.. _docs_layout: + +Documentation directory layout +****************************** + +Cloud-init's documentation directory structure, with respect to the root +directory: + +.. code-block:: text + + /doc/ + - examples/ + - man/ + - rtd/ + - tutorial/ + - howto/ + - explanation/ + - reference/ + + - development/ + - static/ + - css/ + - js/ + - *logos* + - *conf.py* + - *index.rst* + - *links.txt* + - rtd_html/ + - sources/ + + +``examples/`` +============= + + +``man/`` +======== + +This subdirectory contains the Linux man pages for the binaries provided +by cloud-init. + + +``rtd/`` +======== + +This subdirectory is of most interest to anyone who wants to create or update +either the content of the documentation, or the styling of it. + +* The content of the documentation is organised according to the `Diataxis`_ + framework and can be found in the subdirectories: ``tutorial/``, ``howto/``, + ``explanation/``, and ``reference/``. + +* The ``development/`` subdirectory contains documentation for contributors. + +* ``static/`` contains content that relates to the styling of the documentation + in the form of custom CSS or javascript files found in ``css/`` and ``js/`` + respectively. This is also where you can find the cloud-init logo. + +* ``conf.py`` contains Sphinx configuration commands. +* ``index.rst`` is the front page of the documentation. +* ``links.txt`` contains common (and reusable) links so that you do not need to + define the same URLs on every page and can use a more convenient shorthand + when referencing often-used links. + +``rtd_html/`` +============= + +When the documentation is built locally using ``tox -e doc``, the built pages +can be found in this folder. + +``sources/`` +============ + +This subdirectory contains demos which can help the reader understand +how parts of the product work. + +.. LINKS + +.. include:: ../links.txt diff -Nru cloud-init-23.3.3/doc/rtd/development/find_issues.rst cloud-init-23.4.3/doc/rtd/development/find_issues.rst --- cloud-init-23.3.3/doc/rtd/development/find_issues.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/find_issues.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,60 @@ +Find issues to work on +********************** + +We track bugs and issues with GitHub issues, and use labels to categorise them. +You can filter the list of open issues by different labels according to what +you're interested in contributing. + +For new contributors, we especially recommend using the "good first issue" +label, and during Hacktoberfest we also add the "hacktoberfest" label to +smaller, self-contained issues suitable for participants. + +If you would like to work on documentation specifically, you can also use the +"documentation" label to filter the list of issues. + +Claiming an issue +================= + +You can express your interest in an issue by posting a comment on it. You +should only work on one open issue at a time to avoid overloading yourself. + +If you have not submitted a PR or commented with an updated status within a +week of leaving your initial comment, other contributors should consider the +issue to be available for them to work on. + +When you submit your proposed fix for an issue, including the issue number in +the PR commit message will link the issue to your proposed fix. This is the +``Fixes GH-0000`` line in the template PR commit message. + +Creating an issue +================= + +If you've spotted something that doesn't already have an issue, you can always +create one in GitHub. + +For documentation issues, you can submit an issue in GitHub using the +"Give feedback" button at the top of each documentation page. It will +automatically include the URL of the page you came from, so all you need to +do is describe the issue you've found. + +No-code or low-code options +=========================== + +Contributing to our documentation is a great way to get involved with no or +minimal coding experience. + +If you can't find a documentation issue you want to work on, you can always +`check out the documentation `_ for yourself and see what +improvements you think can be made. This might be related to: + +* spelling and grammar +* the user interface/experience (UI/UX) +* accessibility +* the CSS theming +* or even just highlighting things you found confusing or unclear + +Feel free to `contact us on IRC `_ if you have other ideas about +contributions you might want to make, such as blog posts, guides, or tutorials. + +.. LINKS +.. include:: ../links.txt diff -Nru cloud-init-23.3.3/doc/rtd/development/first_PR.rst cloud-init-23.4.3/doc/rtd/development/first_PR.rst --- cloud-init-23.3.3/doc/rtd/development/first_PR.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/first_PR.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,81 @@ +Submit your first pull request +****************************** + +Follow these steps prior to submitting your first pull request to cloud-init: + +Setup Git and GitHub appropriately +================================== + +Understanding how to use Git and GitHub is a prerequisite for contributing to +cloud-init. Please refer to the `GitHub quickstart`_ documentation +for more information. + +Sign the CLA +============ + +To contribute to cloud-init, you must first sign the Canonical +`contributor license agreement `_ (CLA). + +If you have already signed it as an individual, your Launchpad username will be +listed in the `contributor-agreement-canonical`_ group. Unfortunately there is +no easy way to check if the organisation or company you are working for has +signed it. + +When you sign: + +* ensure that you fill in the GitHub username field, +* when prompted for a 'Project contact' or 'Canonical Project Manager', enter + 'James Falcon'. + +If your company has signed the CLA for you, please contact us to help in +verifying which Launchpad/GitHub accounts are associated with the company. + +For any questions or help with the process, email +`James Falcon `_ with the subject: +"Cloud-init CLA". You can also contact user ``falcojr`` in the #cloud-init +channel on the `Libera IRC network `_. + +Add your name to the CLA signers list +===================================== + +As part of your first PR to cloud-init, you should also add your GitHub +username (alphabetically) to the in-repository list that we use to track CLA +signatures: `tools/.github-cla-signers`_. + +`PR #344`_ and `PR #345`_ are good examples of what this should look like in +your pull request, though please do not use a separate PR for this step. + +Create a sandbox environment +============================ + +It is very often helpful to create a safe and sandboxed environment to test +your changes in while you work. If you are not sure how to do this, check out +:ref:`our QEMU tutorial`, which walks through this process +step-by-step. + +Format the code +=============== + +Apply the ``black`` and ``isort`` formatting rules with `tox`_: :: + + tox -e do_format + +Run unit tests +============== + +Run unit tests and lint/formatting checks with `tox`_: :: + + tox + +Read our code review process +============================ + +Once you have submitted your PR (if not earlier!) you will want to read the +cloud-init :ref:`Code Review Process`, so you can +understand how your changes will end up in cloud-init's codebase. + +.. include:: ../links.txt +.. _tools/.github-cla-signers: https://github.com/canonical/cloud-init/blob/main/tools/.github-cla-signers +.. _contributor-agreement-canonical: https://launchpad.net/%7Econtributor-agreement-canonical/+members +.. _PR #344: https://github.com/canonical/cloud-init/pull/344 +.. _PR #345: https://github.com/canonical/cloud-init/pull/345 diff -Nru cloud-init-23.3.3/doc/rtd/development/index.rst cloud-init-23.4.3/doc/rtd/development/index.rst --- cloud-init-23.3.3/doc/rtd/development/index.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/index.rst 2024-02-02 21:21:52.000000000 +0000 @@ -1,56 +1,88 @@ -Development -*********** +.. _contributing: -``Cloud-init`` is an open source project that warmly welcomes community -projects, contributions, suggestions, fixes and constructive feedback. If you -would like to contribute to ``cloud-init``, this set of documentation will help -orient you with our processes. +How to contribute to cloud-init +******************************* ------ +Thank you for wanting to help us improve cloud-init! There are a variety of +ways you can contribute to this project, including no-code and low-code +options. This documentation will help orient you with our processes. -Contributing -============ +Please make sure that you read this guide before starting to contribute. It +contains all the details you need to know to give your contribution the best +chance of being accepted. -.. toctree:: - :maxdepth: 1 +Cloud-init is hosted and managed on `GitHub`_. If you're not +familiar with how GitHub works, their +`quickstart documentation`_ +provides an excellent introduction to all the tools and processes you'll need +to know. - contributing.rst - module_creation.rst - datasource_creation.rst - code_review.rst - dir_layout.rst +.. _contributing-prerequisites: -Debugging and reporting -======================= +Prerequisites +============= -.. toctree:: - :maxdepth: 1 +Before you can begin, you will need to: - ../howto/bugs.rst - logging.rst - debugging.rst +* Read and agree to abide by our `Code of Conduct`_. -Testing -======= +* Sign the Canonical `contributor license agreement `_. This grants us your + permission to use your contributions in the project. -.. toctree:: - :maxdepth: 1 +* Create (or have) a GitHub account. We will refer to your GitHub username as + ``GH_USER``. - testing.rst - integration_tests.rst +Getting help +============ -Documentation -============= +We use IRC and have a dedicated `#cloud-init` channel where you can contact +us for help and guidance. This link will take you directly to our +`IRC channel on Libera `_. + +Getting started +=============== .. toctree:: - :maxdepth: 1 + :maxdepth: 1 - docs.rst + find_issues.rst + first_PR.rst + code_review.rst + +Contribute +========== + +Pull request checklist +---------------------- + +Before any pull request can be accepted, remember to do the following: + +* Make sure your GitHub username is added (alphabetically) to the in-repository + list that we use to track CLA signatures: `tools/.github-cla-signers`_. +* Add or update any :ref:`unit tests` accordingly. +* Add or update any :ref:`integration_tests` (if applicable). +* Format code (using ``black`` and ``isort``) with ``tox -e do_format``. +* Ensure unit tests and/or linting checks pass using ``tox``. +* Submit a PR against the ``main`` branch of the cloud-init repository. -Cloud-init summit -================= +Debugging and reporting +----------------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 1 + + ../howto/bugs.rst + logging.rst + debugging.rst - summit.rst +.. LINKS: +.. include:: ../links.txt +.. _quickstart documentation: https://docs.github.com/en/get-started/quickstart +.. _tools/.github-cla-signers: https://github.com/canonical/cloud-init/blob/main/tools/.github-cla-signers +.. _repository: https://github.com/canonical/cloud-init +.. _contributor-agreement-canonical: https://launchpad.net/%7Econtributor-agreement-canonical/+members +.. _PR #344: https://github.com/canonical/cloud-init/pull/344 +.. _PR #345: https://github.com/canonical/cloud-init/pull/345 +.. _tox: https://tox.readthedocs.io/en/latest/ +.. _PEP-484: https://www.python.org/dev/peps/pep-0484/ +.. _PEP-526: https://www.python.org/dev/peps/pep-0526/ diff -Nru cloud-init-23.3.3/doc/rtd/development/integration_tests.rst cloud-init-23.4.3/doc/rtd/development/integration_tests.rst --- cloud-init-23.3.3/doc/rtd/development/integration_tests.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/integration_tests.rst 2024-02-02 21:21:52.000000000 +0000 @@ -180,6 +180,7 @@ Examples -------- + A simple test case using the ``client`` fixture: .. code-block:: python diff -Nru cloud-init-23.3.3/doc/rtd/development/logging.rst cloud-init-23.4.3/doc/rtd/development/logging.rst --- cloud-init-23.3.3/doc/rtd/development/logging.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/logging.rst 2024-02-02 21:21:52.000000000 +0000 @@ -3,28 +3,29 @@ Logging ******* -``Cloud-init`` supports both local and remote logging configurable through +Cloud-init supports both local and remote logging configurable through multiple configurations: - Python's built-in logging configuration -- ``Cloud-init``'s event reporting system +- Cloud-init's event reporting system - The ``cloud-init`` ``rsyslog`` module Python logging ============== -``Cloud-init`` uses the Python logging module, and can accept config for this -module using the standard Python ``fileConfig`` format. ``Cloud-init`` looks +Cloud-init uses the Python logging module, and can accept config for this +module using the standard Python ``fileConfig`` format. Cloud-init looks for config for the logging module under the ``logcfg`` key. .. note:: - The logging configuration is not YAML, it is Python ``fileConfig`` format, - and is passed through directly to the Python logging module. Please use - the correct syntax for a multi-line string in YAML. -By default, ``cloud-init`` uses the logging configuration provided in + The logging configuration is not YAML, it is Python ``fileConfig`` format, + and is passed through directly to the Python logging module. Please use + the correct syntax for a multi-line string in YAML. + +By default, cloud-init uses the logging configuration provided in :file:`/etc/cloud/cloud.cfg.d/05_logging.cfg`. The default Python logging -configuration writes all ``cloud-init`` events with a priority of ``WARNING`` +configuration writes all cloud-init events with a priority of ``WARNING`` or higher to console, and writes all events with a level of ``DEBUG`` or higher to :file:`/var/log/cloud-init.log` and via :file:`syslog`. @@ -52,13 +53,12 @@ ``stderr``), and ``FileHandler`` which outputs to a log file. The logging module also supports logging over net sockets, over http, via smtp, and additional complex configurations. For full details about the handlers -available for Python logging, please see the documentation for -`python logging handlers`_. +available for Python logging, see the `python logging handlers`_ documentation. Log messages are formatted using the ``logging.Formatter`` class, which is configured using ``formatter`` config entities. A default format of ``%(message)s`` is given if no formatter configs are specified. Formatter -config entities accept a format string which supports variable replacements. +config entities accept a format string that supports variable replacements. These may also accept a ``datefmt`` string which may be used to configure the timestamp used in the log messages. The format variables ``%(asctime)s``, ``%(levelname)s`` and ``%(message)s`` are commonly used and represent the @@ -66,12 +66,13 @@ additional information on logging formatters see `python logging formatters`_. .. note:: + By default, the format string used in the logging formatter are in Python's old style ``%s`` form. The ``str.format()`` and ``string.Template`` styles can also be used by using ``{`` or ``$`` in place of ``%`` by setting the ``style`` parameter in formatter config. -A simple (but functional) Python logging configuration for ``cloud-init`` is +A simple (but functional) Python logging configuration for cloud-init is below. It will log all messages of priority ``DEBUG`` or higher to both :file:`stderr` and :file:`/tmp/my.log` using a ``StreamHandler`` and a ``FileHandler``, using the default format string ``%(message)s``: :: @@ -107,9 +108,9 @@ Command output ============== -``Cloud-init`` can redirect its :file:`stdout` and :file:`stderr` based on +Cloud-init can redirect its :file:`stdout` and :file:`stderr` based on config given under the ``output`` config key. The output of any commands run -by ``cloud-init`` and any user or vendor scripts provided will also be +by cloud-init and any user or vendor scripts provided will also be included here. The ``output`` key accepts a dictionary for configuration. Output files may be specified individually for each stage (``init``, ``config``, and ``final``), or a single key ``all`` may be used to specify @@ -120,13 +121,13 @@ with :file:`stdout` first and :file:`stderr` second, or as a single string to use for both. The strings passed to all of these keys are handled by the system shell, so any form of redirection that can be used in bash is valid, -including piping ``cloud-init``'s output to ``tee``, or ``logger``. If only a -filename is provided, ``cloud-init`` will append its output to the file as +including piping cloud-init's output to ``tee``, or ``logger``. If only a +filename is provided, cloud-init will append its output to the file as though ``>>`` was specified. -By default, ``cloud-init`` loads its output configuration from +By default, cloud-init loads its output configuration from :file:`/etc/cloud/cloud.cfg.d/05_logging.cfg`. The default config directs both -:file:`stdout` and :file:`stderr` from all ``cloud-init`` stages to +:file:`stdout` and :file:`stderr` from all cloud-init stages to :file:`/var/log/cloud-init-output.log`. The default config is given as: :: output: { all: "| tee -a /var/log/cloud-init-output.log" } @@ -152,20 +153,20 @@ Event reporting =============== -``Cloud-init`` contains an eventing system that allows events to be emitted +Cloud-init contains an eventing system that allows events to be emitted to a variety of destinations. Three configurations are available for reporting events: - ``webhook``: POST to a web server. -- ``log``: Write to the ``cloud-init`` log at configurable log level. +- ``log``: Write to the cloud-init log at configurable log level. - ``stdout``: Print to :file:`stdout`. -The default configuration is to emit events to the ``cloud-init`` log file +The default configuration is to emit events to the cloud-init log file at ``DEBUG`` level. -Event reporting can be configured using the ``reporting`` key in -``cloud-config`` user data. +Event reporting can be configured using the ``reporting`` key in cloud-config +user data. Configuration ------------- @@ -238,19 +239,19 @@ ``rsyslog`` module ================== -``Cloud-init``'s ``cc_rsyslog`` module allows for fully customizable -``rsyslog`` configuration under the ``rsyslog`` config key. The simplest way -to use the ``rsyslog`` module is by specifying remote servers under the -``remotes`` key in ``rsyslog`` config. The ``remotes`` key takes a dictionary -where each key represents the name of an ``rsyslog`` server and each value is -the configuration for that server. The format for server config is: - - - optional filter for log messages (defaults to ``*.*``) - - optional leading ``@`` or ``@@``, indicating UDP and TCP respectively - (defaults to ``@``, for UDP) - - IPv4 or IPv6 hostname or address. IPv6 addresses must be in ``[::1]`` - format (e.g., ``@[fd00::1]:514``) - - optional port number (defaults to ``514``) +Cloud-init's ``cc_rsyslog`` module allows for fully customizable ``rsyslog`` +configuration under the ``rsyslog`` config key. The simplest way to use the +``rsyslog`` module is by specifying remote servers under the ``remotes`` key +in ``rsyslog`` config. The ``remotes`` key takes a dictionary where each key +represents the name of an ``rsyslog`` server and each value is the +configuration for that server. The format for server config is: + +- optional filter for log messages (defaults to ``*.*``) +- optional leading ``@`` or ``@@``, indicating UDP and TCP respectively + (defaults to ``@``, for UDP) +- IPv4 or IPv6 hostname or address. IPv6 addresses must be in ``[::1]`` + format (e.g., ``@[fd00::1]:514``) +- optional port number (defaults to ``514``) For example, to send logging to an ``rsyslog`` server named ``log_serv`` with address ``10.0.4.1``, using port number ``514``, over UDP, with all log diff -Nru cloud-init-23.3.3/doc/rtd/development/style_docs.rst cloud-init-23.4.3/doc/rtd/development/style_docs.rst --- cloud-init-23.3.3/doc/rtd/development/style_docs.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/style_docs.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,133 @@ +Documentation style guide +************************* + +Language +-------- + +Where possible, text should be written in UK English. However, discretion and +common sense can both be applied. For example, where text refers to code +elements that exist in US English, the spelling of these elements should not +be changed to UK English. + +Try to be concise and to the point in your writing. It is acceptable to link +to official documentation elsewhere rather than repeating content. It's also +good practice not to assume that your reader has the same level of knowledge +as you, so if you're covering a new or complicated topic, then providing +contextual links to help the reader is encouraged. + +Feel free to include a "Further reading" section at the end of a page if you +have additional resources an interested reader might find helpful. + +Headings +-------- + +In reStructuredText, headings are denoted using symbols to underline the text. +The headings used across the documentation use the following hierarchy, which +is borrowed from the `Python style guide`_: + +- ``#####``: Top level header (reserved for the main index page) +- ``*****``: Title header (used once at the top of a new page) +- ``=====``: Section headers +- ``-----``: Subsection headers +- ``^^^^^``: Sub-subsection headers +- ``"""""``: Paragraphs + +The length of the underline must be at least as long as the title itself. + +Ensure that you do not skip header levels when creating your document +structure, i.e., that a section is followed by a subsection, and not a +sub-subsection. + +Line length +----------- + +Please keep the line lengths to a maximum of **79** characters. This ensures +that the pages and tables do not get so wide that side scrolling is required. + +Blank spaces at the ends of lines must also be removed, otherwise the `tox` +build checks will fail (it will warn you about trailing whitespace). + +Anchor labels +------------- + +Adding an anchor label at the top of the page allows for the page to be +referenced by other pages. For example for the FAQ page this would be: + +.. code-block:: rst + + .. _faq: + + FAQ + *** + +When the reference is used in a document, the displayed text will be that of +the next heading immediately following the label (so, FAQ in this example), +unless specifically overridden. + +If you use labels within a page to refer, for example, to a subsection, use a +label that follows the format: ``[pagelabel]-[Section]`` e.g., for this +"Anchor labels" section, something like ``_docs-Anchor:`` or ``_docs-Label:``. +Using a consistent style will aid greatly when referencing from other pages. + +Links +----- + +To aid in documentation maintenance and keeping links up-to-date, links should +be presented in a single block at the end of the page. + +Where possible, use contextual text in your links to aid users with screen +readers and other accessibility tools. For example, "check out our +:ref:`documentation style guide`" is preferable to "click +:ref:`here` for more". + +Images +------ + +It is generally best to avoid screenshots where possible. If you need to refer +to text output, you can use code blocks. For diagrams, we recommend the use of +`Mermaid`_. + +Code blocks +----------- + +Our documentation uses the Sphinx extension "sphinx-copybutton", which creates +a small button on the right-hand side of code blocks for users to copy the +code snippets we provide. + +The copied code will strip out the prompt symbol (``$``) so that users can +paste commands directly into their terminal. For user convenience, please +ensure that code output is presented in a separate code block to the commands. + +Vertical whitespace +------------------- + +One newline between each section helps ensure readability of the documentation +source code. + +Common words +------------ + +There are some common words that should follow specific usage in text: + +- **cloud-init**: Always hyphenated, and follows sentence case, so only + capitalised at the start of a sentence. +- **metadata**, **datasource**: One word. +- **user data**, **vendor data**: Two words, not to be combined or hyphenated. + +When referring to file names, which may be hyphenated, they should be decorated +with backticks to ensure monospace font is used to distinguish them from +regular text. + +Acronyms +-------- + +Acronyms are always capitalised (e.g., JSON, YAML, QEMU, LXD) in text. + +The first time an acronym is used on a page, it is best practice to introduce +it by showing the expanded name followed by the acronym in parentheses. E.g., +Quick EMUlator (QEMU). If the acronym is very common, or you provide a link to +a documentation page that provides such details, you will not need to do this. + +.. _Read the Docs: https://readthedocs.com/ +.. _Python style guide: https://devguide.python.org/documentation/markup/ +.. _Mermaid: https://mermaid.js.org/ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2017_bridge.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2017_bridge.jpg differ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2017_meeting.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2017_meeting.jpg differ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2017_seattle.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2017_seattle.jpg differ diff -Nru cloud-init-23.3.3/doc/rtd/development/summit/2017_summit.rst cloud-init-23.4.3/doc/rtd/development/summit/2017_summit.rst --- cloud-init-23.3.3/doc/rtd/development/summit/2017_summit.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/summit/2017_summit.rst 2024-02-02 21:21:52.000000000 +0000 @@ -6,10 +6,8 @@ This article was written by Joshua Powers and `originally published`_ on 31 August 2017. It is shared here `under license`_ with no changes. -.. image:: 2017_seattle.jpg - :alt: +.. image:: https://assets.ubuntu.com/v1/e4b097cf-2017_seattle.jpg :align: center - :scale: 70% Last week the cloud-init development team from Canonical ran a two-day summit in Seattle, Washington. The purpose of the summit was to meet with contributors @@ -22,8 +20,7 @@ of course, Ubuntu. Special thanks go to Google for hosting us and to Microsoft for buying everyone dinner! -.. image:: 2017_meeting.jpg - :alt: +.. image:: https://assets.ubuntu.com/v1/8c221862-2017_meeting.jpg :align: center Demos @@ -65,8 +62,7 @@ During this time, attendees came with outstanding bugs to discuss possible fixes as well as go through outstanding merge requests and get live reviews. -.. image:: 2017_bridge.jpg - :alt: +.. image:: https://assets.ubuntu.com/v1/ac44f440-2017_bridge.jpg :align: center Conclusions Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2018_roadmap.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2018_roadmap.jpg differ diff -Nru cloud-init-23.3.3/doc/rtd/development/summit/2018_summit.rst cloud-init-23.4.3/doc/rtd/development/summit/2018_summit.rst --- cloud-init-23.3.3/doc/rtd/development/summit/2018_summit.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/summit/2018_summit.rst 2024-02-02 21:21:52.000000000 +0000 @@ -6,7 +6,7 @@ This article was written by Joshua Powers and `originally published`_ on 27 August 2018. It is shared here `under license`_ with no changes. -.. image:: 2018_waterfront.jpg +.. image:: https://assets.ubuntu.com/v1/04d0b05b-2018_waterfront.jpg :alt: Seattle waterfront :align: center @@ -23,7 +23,7 @@ to Microsoft for providing breakfast and lunch while hosting us and to the Amazon Linux and AWS teams for buying everyone dinner! -.. image:: 2018_roadmap.jpg +.. image:: https://assets.ubuntu.com/v1/78642b7c-2018_roadmap.jpg :alt: Roadmap talk :align: center @@ -73,7 +73,7 @@ During this time, attendees came with outstanding bugs to discuss possible fixes as well as go through outstanding merge requests and get live reviews. -.. image:: 2018_talk.jpg +.. image:: https://assets.ubuntu.com/v1/7d203365-2018_talk.jpg :alt: Another talk :align: center Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2018_talk.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2018_talk.jpg differ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2018_waterfront.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2018_waterfront.jpg differ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2019_amazon.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2019_amazon.jpg differ Binary files /tmp/tmpetjw4yhx/OCB_Wipq3o/cloud-init-23.3.3/doc/rtd/development/summit/2019_market.jpg and /tmp/tmpetjw4yhx/jeQvx9MDTx/cloud-init-23.4.3/doc/rtd/development/summit/2019_market.jpg differ diff -Nru cloud-init-23.3.3/doc/rtd/development/summit/2019_summit.rst cloud-init-23.4.3/doc/rtd/development/summit/2019_summit.rst --- cloud-init-23.3.3/doc/rtd/development/summit/2019_summit.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/summit/2019_summit.rst 2024-02-02 21:21:52.000000000 +0000 @@ -6,7 +6,7 @@ This article was written by Joshua Powers and `originally published`_ on 21 October 2019. It is shared here `under license`_ with no changes. -.. image:: 2019_market.jpg +.. image:: https://assets.ubuntu.com/v1/e0319f90-2019_market.jpg :alt: Seattle market :align: center @@ -58,7 +58,7 @@ During this time, attendees came with outstanding bugs to discuss possible fixes as well as go through outstanding merge requests and get live reviews. -.. image:: 2019_amazon.jpg +.. image:: https://assets.ubuntu.com/v1/d774b249-2019_amazon.jpg :alt: Another talk :align: center diff -Nru cloud-init-23.3.3/doc/rtd/development/summit/2023_summit.rst cloud-init-23.4.3/doc/rtd/development/summit/2023_summit.rst --- cloud-init-23.3.3/doc/rtd/development/summit/2023_summit.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/summit/2023_summit.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,183 @@ +cloud-init: Summit 2023 +*********************** + +After a three-year hiatus the two-day cloud-init summit finally resumed in +August, giving Canonical a chance to reconnect with the community in person, +and to realign on the direction and goals of the project. + +.. figure:: https://assets.ubuntu.com/v1/d8ed72fb-2023_image1.jpg + :alt: Man running along a trail through the forest near the venue + :align: center + + Enjoying a jog through the beautiful forest around the Microsoft Redmond + campus! + +The event was generously hosted by Microsoft this year at their Redmond campus +in Seattle, Washington, and we are grateful to the Microsoft community members +"on the ground" who coordinated with Canonical's cloud-init development team to +help organise and run the event. Big thanks go as well to the Canonical +community team for helping us to set up the event site, as well as for their +support and guidance with all the planning involved. + +As in previous years, the summit was a great opportunity for cloud-init +contributors to get together and discuss the most recent developments in the +project, provide demos of new features, resolve outstanding issues, and shape +the future development direction of the project. It was wonderful to see some +"old" faces again after such a long time, as well as getting to meet some of +our newer contributors in person. + +The first hybrid summit +======================= + +This summit was organised as a hybrid event for the first time, and despite +some initial uncertainties about how to implement that, it worked very well. +In-person attendees included developers and contributors from Microsoft, +Google, Amazon, Oracle, openSUSE and we had remote presentations provided by +FreeBSD and AlpineLinux maintainers. + +In addition to our in-person gathering, we also had lively participation from +our remote attendees from around the world. With this hybrid format allowing +attendance from community members who might not otherwise have been able to +take part, this is a format that we’ll want to carry forward to next year to +open the event to the widest possible audience. + +Special thanks go to Canonical for sponsoring the dinner! It was a great chance +to build community interactions "after hours", with topics ranging far and +wide. Overall, it was a perfect opportunity to dig into industry dynamics that +influence cloud-init engagement. + +.. image:: https://assets.ubuntu.com/v1/2687e23a-2023_image3.jpg + :alt: The in-person participants enjoying the group dinner + :align: center + +Highlights of the discussions +============================= + +Thanks to all our presenters; Mina Galic (FreeBSD) and Dermot Bradley +(AlpineLinux), Chris Patterson (Microsoft) James Falcon, Brett Holman, +Catherine Redfield, Alberto Contreras, Sally Makin, John Chittum, Daniel +Bungert and Chad Smith. You really helped to make this event a success. + +Presentation take-aways +----------------------- + +* **Integration-testing tour/demo**: James showed how Canonical uses our + integration tests and pycloudlib during SRU verification, and demonstrated + how we think other clouds should be involved in standard evaluation of + cloud-init releases before publication. + + There was interest in leveraging this testing at openSUSE, Amazon Linux and + possibly Microsoft’s Mariner; they may be looking to extend our framework for + their release testing. Our homework from this is improved developer docs on + extending integration tests for other distributions. + + Azure are looking to invest in distribution-agnostic integration test + frameworks and want to knowledge-share with the cloud-init community on that. + +* **Security-policy for cloud-init CVE handling**: Our community would like + Canonical to provide more context during the embargo period on CVEs about the + mitigation steps required. This is especially the case in any downstream + packaging, to allow downstream package maintainers more time to prepare. + +* **Cloud platforms want/like strict schema validation** and errors on invalid + user-data/config. They also requested more visibility into any warnings + surfaced by cloud-init with simple tools so that they can avoid costly "log + spelunking". + + This aligns well with Brett’s ongoing roadmap work to raise warnings from the + CLI and some of the strict JSON schema validation on network-config and user + data/vendor data. + +* Good lessons from both AlpineLinux (Dermot Bradley), who investigated SSH + alternatives like dropbearSSH and tinySSH, and FreeBSD (Mina Galić), who + reported on the development and publishing process and on finding better ways + for FreeBSD and Alpine to engage with clouds, so they can get sponsorship of + open source images with cloud-init hosted in AWS. + +.. image:: https://assets.ubuntu.com/v1/5640b4ed-2023_image2.jpg + :alt: Overview of the meeting room, with the remote-attending participants + :align: center + +Round-table discussions +----------------------- + +* **Boot-speed**: The discussion hosted by Catherine, Alberto and Chad + confirmed that our ongoing boot speed work is critical to clouds and + cloud-customers, who continue to gauge boot speed based on wall time to SSH + into the instance. + + This is a more critical measurement than the time to all services being up. + In our discussion, we received feedback that every millisecond counts. We + also learned that there is hesitation about moving to precompiled languages + such as Go, due to the potential image size impacts, or Rust, due to the + somewhat limited platform support. + + Partners are also looking for `cloud-init analyze` to report on external + systemd-related impacts (such as `NetworkManager-wait-online.service` or + `systemd-network-wait-online.service` delays) due to external units/services + that affect boot. + +* **Review of our Python support matrix** for all downstreams, with the goal of + Python 3.6 version deprecation. Due to ongoing downstream support needs, we + are looking to retain Python 3.6 support until March 2024. + +* **Shared test frameworks**: Azure intends to invest in integration testing + with the cloud-init community, to develop distribution-agnostic best + practices for verification of distribution releases, and boot-speed and image + health analysis. If there are ways we want to collaborate on generalised + testing and verification of images, they may provide some development toward + this cause. + +Breakout sessions +----------------- + +* **Private reviews of partner engagements** with Oracle and AWS, and Fabio + Martins, Kyler Horner, and James to prioritise ongoing work and plan for the + future development of IPv6-only datasource support - as well as other + features. + +* **Brett and openSUSE’s Robert Schweikert** worked through downstream patch + review with the intent of merging many openSUSE patches upstream. Amazon + Linux has a couple of downstream patches that they may want to upstream as + well. + +Conclusions +=========== + +This two-day event gave us a fantastic chance to take the pulse of the +cloud-init project. It’s given us a healthy understanding of areas in which we +can better serve the community and how we can continue to build momentum. + +Meeting face-to-face to reflect our cloud-init plans with the community helped +confirm interest in some of the usability features we are developing, such as +better error and warning visibility and improving boot speed in cloud-init. +There is plenty of enthusiasm for continued collaboration on improved testing +and verification that all distributions and clouds can leverage. + +We also appreciated the opportunity to get valuable feedback on our +documentation, our communication, and our security processes. We’ve also +discussed and gained input into better practices we can adopt through GitHub +automation, workflows that automate pull request digests, and upstream test +matrix coverage for downstreams (beside Ubuntu). All of these things will help +us to maintain the momentum of the cloud-init project and ensure that we are +best serving the needs of our community. + +Thank you! +========== + +This event could not have taken place without the hard work and preparation of +all our presenters, organisers, and the voices of our community members in +attendance. So, thank you again to everyone who participated, and we very much +hope to see you again at the next cloud-init summit! + +Notes of both days can be found on the `cloud-init mailing list`_, and also +are `hosted in our GitHub`_ repository. There you will find additional details +about each topic and related discussions. + +Finally, if you are interested in following or getting involved in cloud-init +development check out #cloud-init on Libera.chat or subscribe to the cloud-init +mailing list. + +.. LINKS +.. _cloud-init mailing list: https://lists.launchpad.net/cloud-init/msg00460.html +.. _hosted in our GitHub: https://github.com/canonical/cloud-init/blob/main/doc/summit/2023_summit_shared_notes.md diff -Nru cloud-init-23.3.3/doc/rtd/development/summit.rst cloud-init-23.4.3/doc/rtd/development/summit.rst --- cloud-init-23.3.3/doc/rtd/development/summit.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/development/summit.rst 2024-02-02 21:21:52.000000000 +0000 @@ -22,3 +22,4 @@ cloud-init: Summit 2017 summit/2018_summit.rst summit/2019_summit.rst + summit/2023_summit.rst diff -Nru cloud-init-23.3.3/doc/rtd/explanation/analyze.rst cloud-init-23.4.3/doc/rtd/explanation/analyze.rst --- cloud-init-23.3.3/doc/rtd/explanation/analyze.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/explanation/analyze.rst 2024-02-02 21:21:52.000000000 +0000 @@ -27,8 +27,8 @@ Availability ============ -The :command:`analyze` subcommand is generally available across all -distributions, with the exception of Gentoo and FreeBSD. +The :command:`analyze boot` subcommand only works on operating systems that +use `systemd`. Subcommands =========== diff -Nru cloud-init-23.3.3/doc/rtd/explanation/boot.rst cloud-init-23.4.3/doc/rtd/explanation/boot.rst --- cloud-init-23.3.3/doc/rtd/explanation/boot.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/explanation/boot.rst 2024-02-02 21:21:52.000000000 +0000 @@ -107,9 +107,8 @@ :file:`/etc/fstab`. As such, entries in :file:`/etc/fstab` other than those necessary for cloud-init to run should not be done until after this stage. -A part-handler will run at this stage, as will boothooks including -cloud-config ``bootcmd``. The user of this functionality has to be aware -that the system is in the process of booting when their code runs. +A part-handler and :ref:`boothooks` +will run at this stage. .. _boot-Config: diff -Nru cloud-init-23.3.3/doc/rtd/explanation/index.rst cloud-init-23.4.3/doc/rtd/explanation/index.rst --- cloud-init-23.3.3/doc/rtd/explanation/index.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/explanation/index.rst 2024-02-02 21:21:52.000000000 +0000 @@ -10,6 +10,7 @@ .. toctree:: :maxdepth: 1 + introduction.rst configuration.rst boot.rst format.rst diff -Nru cloud-init-23.3.3/doc/rtd/explanation/introduction.rst cloud-init-23.4.3/doc/rtd/explanation/introduction.rst --- cloud-init-23.3.3/doc/rtd/explanation/introduction.rst 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/explanation/introduction.rst 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,118 @@ +.. _introduction: + +Introduction to cloud-init +************************** + +Managing and configuring cloud instances and servers can be a complex +and time-consuming task. Cloud-init is an open source initialisation tool that +was designed to make it easier to get your systems up and running with a +minimum of effort, already configured according to your needs. + +It’s most often used by developers, system administrators and other IT +professionals to automate configuration of VMs, cloud instances, or machines on +a network. For brevity, we’ll refer only to instances for the rest of this +page, but assume we’re also including bare metal and VMs in the discussion as +well. By automating routine setup tasks, cloud-init ensures repeatability and +efficiency in your system provisioning. + +What is the benefit of cloud-init? +================================== + +When you deploy a new cloud instance, cloud-init takes an initial configuration +that you supply, and it automatically applies those settings when the instance +is created. It’s rather like writing a to-do list, and then letting cloud-init +deal with that list for you. + +The real power of cloud-init comes from the fact that you can re-use your +configuration instructions as often as you want, and always get consistent, +reliable results. If you’re a system administrator and you want to deploy a +whole fleet of machines, you can do so with a fraction of the time and effort +it would take to manually provision them. + +What does cloud-init do? +======================== + +Cloud-init can handle a range of tasks that normally happen when a new instance +is created. It's responsible for activities like setting the hostname, +configuring network interfaces, creating user accounts, and even running +scripts. This streamlines the deployment process; your cloud instances will all +be automatically configured in the same way, which reduces the chance to +introduce human error. + +How does cloud-init work? +========================= + +The operation of cloud-init broadly takes place in two separate phases during +the boot process. The first phase is during the early (local) boot stage, +before networking has been enabled. The second is during the late boot stages, +after cloud-init has applied the networking configuration. + +During early boot +----------------- + +In this pre-networking stage, cloud-init discovers the datasource, obtains +all the configuration data from it, and configures networking. In this phase, +it will: + +* **Identify the datasource**: + The hardware is checked for built-in values that will identify the datasource + your instance is running on. The datasource is the source of all + configuration data. +* **Fetch the configuration**: + Once the datasource is identified, cloud-init fetches the configuration data + from it. This data tells cloud-init what actions to take. This can be in the + form of: + + * **Metadata** about the instance, such as the machine ID, hostname and + network config, or + * **Vendor data** and/or **user data**. These take the same form, although + Vendor data is provided by the cloud vendor, and user data is provided by + the user. These data are usually applied in the post-networking phase, and + might include: + + * Hardware optimisations + * Integration with the specific cloud platform + * SSH keys + * Custom scripts + +* **Write network configuration**: + Cloud-init writes the network configuration and configures DNS, ready to be + applied by the networking services when they come up. + +During late boot +---------------- + +In the boot stages that come after the network has been configured, cloud-init +runs through the tasks that were not critical for provisioning. This is where +it configures the running instance according to your needs, as specified in the +vendor data and/or user data. It will take care of: + +* **Configuration management**: + Cloud-init can interact with tools like Puppet, Ansible, or Chef to apply + more complex configuration - and ensure the system is up-to-date. +* **Installing software**: + Cloud-init can install software at this stage, and run software updates to + make sure the system is fully up-to-date and ready to use. +* **User accounts**: + Cloud-init is able to create and modify user accounts, set default passwords, + and configure permissions. +* **Execute user scripts**: + If any custom scripts were provided in the user data, cloud-init can run + them. This allows additional specified software to be installed, security + settings to be applied, etc. It can also inject SSH keys into the instance’s + ``authorized_keys`` file, which allows secure remote access to the machine. + +After this stage is complete, your instance is fully configured! + +What's next? +============ + +Now that you have an overview of the basics of what cloud-init is, what it does +and how it works, you will probably want to +:ref:`try it out for yourself`. + +You can also read in more detail about what cloud-init does +:ref:`during the different boot stages`, and the +:ref:`types of configuration` you can pass to cloud-init and +how they're used. + diff -Nru cloud-init-23.3.3/doc/rtd/howto/debug_user_data.rst cloud-init-23.4.3/doc/rtd/howto/debug_user_data.rst --- cloud-init-23.3.3/doc/rtd/howto/debug_user_data.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/howto/debug_user_data.rst 2024-02-02 21:21:52.000000000 +0000 @@ -33,5 +33,12 @@ To report any bugs you find, :ref:`refer to this guide `. +Validation service +------------------ + +Another option to is to use the self-hosted HTTP `validation service`_, +refer to its documentation for more info. + .. LINKS .. _validate-yaml.py: https://github.com/canonical/cloud-init/blob/main/tools/validate-yaml.py +.. _validation service: https://github.com/aciba90/cloud-config-validator diff -Nru cloud-init-23.3.3/doc/rtd/howto/disable_cloud_init.rst cloud-init-23.4.3/doc/rtd/howto/disable_cloud_init.rst --- cloud-init-23.3.3/doc/rtd/howto/disable_cloud_init.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/howto/disable_cloud_init.rst 2024-02-02 21:21:52.000000000 +0000 @@ -6,7 +6,7 @@ One may wish to disable cloud-init to ensure that it doesn't do anything on subsequent boots. Some parts of cloud-init may run once per boot otherwise. -There are two cross-platform methods of disabling ``cloud-init``. +There are three cross-platform methods of disabling ``cloud-init``. Method 1: text file ==================== @@ -31,9 +31,18 @@ .. code-block:: - $ echo 'GRUB_CMDLINE_LINUX=cloud-init.disabled' >> /etc/default/grub + $ echo 'GRUB_CMDLINE_LINUX="cloud-init=disabled"' >> /etc/default/grub $ grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg -.. note:: - When running in containers, ``cloud-init`` will read an environment - variable named ``KERNEL_CMDLINE`` in place of a kernel commandline. +Method 3: environment variable +============================== + +To disable cloud-init, pass the environment variable +``KERNEL_CMDLINE=cloud-init=disabled`` into each of cloud-init's +processes. + +Example (using systemd): + +.. code-block:: + + $ echo "DefaultEnvironment=KERNEL_CMDLINE=cloud-init=disabled" >> /etc/systemd/system.conf diff -Nru cloud-init-23.3.3/doc/rtd/index.rst cloud-init-23.4.3/doc/rtd/index.rst --- cloud-init-23.3.3/doc/rtd/index.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/index.rst 2024-02-02 21:21:52.000000000 +0000 @@ -20,6 +20,10 @@ management of a cloud instance. For cloud providers, it provides instance setup that can be integrated with your cloud. +If you would like to read more about what cloud-init is, what it does and how +it works, check out our :ref:`high-level introduction` +to the tool. + ----- .. grid:: 1 1 2 2 @@ -71,7 +75,7 @@ projects, contributions, suggestions, fixes and constructive feedback. * Read our `Code of Conduct`_ -* Ask questions in the ``#cloud-init`` `IRC channel on Libera`_ +* Ask questions in the ``#cloud-init`` `IRC channel on Libera `_ * Follow announcements or ask a question on `the cloud-init Discourse forum`_ * Join the `cloud-init mailing list`_ * :ref:`Contribute on GitHub` @@ -85,12 +89,20 @@ howto/index explanation/index reference/index - development/index + +.. toctree:: + :caption: Development + :hidden: + :maxdepth: 1 + + Contributing overview + Contribute to code + Contribute to docs + Community .. LINKS -.. _Code of Conduct: https://ubuntu.com/community/code-of-conduct -.. _IRC channel on Libera: https://kiwiirc.com/nextclient/irc.libera.chat/cloud-init +.. include:: links.txt .. _the cloud-init Discourse forum: https://discourse.ubuntu.com/c/server/cloud-init/ .. _cloud-init mailing list: https://launchpad.net/~cloud-init .. _mailing list archive: https://lists.launchpad.net/cloud-init/ diff -Nru cloud-init-23.3.3/doc/rtd/links.txt cloud-init-23.4.3/doc/rtd/links.txt --- cloud-init-23.3.3/doc/rtd/links.txt 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/links.txt 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,13 @@ +.. LINKS + +.. _CLA: https://ubuntu.com/legal/contributors +.. _Code of Conduct: https://ubuntu.com/community/code-of-conduct +.. _Diataxis: https://diataxis.fr/ +.. _GH repo: https://github.com/canonical/cloud-init +.. _GitHub: https://github.com +.. _IRC: https://kiwiirc.com/nextclient/irc.libera.chat/cloud-init +.. _Launchpad: https://launchpad.net +.. _the docs: https://cloudinit.readthedocs.io/en/latest/ +.. _tox: https://tox.readthedocs.io/en/latest/ +.. _Discourse: https://discourse.ubuntu.com/c/server/cloud-init/54 +.. _Github quickstart: https://docs.github.com/en/get-started/quickstart diff -Nru cloud-init-23.3.3/doc/rtd/reference/base_config_reference.rst cloud-init-23.4.3/doc/rtd/reference/base_config_reference.rst --- cloud-init-23.3.3/doc/rtd/reference/base_config_reference.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/base_config_reference.rst 2024-02-02 21:21:52.000000000 +0000 @@ -111,8 +111,8 @@ - ``paths``: Definitions of common paths used by ``cloud-init``. - + ``cloud_dir``: Defaults to :file:`/var/lib/cloud`. - + ``templates_dir``: Defaults to :file:`/etc/cloud/templates`. + + ``cloud_dir``: Default: :file:`/var/lib/cloud`. + + ``templates_dir``: Default: :file:`/etc/cloud/templates`. - ``distro``: Name of distro being used. - ``default_user``: Defines the default user for the system using the same @@ -147,6 +147,18 @@ * ``network-manager``: For ``nmcli connection load``/ ``nmcli connection up``. * ``networkd``: For ``ip link set up``/``ip link set down``. + - ``apt_get_command``: Command used to interact with APT repositories. + Default: ``apt-get``. + - ``apt_get_upgrade_subcommand``: APT subcommand used to upgrade system. + Default: ``dist-upgrade``. + - ``apt_get_wrapper``: Command used to wrap the apt-get command. + + + ``enabled``: Whether to use the specified ``apt_wrapper`` command. + If set to ``auto``, use the command if it exists on the ``PATH``. + Default: ``true``. + + + ``command``: Command used to wrap any ``apt-get`` calls. + Default: ``eatmydata``. Logging keys ------------ @@ -246,6 +258,13 @@ ``vendor_data``. * ``prefix``: A path to prepend to any ``vendor_data``-provided script. +``manual_cache_clean`` +^^^^^^^^^^^^^^^^^^^^^^ + +By default, cloud-init searches for a datasource on every boot. Setting +this to ``true`` will disable this behaviour. This is useful if your datasource +information will not be present every boot. Default: ``false``. + Example ======= @@ -388,6 +407,32 @@ security: http://ports.ubuntu.com/ubuntu-ports ssh_svcname: ssh + # configure where output will go + output: + init: "> /var/log/my-cloud-init.log" + config: [ ">> /tmp/foo.out", "> /tmp/foo.err" ] + final: + output: "| tee /tmp/final.stdout | tee /tmp/bar.stdout" + error: "&1" + + # Set `true` to enable the stop searching for a datasource on boot. + manual_cache_clean: False + + # def_log_file and syslog_fix_perms work together + # if + # - logging is set to go to a log file 'L' both with and without syslog + # - and 'L' does not exist + # - and syslog is configured to write to 'L' + # then 'L' will be initially created with root:root ownership (during + # cloud-init), and then at cloud-config time (when syslog is available) + # the syslog daemon will be unable to write to the file. + # + # to remedy this situation, 'def_log_file' can be set to a filename + # and syslog_fix_perms to a string containing ":" + def_log_file: /var/log/my-logging-file.log + syslog_fix_perms: syslog:root + + .. _configuration is templated: https://github.com/canonical/cloud-init/blob/main/config/cloud.cfg.tmpl .. _cc_final_message.py: https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_final_message.py diff -Nru cloud-init-23.3.3/doc/rtd/reference/cli.rst cloud-init-23.4.3/doc/rtd/reference/cli.rst --- cloud-init-23.3.3/doc/rtd/reference/cli.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/cli.rst 2024-02-02 21:21:52.000000000 +0000 @@ -66,9 +66,9 @@ :command:`clean` ================ -Remove ``cloud-init`` artifacts from :file:`/var/lib/cloud` to simulate a clean -instance. On reboot, ``cloud-init`` will re-run all stages as it did on -first boot. +Remove ``cloud-init`` artifacts from :file:`/var/lib/cloud` and config files +(best effort) to simulate a clean instance. On reboot, ``cloud-init`` will +re-run all stages as it did on first boot. * :command:`--logs`: Optionally remove all ``cloud-init`` log files in :file:`/var/log/`. @@ -78,6 +78,10 @@ remove the file. Best practice when cloning a golden image, to ensure the next boot of that image auto-generates a unique machine ID. `More details on machine-id`_. +* :command:`--configs [all | ssh_config | network ]`: Optionally remove all + ``cloud-init`` generated config files. Argument `ssh_config` cleans + config files for ssh daemon. Argument `network` removes all generated + config files for network. `all` removes config files of all types. .. _cli_collect_logs: @@ -303,6 +307,9 @@ * :command:`-h, --help`: Show this help message and exit. * :command:`-c CONFIG_FILE, --config-file CONFIG_FILE`: Path of the cloud-config YAML file to validate. +* :command:`-t SCHEMA_TYPE, --schema-type SCHEMA_TYPE`: The schema type to + validate --config-file against. One of: cloud-config, network-config. + Default: cloud-config. * :command:`--system`: Validate the system cloud-config user data. * :command:`-d DOCS [cc_module ...], --docs DOCS [cc_module ...]`: Print schema module diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/azure.rst cloud-init-23.4.3/doc/rtd/reference/datasources/azure.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/azure.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/azure.rst 2024-02-02 21:21:52.000000000 +0000 @@ -41,6 +41,10 @@ Boolean set to True to use network configuration described by Azure's IMDS endpoint instead of fallback network config of DHCP on eth0. Default is True. +* :command:`apply_network_config_for_secondary_ips` + + Boolean to configure secondary IP address(es) for each NIC per IMDS + configuration. Default is True. * :command:`data_dir` Path used to read metadata files and write crawled data. @@ -62,6 +66,7 @@ datasource: Azure: apply_network_config: true + apply_network_config_for_secondary_ips: true data_dir: /var/lib/waagent disk_aliases: ephemeral0: /dev/disk/cloud/azure_resource diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/cloudstack.rst cloud-init-23.4.3/doc/rtd/reference/datasources/cloudstack.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/cloudstack.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/cloudstack.rst 2024-02-02 21:21:52.000000000 +0000 @@ -63,4 +63,4 @@ .. _Apache CloudStack: http://cloudstack.apache.org/ -.. _CloudStack Administrator Guide: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/virtual_machines.html#user-data-and-meta-data +.. _CloudStack Administrator Guide: https://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines.html#user-data-and-meta-data diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/configdrive.rst cloud-init-23.4.3/doc/rtd/reference/datasources/configdrive.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/configdrive.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/configdrive.rst 2024-02-02 21:21:52.000000000 +0000 @@ -95,7 +95,7 @@ datasources after network config. The difference between 'local' and 'net' is that local will not require -networking to be up before user-data actions (or boothooks) are run. +networking to be up before user-data actions are run. ``instance-id`` --------------- diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/nocloud.rst cloud-init-23.4.3/doc/rtd/reference/datasources/nocloud.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/nocloud.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/nocloud.rst 2024-02-02 21:21:52.000000000 +0000 @@ -171,7 +171,7 @@ .. code-block:: - $ mcopy -oi seed.iso user-data meta-data + $ mcopy -oi seed.iso user-data meta-data :: 3. Create a new qcow image to boot, backed by your original image: diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/ovf.rst cloud-init-23.4.3/doc/rtd/reference/datasources/ovf.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/ovf.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/ovf.rst 2024-02-02 21:21:52.000000000 +0000 @@ -6,6 +6,33 @@ The OVF datasource provides a datasource for reading data from an `Open Virtualization Format`_ ISO transport. +Graceful rpctool fallback +------------------------- + +The datasource initially attempts to use the program ``vmware-rpctool`` if it +is available. However, if the program returns a non-zero exit code, then the +datasource falls back to using the program ``vmtoolsd`` with the ``--cmd`` +argument. + +On some older versions of ESXi and open-vm-tools, the ``vmware-rpctool`` +program is much more performant than ``vmtoolsd``. While this gap was +closed, it is not reasonable to expect the guest where cloud-init is running to +know whether the underlying hypervisor has the patch. + +Additionally, vSphere VMs may have the following present in their VMX file: + +.. code-block:: ini + + guest_rpc.rpci.auth.cmd.info-set = "TRUE" + guest_rpc.rpci.auth.cmd.info-get = "TRUE" + +The above configuration causes the ``vmware-rpctool`` command to return a +non-zero exit code with the error message ``Permission denied``. If this should +occur, the datasource falls back to using ``vmtoolsd``. + +Additional information +---------------------- + For further information see a full working example in ``cloud-init``'s source code tree in :file:`doc/sources/ovf`. diff -Nru cloud-init-23.3.3/doc/rtd/reference/datasources/vmware.rst cloud-init-23.4.3/doc/rtd/reference/datasources/vmware.rst --- cloud-init-23.3.3/doc/rtd/reference/datasources/vmware.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/datasources/vmware.rst 2024-02-02 21:21:52.000000000 +0000 @@ -11,7 +11,7 @@ The configuration method is dependent upon the transport. -Guest OS customisation +Guest OS customization ====================== The following configuration can be set for this datasource in ``cloud-init`` @@ -22,56 +22,69 @@ -------------------- * ``disable_vmware_customization``: true (disable) or false (enable) the VMware - traditional Linux guest customisation. Traditional Linux guest customisation - is customising a Linux virtual machine with a - `traditional Linux customisation specification`_. Setting this configuration + traditional Linux guest customization. Traditional Linux guest customization + is customizing a Linux virtual machine with a + `traditional Linux customization specification`_. Setting this configuration to false is required to make sure this datasource is found in ``ds-identify`` - when using Guest OS customisation transport. + when using Guest OS customization transport. VMware Tools only checks this + configuration in :file:`/etc/cloud/cloud.cfg`. Default: true Datasource configuration ------------------------ -* ``allow_raw_data``: true (enable) or false (disable) the VMware customisation +* ``allow_raw_data``: true (enable) or false (disable) the VMware customization using ``cloud-init`` metadata and user data directly. Since vSphere 7.0 - Update 3 version, users can create a Linux customisation specification with + Update 3 version, users can create a Linux customization specification with minimal ``cloud-init`` metadata and user data, and apply this specification to a virtual machine. This datasource will parse the metadata and user data and configure the virtual machine with them. See - `Guest customisation using cloud-init`_ for more information. + `Guest customization using cloud-init`_ for more information. Default: true * ``vmware_cust_file_max_wait``: The maximum amount of clock time (in seconds) - that should be spent waiting for VMware customisation files. + that should be spent waiting for VMware customization files. Default: 15 Configuration examples ---------------------- -1. Create the :file:`/etc/cloud/cloud.cfg.d/99-vmware-guest-customization.cfg` - file with the following content, which will enable VMware customisation and - set the maximum waiting time for the VMware customisation file to 10 - seconds: +1. Enable VMware customization and set the maximum waiting time for the + VMware customization file to 10 seconds: + + Set ``disable_vmware_customization`` in the :file:`/etc/cloud/cloud.cfg` .. code-block:: yaml disable_vmware_customization: false + + Create a :file:`/etc/cloud/cloud.cfg.d/99-vmware-guest-customization.cfg` + with the following content + + .. code-block:: yaml + datasource: VMware: vmware_cust_file_max_wait: 10 -2. Create the :file:`/etc/cloud/cloud.cfg.d/99-vmware-guest-customization.cfg` - file with the following content, which will enable VMware customisation but - will only try to apply a traditional Linux Guest Customisation - configuration, and set the maximum waiting time for the VMware - customisation file to 10 seconds: +2. Enable VMware customization but only try to apply a traditional Linux + Guest Customization configuration, and set the maximum waiting time for + the VMware customization file to 10 seconds: + + Set ``disable_vmware_customization`` in the :file:`/etc/cloud/cloud.cfg` .. code-block:: yaml disable_vmware_customization: false + + Create a :file:`/etc/cloud/cloud.cfg.d/99-vmware-guest-customization.cfg` + with the following content + + .. code-block:: yaml + datasource: VMware: allow_raw_data: false @@ -89,15 +102,15 @@ vmware-toolbox-cmd config set

The following VMware Tools configuration option affects this datasource's -behaviour when applying customisation configuration with custom scripts: +behaviour when applying customization configuration with custom scripts: * ``[deploypkg] enable-custom-scripts``: If this option is absent in VMware Tools configuration, the custom script is disabled by default for security reasons. Some VMware products could change this default behaviour (for - example: enabled by default) via customisation of the specification settings. + example: enabled by default) via customization of the specification settings. VMware admins can refer to `customization configuration`_ and set the - customisation specification settings. + customization specification settings. For more information, see `VMware vSphere Product Documentation`_ and specific VMware Tools configuration options. @@ -135,6 +148,30 @@ This section reviews several features available in this datasource. +Graceful rpctool fallback +------------------------- + +The datasource initially attempts to use the program ``vmware-rpctool`` if it +is available. However, if the program returns a non-zero exit code, then the +datasource falls back to using the program ``vmtoolsd`` with the ``--cmd`` +argument. + +On some older versions of ESXi and open-vm-tools, the ``vmware-rpctool`` +program is much more performant than ``vmtoolsd``. While this gap was +closed, it is not reasonable to expect the guest where cloud-init is running to +know whether the underlying hypervisor has the patch. + +Additionally, vSphere VMs may have the following present in their VMX file: + +.. code-block:: ini + + guest_rpc.rpci.auth.cmd.info-set = "TRUE" + guest_rpc.rpci.auth.cmd.info-get = "TRUE" + +The above configuration causes the ``vmware-rpctool`` command to return a +non-zero exit code with the error message ``Permission denied``. If this should +occur, the datasource falls back to using ``vmtoolsd``. + Instance data and lazy networks ------------------------------- @@ -441,8 +478,8 @@ .. LINKS .. _Guest OS Customization: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-58E346FF-83AE-42B8-BE58-253641D257BC.html .. _GuestInfo keys: https://github.com/vmware/govmomi/blob/master/govc/USAGE.md -.. _traditional Linux customisation specification: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-EB5F090E-723C-4470-B640-50B35D1EC016.html#GUID-9A5093A5-C54F-4502-941B-3F9C0F573A39__GUID-40C60643-A2EB-4B05-8927-B51AF7A6CC5E -.. _Guest customisation using cloud-init: https://developer.vmware.com/docs/17020/vsphere-web-services-sdk-programming-guide--8-0-/GUID-75E27FA9-2E40-4CBF-BF3D-22DCFC8F11F7.html +.. _traditional Linux customization specification: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-EB5F090E-723C-4470-B640-50B35D1EC016.html#GUID-9A5093A5-C54F-4502-941B-3F9C0F573A39__GUID-40C60643-A2EB-4B05-8927-B51AF7A6CC5E +.. _Guest customization using cloud-init: https://developer.vmware.com/docs/17686/vsphere-web-services-sdk-programming-guide--8-0-/GUID-75E27FA9-2E40-4CBF-BF3D-22DCFC8F11F7.html .. _VMware Tools: https://docs.vmware.com/en/VMware-Tools/index.html .. _customization configuration: https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/helpers/vmware/imc/config.py .. _VMware vSphere Product Documentation: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-EB5F090E-723C-4470-B640-50B35D1EC016.html#GUID-9A5093A5-C54F-4502-941B-3F9C0F573A39__GUID-40C60643-A2EB-4B05-8927-B51AF7A6CC5E diff -Nru cloud-init-23.3.3/doc/rtd/reference/faq.rst cloud-init-23.4.3/doc/rtd/reference/faq.rst --- cloud-init-23.3.3/doc/rtd/reference/faq.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/faq.rst 2024-02-02 21:21:52.000000000 +0000 @@ -81,16 +81,15 @@ Can I use cloud-init as a library? ================================== -Yes, in fact some projects `already do`_. However, ``cloud-init`` does not -currently make any API guarantees to external consumers - current library -users are projects that have close contact with ``cloud-init``, which is why -this model currently works. - -It is worth mentioning for library users that ``cloud-init`` defines a custom -log level. This log level, ``35``, is dedicated to logging info -related to deprecation information. Users of ``cloud-init`` as a library -may wish to ensure that this log level doesn't collide with external -libraries that define their own custom log levels. +Please don't. Some projects `do`_. However, ``cloud-init`` does not +currently make any API guarantees to either external consumers or out-of-tree +datasources / modules. Current users of cloud-init as a library are +projects that have close contact with ``cloud-init``, which is why this +(fragile) model currently works. + +For those that choose to ignore this advice, logging in cloud-init is +configured in ``cloud-init/cmd/main.py``, and reconfigured in the +``cc_rsyslog`` module for obvious reasons. Where can I learn more? ======================= @@ -134,7 +133,7 @@ .. _IRC channel on Libera: https://kiwiirc.com/nextclient/irc.libera.chat/cloud-init .. _Juju: https://ubuntu.com/blog/topics/juju .. _discourse page: https://discourse.charmhub.io -.. _already do: https://github.com/canonical/ubuntu-pro-client/blob/9b46480b9e4b88e918bac5ced0d4b8edb3cbbeab/lib/auto_attach.py#L35 +.. _do: https://github.com/canonical/ubuntu-pro-client/blob/9b46480b9e4b88e918bac5ced0d4b8edb3cbbeab/lib/auto_attach.py#L35 .. _cloud-init - The Good Parts: https://www.youtube.com/watch?v=2_m6EUo6VOI .. _Utilising cloud-init on Microsoft Azure (Whitepaper): https://ubuntu.com/engage/azure-cloud-init-whitepaper diff -Nru cloud-init-23.3.3/doc/rtd/reference/network-config.rst cloud-init-23.4.3/doc/rtd/reference/network-config.rst --- cloud-init-23.3.3/doc/rtd/reference/network-config.rst 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/doc/rtd/reference/network-config.rst 2024-02-02 21:21:52.000000000 +0000 @@ -16,6 +16,12 @@ - **Kernel command line**: ``ip=`` or ``network-config=`` +Cloud-init will write out the following files representing the network-config +processed: + +- :file:`/run/cloud-init/network-config.json`: world-readable JSON containing + the selected source network-config JSON used by cloud-init network renderers. + User data cannot change an instance's network configuration. In the absence of network configuration in any of the above sources, ``cloud-init`` will write out a network configuration that will issue a DHCP request on a "first" @@ -297,7 +303,7 @@ .. code-block:: - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # BOOTPROTO=static DEVICE=eth7 @@ -306,7 +312,7 @@ ONBOOT=yes TYPE=Ethernet USERCTL=no - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp DEVICE=eth9 diff -Nru cloud-init-23.3.3/doc/summit/2023_summit_shared_notes.md cloud-init-23.4.3/doc/summit/2023_summit_shared_notes.md --- cloud-init-23.3.3/doc/summit/2023_summit_shared_notes.md 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/doc/summit/2023_summit_shared_notes.md 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,908 @@ +# Cloud-init summit digest + +Hello cloud-init community! + +We wanted to provide a summary of the notes from the August 2023 cloud-init +Summit in a more digestible fashion so that those without Google accounts can +also review the meeting notes to get up to speed on the conversations and +topics covered at the event. + +We will soon link event highlights, presentation materials and maybe a couple +of videos of presentations on our documentation site at: +https://cloudinit.readthedocs.io/en/latest/development/summit.html + +For those who weren't able to attend, this meeting was Canonical's first +hybrid attendance summit, with both physical and virtual participants. +The running notes are transcribed from a shared Google document to provide +simpler access. The original shared notes +[are also available](https://docs.google.com/document/d/1unINhucL6wcn7xqIL6IIAik8nwuacq5d-u0e3K0437s). + +## Our thanks + +First of all, many thanks to our sponsors for the event: + +- Microsoft Azure folks for providing meeting rooms, video conferencing and + lunches/snacks for both days at their Redmond Campus. +- Canonical for sponsoring travel and the group dinner on Wednesday night. +- The presenters who made this event possible by providing excellent content, + demos and discussions. + +## Topic schedule + +Below is an ordered list of scheduled topics that were covered, and the +presenters who led those efforts: + +### Summit day 1 + +1. **Presentation** Recent Features / Roadmap / Q & A: Brett Holman (Canonical) +1. **Presentation** Intro to our new technical author: Sally Makin (Canonical) +1. **Presentation** FreeBSD State of the Union: Mina Galic (FreeBSD Foundation) +1. **Presentation** Status of cloud-init versions and support in various + distribution downstreams: Chad Smith (Canonical) +1. **Roundtable** Python version support matrix and deprecation plan: James + Falcon (Canonical) +1. **Roundtable** Status of distribution downstream patches: Brett Holman + (Canonical) +1. **Roundtable** Status of testing and/or publication of cloud-init in various + OSes: Chad Smith (Canonical) +1. **Demo** Integration testing updates for pycloudlib: James Falcon +1. **Presentation** Security policy overview: Chad Smith +1. **Demo** Ubuntu live installer: Dan Bungert (Canonical) +1. **Demo** Canonical CPC, test framework John Chittum (Canonical public cloud + Manager) +1. **Demo** Cloud-init schema validation and validation service: Alberto + Contreras (Canonical) + +### Summit day 2 + +13. **Presentation** Cloud-init and Alpine: Dermot Bradley (AlpineLinux) +1. **Presentation** Documentation overhaul and policy for cloud-init (Sally + Makin) +1. **Roundtable** Boot speed in cloud-init: Chad Smith, Alberto Contreras, + Catherine Redfield (Canonical) +1. **Demo** instance-id in LXD: James Falcon (Canonical) +1. **Demo** Cloud testing tools in the cloud-init ecosystem: Chris Patterson, + Ahn Vo (Microsoft) +1. **Roundtable** Rotten tomatoes: what can the cloud-init project do better? +1. **Discussion** ISC-dhclient deprecation status: Brett Holman +1. **Breakout sessions** With Oracle and AWS: Fabio Martins and Kyler Horner + (Canonical) + + +Many thanks to all of our presenters and participants! + +Below is the wall-of-text transcribed from our shared meeting notes. A response +will also be provided to this thread when we have published our distilled trip +report to our documentation at https://cloudinit.readthedocs.io. + +Happy hacking, + +Chad Smith + +cloud-init upstream developer + + + + + +# cloud-init summit August 2023 notes + +## Attendees + +The cloud-init summit had representation both in-person and remotely from the +following community members: + +| Name | Company/Project/Role | Attendance | +| ------------------ | ---------------------------------- | ----------------- | +| Alberto Contreras | Canonical / Server-CPC / SW Eng | In person | +| Brett Holman | Canonical / Server / SW Eng | In person | +| Calvin Mwadime | Canonical / Server-CPC / SW Eng | Remote | +| Catherine Redfield | Canonical / Server-CPC / SW Eng | Remote | +| Chad Smith | Canonical / Server / SW Eng | In person | +| Christian Ehrhardt | Canonical / Server / Manager | Remote | +| Daniel Bungert | Canonical / Foundations / SW Eng | In person | +| Fabio Martins | Canonical / Support Eng | In person | +| James Falcon | Canonical / Server / SW Eng | In person | +| John Chittum | Canonical / CPC / Manager | In person | +| Kyler Horner | Canonical / Support Eng / TAM | In person | +| Sally Makin | Canonical / Server / Documentation | Remote | +| Deepika Rajadesingan | Oracle | Remote | +| Eric Mackay | Oracle | In person | +| Guillaume Beaudoin | Oracle / Platform images | In person | +| Harsha Koonaparaju | Oracle | Remote | +| Paul Graydon | Oracle | In person | +| Rajesh Harekal | Oracle | Remote | +| Anh Vo | Microsoft / Azure Linux | In person | +| Chris Patterson | Microsoft | In person | +| Lexi Nadolski | Microsoft | Remote | +| Minghe Ren | Microsoft | In person | +| Moustafa Moustafa | Microsoft | Remote | +| Nell Shamrell-Harrington | Microsoft / Rust Foundation | In person | +| Peyton Robertson | Microsoft | In person | +| Huijuan Zhao | Redhat | Remote | +| Wei Shi | Redhat / CentOS | Remote | +| Yaju Cao | Redhat | Remote | +| Bastian Blank | Amazon / Debian | In person | +| Frederick Lefebvre | ASW / Amazon Linux | In Person | +| Noah Meyerhans | Amazon | In Person | +| Andrew Jorgensen | GCP / Guest Ecosystem / Staff Eng | Both | +| Justin Haynes | GCP | In person | +| Mina Galić | FreeBSD Foundation / maintainer | Remote | +| Dermot Bradly | AlpineLinux / Maintainer | Remote | +| Robert Schweikert | SUSE / Distinguished eng | In person | +| Matthew Penington | AWS / EC2 Commercial Linux/SDE | In person | +| Matt Chidambaram | Cloud Migration/ DevOs SRE | Remote | +| Kristjan Onu | National Research Council Canada | Remote | + +## Day 1 running notes, per session + +### Presentation: recent features / roadmap / Q and A + +Presented by: Brett Holman (Canonical) + +* [Slides: cloud-init summit intro](https://docs.google.com/presentation/d/1i4ufgXcfQ9rQLIfgeqMMkr6w5Bpu-RrhAaPTxo0BofM/edit?usp=sharing) +* CVEs, code changes, new clouds added and config modules provided +* Versioning policy: time-based versions `YY.` +e.g. 22.1, 22.2 etc +* Feedback / discussion: + * NoahM: New docs -- I can now quickly navigate to solutions + +### Presentation: Intro to our new Technical Author + +Presented by: Sally Makin (Canonical) + +* [Slides](https://docs.google.com/presentation/d/12ZFawGOxlzg_J5bYtcE_anYbNFiFVHb2xFTyOQy9rio/edit?usp=sharing) +* [Recording](https://drive.google.com/file/d/1v23sXtyo4AILTyqRIH2CtVkl5zVgi7oY/view?usp=sharing) + +### Presentation: FreeBSD State of the Union + +Presented by: Mina Galic (FreeBSD Foundation) + +* [Talk notes](https://scratchpad.pkgbase.live/q0Ajf3LtTaqXXH0v99Sykg?view#) +* At the start of this project there were 130 failing tests on BSD. +* Since FreeBSD 13.2 release, unit tests are being fixed upstream. +* Biggest issues: + * Not being able to test on your target platform has been a real challenge. + * Trying to ensure we have fully integrated cloud-init CI within FreeBSD's + test infrastructure. +* Looking to get cloud vendors to provide official FreeBSD images with + cloud-init installed. +* Mina has been driving integration tests working on LXD platform for ease of + free and local integration test runs. + * LXD Virtio vSocket support complexities are causing trouble for BSD due to + the lack of a vSocket module. + * BSD has a Hyper-V module, but no vSocket. + * Spent time trying to convert Hyper-V socket module to support vSocket. +* Network refactors that resulted in InfiniBand support for cloud-init. Tabled + that effort due to lack of resources, testing and expertise. + * What is missing from BSD for network refactor: + * Missing a lot of IPv6 support; progress is slow but outside contributors + have added a lot of functionality. +* Two big issues with BSD: filesystem hierarchy standard (FHS) + * Every Unix has its own FHS standard. + * BSD use of `/run` is actually written to the root filesystem (it's not + ephemeral as cloud-init treats it). + * Need to push distro-specific treatment of ephemeral filesystem. +* AIX has announced that they would like to join the cloud hype. +* The hope from Mina's work has paved enough of a path to [sic] + +### Presentation: Status of cloud-init versions and support in various distribution downstreams + +Presented by: Chad Smith (Canonical) + +* Refresh our understanding of downstream support matrix lifetime. +* Review Ubuntu, Debian, openSUSE, FreeBSD, Alpine, Amazon Linux. +* Are distributions pushing updates into stable releases? + * Ubuntu pulls the latest release to stable releases: 20.04, 22.04, 23.04 + * Canonical provides ESM (Expanded Security Maintenance). + * cloud-init +* Frederick: Can cloud-init make a backwards compatibility for release? + * Define with labels whether changesets are backward compatible or not +* Noah: cloud-init is not consistently clear whether interface changes or + behavior changes, so downstream consumers can better assess risk profile. + * [Concern]: Don't have a defined policy for determining whether we will + pull the latest cloud-init releases into stable releases on Debian/Amazon + Linux. + * Frederick: Lack of a contract is what hinders us here, if we can be more + strenuous for our [sic] +* James: Ubuntu tries to keep behavior consistent on stable releases, and has + downstream debian/patches to hold stable behavior. +* ChrisP: Azure finds the update process valuable especially in relation to + their datasource. +* Chad: This release process is a large burden on the project. +* John: New features on ESM Ubuntu versions? Divergence between cloud-init + versions. +* Noah: Amazon Linux and Debian try to suggest to customers to migrate to newer + releases, instead of trying to backport those features to stable releases +* NoahM: Would it make sense to generalize the SRU process more for other + distros to better engage a broader group of contributors and drive quality? +* James: From the Ubuntu perspective we have a history of finding regressions + after each SRU. This may impact partners if we require a vote on releases to + ensure stable release updates are 'approved' for various downstreams. +* Noah: Try leaning toward pushing some of the SRU. +* John C: Have clouds/partners/distros publish/queue testing. +* Robert S: We could ask clouds to sponsor accounts for distribution testing + * Want to drive ability to [sic] +* [ACTION] Releases: document and downstream patches we apply +* Noah M: do you have a separate policy for how you approach those downstream + patches +* [ACTION] cloud-init PRs that break compatibility add a new 'breaking-change' + label. + * Alberto: Also in the commit message? + * John C: Potential templates in commit messages indicating breaking change. + * John C: [Example of convention](https://www.conventionalcommits.org/en/v1.0.0) +* Robert S: Need more conspicuous [breaking change] in release notes for the + release. + * Want this in the release changelog or in git commit messages as metadata + -- it raises awareness. +* [Quotable]: "You have to deal with me if you want the stuff" -- Robert S + * Robert S: Can we add a prefix in commit message metadata to differentiate + "bug fix" from "feature". + * [Outcome] Multiple people preferring changelog route +* SUSE patches are found here: https://build.opensuse.org/package/show/Cloud:Tools/cloud-init + * Is https://repology.org/project/cloud-init/versions up to date? +* [ACTION] Community, Partners Clouds and Distro maintainers please update the + support matrix by end of cloud-init summit with the latest details of what + version of cloud-init is where, and whether that release receives updates. + +| Distro/Release | Cloud-init version | Receives cloud-init updates | End of Life | +| -------------------- | ------------------ | --------------------------- | ---------------- | +| Ubuntu 16.04, 18.04 | 21.1-19, 23.1.2 | CVE / Security | Depends on ESM | +| Ubuntu 20.04, 22.04, 23.04 | latest | yes | Not EOL | +| SLE 12 | 20.2 | Only CVE fixes | Oct 31st 2027 | +| SLE 15 | 23.1 | Yes | Jul 31st 2031 | +| openSUSE Leap | 23.1 | Yes | Not EOL | +| openSUSE Tumbleweed | 23.1 | Yes | Never | +| Oracle Linux 7 | 19.4 | CVE / Security | Dec 24 + depends | +| Oracle Linux 8 | 22.1 | | Jul 29 + depends | +| Oracle Linux 9 | 22.1 | | Jul 32 + depends | +| RHEL 8 | 23.1 | CVE / Security | | +| RHEL 9 | 23.1 | CVE / Security | | + +#### Release patch links + +* [Ubuntu 20.04](https://github.com/canonical/cloud-init/tree/ubuntu/focal/debian/patches) +* [Ubuntu 22.04](https://github.com/canonical/cloud-init/tree/ubuntu/jammy/debian/patches) +* [Suse Patches](https://build.opensuse.org/package/show/Cloud:Tools/cloud-init) + +### Roundtable: Python version support matrix and deprecation plan + +Chaired by: James Falcon (Canonical) + +* Record current Python versions under long term support by various + distributions versions. +* Ubuntu: LTS support matrix. +* Policy for deprecation. +* What keeps you on an older version? +* SUSE freezes once the distro hits stable. + * openSUSE 15 concluded that Python 3.11 and all dependencies [sic] +* Chris P: Why do you want to update to newer python version? + * James: Newer features provided. +* John C: how to add new Python features and maintain ESM / LTS support? +* Fred L/Noah M: The cost of moving to newer Python versions is "beyond + annoying" -- it would be helpful to take into account the costs of that + shift to newer features as it forces downstream work. +* Noah M: If there are critical security fixes added to upstream, is it + possible for ease of downstream adoption of security features? +* Robert S: How do we test this across multiple versions when we end up + developing with our local Python version. This leads to gaps because we are + human. Does this come down to testing to assert? + * How can we integrate unit tests so that each flavor and/or distribution can + be exercised everywhere to assert we don't have regressions? +* Robert S: I still pull in new major versions of cloud-init into supported + versions (including 3.6). + * "I have no fear", update twice a year. + * It's not possible to test everywhere, OpenStack is one of those cases. + * Customers are very helpful in that regard as they will prefer to test + newer versions and collaborate with SUSE on verification. +* Robert S: Potential pivot to Python 3.11 in 9 months. +* Mina: 3 supported BSD releases (12.4, 13.2. 11.X) have been EOL for a few + months now: + * Next release is 14, coming out as soon as OpenSSL integration is completed + and current LLVM release 16. + * 3rd party software: all FreeBSD releases contain the same version of + cloud-init. So latest cloud-init releases are published to all active + supported releases. + * Also, there is a quarterly release of those ports to provide "stable + adopters" with only bug and security issues being fixed. + * Bleeding edge version of cloud-init is available in the form of the + `net/cloud-init-devel` port. +* Chris P: How many distros actually pull regular new upstream cloud-init + versions which may need additional testing? + * Noah: Debian don't currently regularly pull cloud-init into supported + releases -- only backport when we need to. + +| Distribution/Release | Python version | End of life | Takes upstream release | +| -------------------- | -------------- | ----------- | ---------------------- | +| Ubuntu/Bionic (18.04)| 3.6 | ~2030? | Security only | +| Ubuntu/Focal (20.04) | 3.8 | ~2025 | Yes | +| SUSE | 3.6 | ~9--12 months | | +| openSUSE 15 | 3.6 (soon 3.11)| Not EOL | Yes | +| Mariner 2 | 3.9 | No EOL | Yes | +| RHEL 8 | 3.6 | May 2024? | Python 38 available but not default | +| Amazon Linux 2023 | 3.9 | ~2028 | Not systematically | + + +### Roundtable: status of distribution downstream patches + +Hosted by: Brett Holman (Canonical) + +* [REQUEST] Looking for public download patches that partners are willing to + review. +* Chris P: Azure Mariner team carries downstream patches. +* Amazon Linux has private patches that it may make sense to upstream -- it's a + [ACTION]. +* Cloud-init upstream wants to avoid breaking private downstream patches if/when we change internal APIs or function signatures. +* Robert S: + * cloud-init documentation would be helpful to generate a list of what + distributions. + * James: We already generate distribution support in the rendered docs. + * Noah: `cloud-config.tmpl` does also override and filter out certain modules + +### Roundtable: status of testing and/or publication of cloud-init in various OSes + +Hosted by: Chad Smith (Canonical) + +* What gates publication of cloud-init on each distribution? +* Review Ubuntu, Debian, openSUSE, FreeBSD, Alpine, RedHat if possible. +* Noah M: May not be giving away that we are not running integration tests locally. + * We rely on upstream testing behavior during releases for quality + * Debian: Unittests run as part of the package build process. + * Existing upstream integration tests not run in Debian or Amazon Linux right + now. + * Amazon Linux has similar integration-type tests -- realistically, + can/should be converted to [sic] + * Different framework for testing packages in the distribution in general: + * Possible refactor of Amazon Linux could be reasonable for [sic] + * May be worthwhile to investigate running integration tests with Debian + and Amazon Linux. +* Chad: Better documentation would take it a long way. +* Chris P: We rely on cloud-int upstream testing to get the breadth of testing + for broad features. + * Test with cloud-init upgraded in images at scale, to check for regressions. + * Looking for checkbox for release. + * Noah M: are you testing cloud-init features specifically or using existing + test infra to see if you have regressions related to [sic] +* Chad: So we have people leverage existing testing as well as people testing + things more pertinent to them. +* Guillerme: Oracle have general infrastructure tests that watch for + regressions with new cloud-init images. +* Chad: We want to make it easy to let others run integration tests while also + getting that feedback from other platforms. Are there any public endpoints + that upstream can consume? +* Dermot: What about cloud providers provide a "stock metadata" that allows + providers to provide emulated content to mock the IMDS? + * Robert: Once we mock IMDS, there are other platform services. + +### Demo: Integration testing updates for pycloudlib + +Presented by: James Falcon (Canonical) + +* [Slides: Integration testing updates](https://docs.google.com/presentation/d/1e5p9FW9EI9FxnEyLPhpIA79GVGdyQ2aIVyX8Ggb29L8/) +* Generalized for all clouds. +* Noah M: How do you get the new version of cloud-init on the image under test? + * James: We start from a public image -> + * Launch image -> + * Upgrade cloud-init -> + * Run `cloud-init clean --logs` +* Pycloudlib: + * Most Linux distributions are supported given the ability to discover. +* James: + * Per-PR testing CI runs on Ubuntu on LXD. + * Why would CI [sic] + * Jenkins daily runners give a healthy signal on GCE, EC2, Azure. + * For Rust project: they talked directly to GitHub about increasing governor. + * Chris P: Is there a mechanism to validate certain PRs on Azure? + * In Rust: Bohrs-bot in GitHub to kick off conditional jobs: + * https://forge.rust-lang.org/infra/docs/bors.html + * Fork of https://github.com/rust-lang/homu +* Chris P: Is Jenkins publicly accessible? + * Chad: Nope -- security issue, due to malicious plugins. + * Chris: We would like to trigger specific test runs on Azure. +* Ahn: Do you use merge-queue? To queue CI run after Merging? +* Mina: How has the move from BZR -> GH changed your workflow? + * It feels like there is a stumbling block in how pull requests are being + designed given that some workflows are not ideal for some teams. +* James: I don't think it's realistic for cloud-init upstream to test all + distros in our Jenkins test runner. + * Chris P: What is the blocker to that: cost? + * John C: Some testing accounts are tied to costs and accounts that are + sponsored. + * Chris P: If we can sort cloud accounts/costs, would that provide us with + testing? + * John C: Looking at our matrix, we have to understand the depth we want in + the support matrix -- of how many instance types/sizes. + * Discussions with cloud platforms are needed to determine what instance + type/distro matrix is "good enough". + * Robert S: This matrix only continues to grow and is not sustainable for all + clouds and all distros. It's a combinatorial problem. + * We want the ability to specialize tests to a given cloud based on where the + changes are scoped (per datasource/per distro) to reduce test matrix costs + and make testing sustainable. +* Chris P: If you see upstream tests, does that give you a sense of certainty + that allows you to avoid certain tests? + * Robert S: I just take it and accept the work being done upstream and say + "thank you". +* Robert S: pycloudlib has a dependency problem that makes it hard because it + pulls in all Botocore Azure SDK etc. to support all platforms. + * For some projects in SUSE they have conditional build flags that + limit/segment the dependencies to make it easier to only depend on certain + SDKs. +* Chad: It's possible to add more segmentation in our tests, but would be + valuable. +* Waldi: Debian uses Apache Libcloud. +* Frederick L: There are a number of moving pieces trying to track with the + test matrix. Clouds should be best placed to define proper testing. + * Clouds should do more of the testing as they are best placed to do that + testing. +* John C: Sounds like it talks around a web service that would allow for + posting test results. + * Mina: Can we establish a way to trigger specific builds on Azure on a + per-PR basis? +* Robert S: We have OpenQA, but there's still the issue of cost. + * If upstream works on documentation for integration testing per + distribution, downstreams can consume that and try to leverage those docs + to make per-distro test runs. + * If we have a 30-step procedure for how best to integration test. + * Might want to solve a small problem to define manual steps to enable + downstreams to better integration test, allow developers to enable + integration-tests on their distro/cloud rather than trying to define + bigger funding/policies and public services. + * If it's relatively easy for SUSE to kickoff and run tests, but we'd want more documentation on the procedure for setup and integration test runs +* Takeaways: + * [ACTION] Better docs for integration-tests to onboard other distros/clouds for + testing. + * [ACTION] Improve visibility to upstream test status. + * [ACTION] Microsoft or other clouds engaged in integration-testing should + determine (per-cloud/distro) what makes sense for leveraging where partners + are investing their development time. + * Entire test matrix is likely too large for any single entity to test + everything. +* Robert S: "This is an fdisk problem; this is not my problem". + +### Presentation: Security policy overview + +Presented by: Chad Smith (Canonical) + +* For upstream GitHub pull requests that warrant a security review, add the + label "security". +* [Overview for filing new security issues](https://github.com/canonical/cloud-init/security/policy) +* Either: + 1. Send an email to `cloud-init-security@lists.canonical.com` reporting the + security bug, or + 1. [File a bug](https://bugs.launchpad.net/ubuntu/+source/cloud-init/+filebug) + and mark it as "Private Security". +* After the bug is received, the issue is triaged within 2 working days of + being reported and a response is sent to the reporter. +* The `cloud-init-security@lists.canonical.com` is private. +* Any vulnerability disclosed to the mailing list or filing a private security + bug should be treated as embargoed, since any affected parties will + coordinate a reasonable disclosure release date: + * Disclosure date is based on bug severity, affected party development time + for the release and fix publication +* If a CVE is warranted, the Canonical security team will reserve a CVE ID that + will be represented on the bug and the published bug fix. +* At the disclosure date, an email is sent to `cloud-init@lists.launchpad.net`. +* Review new or dropped contacts needed for security notifications? + [REDACTED emails] + +### Demo: Ubuntu live installer + +Presented by: Dan Bungert (Canonical) + +* [Slides: Subiquity, Autoinstall, and Cloud-init](https://docs.google.com/presentation/d/1Qd3soaBnbz0f3zmtp1IXMotwAqsUW0Llvx5dTbCW_7g/edit#slide=id.g239da370174_0_22) + +### Demo: Canonical CPC, test framework + +Presented by: John Chittum (Canonical public cloud manager) + +* [Slides: Ubuntu Build And Test CloudInit](https://drive.google.com/file/d/1gwrHqpXB9gpY_ddx38jxRY81QDKAV0uu/view?usp=sharing) + + +### Demo: Cloud-init schema validation and validation service + +Presented by: Alberto Contreras (Canonical) + +* [Slides: cloud-init schemas](https://docs.google.com/presentation/d/1Pn2I3dWw4rzbZWxuuU9mTrtbhlJonq3NsX0DLAztkK4) +* Minimal: Some distros have different schemas, how to deal with that? +* Chad: Separate schema files? +* Robert S: This is going to be an "after the fact thing". Users are not going + to use it. +* [ACTION] Print schema warnings / errors to the console / journald: More + visibility. +* Chris P, Robert S, AWS: Hard-error on schema errors is very valuable. +* [ACTION]: Do not show subsequent error traces in the case of a non-valid + cloud-config. +* Noah M: I want hard errors at console for schema validation problems. + * Dermot: Some cloud providers don't give console support by default, so + expecting the "error" output console being your primary means of + communication will not help certain platforms. + * Chris P: Customize strict failures depending on use-case and clouds. + * [ACTION] Investigate providing an option to make schema errors a hard + error at system boot that is configurable by cloud/distro image + creation. Imagine `error_handling` config keys. + +## Day 2 running notes, per session + +### Presentation: cloud-init and Alpine Linux + +Presented by: Dermot Bradley (Alpine Linux) +* [Slides: cloud-init and alpine](https://drive.google.com/file/d/1S1p5d-8aC36H2WPJuE84OK1ynZWv4cfj/view?usp=drive_link) +* 3 years package maintainer for cloud-init and cloud-utils. +* Musl vs. glibc (developer focus is POSIX-compliance, reluctance to + non-POSIX compliance) any strong non-POSIX features devs rely on make this a + blocker to Alpine devs due to Musl dependency. +* Non systemd, busybox's init and [sic] +* Alpine doesn't "not like" systemd, but systemd developers don't support + alternatives to glibc so it makes systemd support tough. +* Alpine not udev, they use mdev. Initframfs uses mdev, so udev support, while + packaged, may cause config friction with other parses of the busybox init + stack. +* `sudo` is packaged but not used in favor of `doas` due to some of the fairly + frequent security concerns with `sudo`. +* Nocould-net writes to ENI and seeing secondary writes of ENI config files + that don't effectively ifdown the previous temporary interfaces. +* Renderer issues with multi-IPv4 addresses showing up in wrong net config + section, likely problems with network config/internal conversion logic. +* Fix hook-hotplug to avoid hadcoding bash for the script. +* Fix lock_passwd: Openssh config: password: * behavior is difference in BSD + vs. Linux; "special cased" due to PAM enabled. + * If PAM is not enabled, * means locked. + * Alpine has PAM/Kerberos/other: + * Alpine has to build images with PAM-enabled otherwise passwords [sic] +* Problems: + * Alpine has to grab latest Python for edge repo. + * Edge repo moves to latest Python 1-2 days for new Python version. + * Could create a window where cloud-init could be broken. +* Future work: + * Alternatives to SSH: dropbear and tinySSH. + * `cc_wireguard` for Alpine, non-Ubuntu. + * Improve console output layout to make it 80 chars wide given IPv6 + output/routes, line-wrap etc. +* [Tooling Dermot uses for creating images](https://github.com/dermotbradley/create-alpine-disk-image) +* Questions: + * Is it worth calling out specific distribution concerns/dependencies in + cloud-init docs (mdev/busybox/non-bash/doas concerns)? + * Noah M: do you need to make it granular per distro: POSIX/non-POSIX + approaches? + * Dermot: May not need much in the way of docs. + * For systemd/non-systemd environments, Alpine isn't alone, Gentoo is in + that, Gentoo directory provides cloud-init code drop-ins. + * Checked in IRC channel to get an impression from the project, didn't + get a handle on the project stance. Opinion is that maybe Alpine is the + only distro using `init.d`. + * Brett H: There was an effort recently -- [Devuan](https://www.devuan.org/) + attempted to add cloud-init and they didn't try to upstream it, COS + (Google) may be using OpenRC, carry sid? + * How do you, as Alpine maintainer, track new issues/features/development or + what needs/ongoing work? + * Prioritization is up to Dermot as maintainer. + * Alpine has official cloud-init images for AWS. + * [Tinycloud](https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud) that core Alpine developers use. + * [ACTION - done] Dermot to provide links to "official images" in clouds or public + repository/website/service: [Alpine Linux website and downloads](https://alpinelinux.org/cloud/) + * Cost prohibitive to host "official" images in multiple clouds etc. + * Some talk about making official images on GCP and other clouds. + * One person paying for all clouds. + * Noah M: talk to Debian publishing from, AWS account is sponsored by + AWS... there are similar accounts. + * [ACTION] Noah M can help Dermot get in touch with right people for + open source engagement to start talks about open source sponsor + accounts for hosting "official images". + +### Presentation: Documentation overhaul and policy for cloud-init + +Presented by: Sally Makin (Canonical) + +* [Slides](https://docs.google.com/presentation/d/1aULy2jy6yil-wncMLLcHYWsDbFwRjS9aVOvxH-UQk08/edit?usp=sharing) +* Mina: I've seen some people complain that the style is a bit hard to read... + that the fonts are too thin and light -- maybe try Atkinson Hyperlegible. +* Kyler just opened an issue on the docs for how arrow navigation works (module + page, not working as expected). +* Robert S: https://cloud-init.readthedocs.io is distro agnostic, people + discovering docs may get the opposite impression with Canonical/Ubuntu in the + URL -- others may be concerned: + * What do other distro downstreams do? Reflect docs? + * Sally: https://readthedocs.com requires using the company name in the slug + as it's pulled from the repo. I asked RTD support if we could get rid of + that, they said absolutely not. +* Mina suggests https://cloud-init.io (+1 from Noah): + * [ACTION] Access may be an issue, Sally will investigate + +### Roundtable: boot speed in cloud-init + +Chaired by: Chad, Alberto, Catherine Redfield (Canonical) + +* [Slides for discussion points](https://docs.google.com/presentation/d/1OunV5fXFBkxg61y4wzApNmpEbmqIUlLpA9DQ92g7D0Y/edit#slide=id.g25f04f72223_2_0) +* Robert S: Ran time to SSH tests on SUSE years ago. GCE: 38.5s EC2: 52.2s + Azure: 104s. After time to SSH, rest of come up is reasonably consistent. + Does time cloud-init takes make a big difference? +* Noah: These numbers have come way down. Cloud-init is now a bigger part of + it. EC2 has made Amazon Linux changes: IO is a really significant contributor + when IO is cold. Reducing data loaded from disk helps a lot. E.g., reduce + size of initrd. All modules and dependencies being loaded contributes a lot. +* Mina: Firecracker is the biggest one contributing to FreeBSD. + * MicroVM hypervisor: AWS open source project, EC2-specific hypervisor + * Optimizing BSD for Firecracker leads to approaches that could be applicable + to generic hypervisors (Colin Percival) ec2-boot-bench for getting + fine-grained launch time instruments (time to network response, port 22 + open etc). +* Chad: Agreement that time to SSH/network availability on first boot is the + priority. +* Andrew J: Time to SSH may not be what all users actually care about, but it's + what Gartner and others measure when they compare clouds, so it matters + anyway, and it's not a bad proxy measurement. +* EC2 -- vast majority of instances don't reboot, they are torn down after + they're no longer used, upgrade to new instances. +* John C: "time to workload" suggestion from some clouds, but "what workload" + is the important question... hard to define besides just treating SSH. +* Dermot: Typically running chrony (waits til time sync) impact boot time, + individual [sic] +* Mina: Some large cloud-init scenarios include reboot to ensure everything + comes up. +* John: In Kubernetes, time to serve container is most important. +* Noah: Early init might not be as small as you think. Especially BIOS boot. + * 13 CPU instruction 508 bytes of disk at the same time lots of I/Os -- that + significantly impacts cost to boot. + * Less pronounced in UEFI. +* Chad: Compression has a large effect on initrd time. Tradeoff between image + size and speed. Also, complexity of storage and networking stack. Can be 2+ + minutes for very complex instance types (back to slides so not taking notes + on that). +* Dermot: Does GRUB currently make use of any seed passed by UEFI? That would + speed up initial entropy seeding. Re: stage 3 improvements. +* Noah: Time spent generating SSH host keys. +* Noah: Can we reduce IMDS keys and limit the initial crawl footprint for + init-local? + * Can we ask for a single JSON representation of critical config data. + * Guillaume: We might not get a warm initial response but if it's never + asked for it will never be put into any roadmap either. + * CatRed: To answer your initial question, we know what minimal keys we need, + * To ask for individual keys, it may make sense to look at a single URL to + obtain that data. + * Could we ask for "cloud-init-local" URL from IMDS to seed content? + * Noah M: Early boot keys is not necessarily cloud-init-specific so it may be + easier to define a generalized "early boot config" that isn't + cloud-specific. + * Noah M: The CPU cost associated with multiple requests amortized across all + of EC2 is likely to be very high; the potential for savings is also high + and therefore may be justified in order to improve boot time, improve + performance, and reduce costs. + * Also consider early boot keys in non-network locations (e.g., SMBIOS). +* Dermot: Also look at caching opening the JSON schema file. +* Chad/John: Avoid cost of ephemeral DHCP by keeping the IMDS data local to + the instance rather than using network (also cuts down security concerns). +* Chad: Considering pre-compiled binary too, but that would be a big + effort/cost. +* Noah: Rather than re-initializing everything, can we use a persistent + process? + * Investigated a signal/flag condition that'd avoid loading Python in later + boot stages to avoid the Python initialization cost. + * Noah has seen hundreds of milliseconds for Python spin-up. + * Chad: Also an option to have a smaller cloud-init with reduced stages. +* Biggest question around removing host key types: are certain ones required + for compliance? In Amazon we had to re-enable older types for FIPS. +* Chris: On some distros, SSH starts early and then needs to restart after SSH + module. If the module moved earlier, we might be able to cut the restart. +* Robert: For use cases that need large scale, you'll probably have a minimal + cloud-config, so we probably shouldn't worry about efficiencies in the later + modules. +* Dan: We've had a few bugs around implicit ordering cycles. +* Chad: If we change/remove waiting for snap seeding, there's no indication + that cloud-init is actually waiting for snap seeding. +* Minimal cloud-init in a precompiled language. + * Nells: Would definitely be interested in collaborating should Rust be + pursued. +* [ACTION] John: Get from clouds/partners: what is the minimal system/features + that partners care about. + * User creation, SSH key setup, what kind of network config is needed. + * E.g., smart NICs, appliances, or immutable apps/solutions. +* Robert: Images with cloud-init vs. images with Ignition + Afterburn. On the + spot numbers for consideration: + * Data collected on EC2 on t2.micro in us-east-1. + * Instance of SLE-Micro which uses Ignition/Afterburn. + * Startup finished in 2.860s (kernel) + 12.437s (initrd). + * SLES 15 SP5 basically the same initrd (no Ignition/Afterburn). + * Startup finished in 2.716s (kernel) + 5.160s (initrd). + * Cloud-init in this instance (all services per `systemd-analyze`) 11.143s. + * Off this 9.248s cloud-init-local.service. +* Log levels: + * []: Noted that they do find the logs saying how many bytes are read -- + logs helpful for debugging. + +### Demo: instance-id in LXD + +Presented by: James + +* Followed by roundtable +* [Roundtable slides: instance-id](https://docs.google.com/presentation/d/1VYYKUErQcMiYLFN0Slisz-bYT8f_iskiu-bkyWrZp70/edit#slide=id.g2595c67feb8_0_0) +* Can we determine a consistent set of rules for when we get a new instance-id + from clouds and what it means to cloud-init for those cases? +* Noah M: If a host is rebooted and IMDS is unavailable, cloud-init (upon + seeing new instance ID) triggers the "re-run" behavior which is undesirable. + * If we have a way to see instance-id out of something like SMBIOS, it would + prevent unnecessarily triggering re-run from cloud-init. +* Chad: This is happening because datasource moved from "DataSourceEC2" to + "DataSourceNone". We can likely improve this behavior as instances usually + aren't being moved between clouds. +* Some clouds use regen network per boot: + * Amazon Linux regenerates network every boot and doesn't use cloud-init for + that. + * Frederick: Some customers felt they wanted to block IMDS access after first + boot for their use case. +* Mina: Any goal to make modules more idempotent? + * Chad: A good ideal and something that we can document as policy/intent for + both module creation as well as ongoing module review. + * [ACTION] follow-up to define policy or approach for better ensuring and + reviewing idempotent `PER_INSTANCE` defined `cc_*` module runs in light of + reruns will be idempotent. +* Is it useful to have a separate mechanism to define a scoped IS change to + limit scope of the config changes cloud-init takes (network vs. everything + vs. an individual config modules)? + +### Demo: cloud testing tools in cloud-init ecosystem + +Presented by: Chris Patterson and Ahn Vo (Microsoft) + +* We all build our own tools/testing/performance/reliability. +* Are there tools out there for perf and reliability that could be leveraged? + * We may all have downstream/specific tools to scrape cloud-init logs looking + at `cloud-init.logs` to scape WARNINGS/ERRORS from logs to determine [sic] +* Azure spends a lot of time to detect errors as discovered in-house, to define + dhclient failed to bring up networking in PPS stage. +* If there were a repo where partners can provide their `regex` for log + analysis, to determine failures and watch for regressions. +* LISA tools do their own thing, we do our own thing. +* Anh: cloud-init collect logs; can we supplement or reduce the files that are + being collected, maybe with a manifest? + * Chad: sounds like a good feature request. + * [ACTION] file a bug for this feature for extending default behavior. +* Chris P: Are there scripts/checkpoints for regressions? + * Robert S: When I do cloud-init update to latest release I create new images + and launch on cloud X,Y,Z uses [sic] +* Chris P: are there plans to surface `cloud-init status --format=json`? + * Brett H: work in progress on warnings raised through cloud-init status + * There is a desire from Azure to also reflect schema warnings as more visible in console logs + +### Roundtable: Rotten tomatoes: what can the cloud-init project do better? + +Brainstorming session on areas for improvement, policy changes, automation, +feature sets, and communication changes that may help improve cloud-init as a +project / engagement from communities. Open season for suggestions, gripes, +improvements to be had! + +**How should we engage the community more? What blocks you?** + +* Former IRC meetings are no longer held, having synchronous meetings wasn't + supportable and significantly useful for broad timezone support. + * Mina: That status meeting was always at a terrible time... for people with + small children in Europe, anyway. + +**Preferred communication?** +* Noah: Mailing list + * Robert: Mail is good. Meetings were nice but too busy and not great time to + attend. Weekly summary would be nice. + * Noah: More than just `git log`, but a top level summary would be best. + * Maybe weekly link list of landed PRs, GH account name. + * [ACTION] Work with NellS (Microsoft) to better define weekly digests from + GitHub and reporting (to cloud-init mailing list) the landed PRs and + authors, maybe supplemental context. +* Robert: It'd be good to better highlight the problems/actions taken for a + CVE, especially when historical data is involved. + * [ACTION] review security policy for data we release to our security + partners. If we know we are dealing with distro-specific and packaging + concerns, it's best to provide that to partners during embargo time to + allow them to prepare. +* Noah M: Small request -- really dislike "squash merge" on all pull requests. + * It discards what I've done and the way I've architected the cohesive + changes and understanding the intent. Prevents my ability in the future to + understand the thought process for my changeset to bisect problems when + they show up. +* Mina: That sounds like what I was talking about yesterday with respect to the + PR workflow... https://jg.gg/2018/09/29/stacked-diffs-versus-pull-requests/ + * Stacked Diffs Versus Pull Requests | Jackson Gabbard's Blog + * A post from Jackson Gabbard on Jackson Gabbard's Blog provided by: + https://jg.gg + * Mina: "I'll just open more pull requests" -- that's really difficult to do + though, if they build on each other. +* [ACTION] Define policy or option for opting out of squash merges and what + expectations there are around pull requests that want to avoid squash merges, + and put it to the mailing list: + * Mina: If opting out of squash merges, we probably want CI to pass on each + commit. + * Noah M: Part of the value of preserving individual commits is that you can + use `git bisect` to check those commits individually and assert which + commit broke. + * Your individual commits shouldn't break the code so you need to + understand what went wrong. + +**Strong opinions around the mailing list vs. Discourse (forums)** + +* John C: The nice thing about Discourse is that you can set up digests that + act like mailing lists, Discourse provides the opportunity to dive deeper if + necessary. +* James: Appreciate the ability to subscribe to individual threads. +* Noah + Robert: Prefer email. +* Robert S: Document list of config modules per distro. + +### Roundtable: ISC-dhclient deprecation status + +Chaired by: Brett Holman (Canonical) + +* Review of changes that have happened in cloud-init for DHCP support. +* Questions/comments? + * Robert S: SUSE has an interest in getting rid of the dhcpclient, file a + request to make NetworkManager's dhclient callable from the command line. +* [INVESTIGATION] Dermot: Is IPv6 support considered in this + migration/deprecation?? +* Noah M: EC2 Amazon Linux: Talked about using a link local IPv4 address + statically, or randomly choosing the address to avoid round trips with DHCP + server, which is unnecessary. + * EC2 doesn't use IPv6 LL but rather Unique Local address. + * VPC == funny network, not proper layer2 network, don't have to worry about + address conflicts + * Brett H: Can be used as performance optimization but not cloud generic. + * Walkthrough of PRs to support this feature set over the last year. +* Robert S: Maybe we rip all it out and farm it out to the kernel, Oracle + requires network address in initrd. +* John C: initrd for Ubuntu is only a fallback mechanism if kernel itself does + not boot -- straight kernel == no initrd. +* Chris P: In most cases we are doing DHCP on boot, why not have that DHCP + config already in the image? + * For iSCSI solutions, initrd has to have network up to get hold of the + storage, so leverage it where necessary. + * And NetworkManager has a DHCP engine that is not exposed on the command + line -- might be able to get solutions that provide this. +* Amazon Linux: uses systemd-networkd. + +### Wrap-up / Thanks + +* Takeaways / roundtable / group retrospective on how the summit was organized. +* Physical summit location query + * Seattle most likely candidate for next summit. + * If European physical summit, may not have as much attendance from west + coast clouds. + * Merging cloud-init with UDS location as a separate conference? + * What could be better handled next summit: + * Communication? + * Planning of firm talk start/stop talk times Y/N? + * Types of talks, discussions, demos? + * Timing of talks/timezone? + * How to increase participation? + * How was remote accessibility? + * Should we open this up this event more broadly to the general community + for drop-ins via public mailing list post/IRC post of talk times? + * [YOUR_SUGGESTION_HERE] +* Would virtual conference get engagement? +* Project improvements + * Communication channels? + * Security publications? + * Release frequencies: ~time based quarterly + * Release hotfixes: patch release versioning on a 23.2.X pushed to GitHub, + email sent out with the update. + * [YOUR_IDEA_HERE] + * Action items, investigations and follow-ups. + * Continue to drive toward multi-distro/cloud integration-test adoption: + * Add more official docs around integration test procedure for a new + platform/distro. + +Potential breakout development sessions from earlier discussions: +* Nells: Bring RUST GitHub workflows and policy to cloud-init. +* Better enabling integration-test use in non-Ubuntu environments. + * Walk through procedure to get other distributions, drive live demo of + integration testing, propose doc updates for extending integration tests on + a new distribution. +* Strawman proposal for Schema warnings treated as hard errors (want + configurable setting in images for certain platforms bool T/F, default False). +* Branch schema warnings emitted to console, not just logs. +* [ACTION] Review and discuss policy changes [around commits](https://www.conventionalcommits.org/en/v1.0.0). + +Notes: +* Amazon Linux: every 2 years for release, would be more likely to snapshot + cloud-init upstream released. + * Some custom config modules for SELinux startup and other modules that may + make sense to drive some of those changes. +* AL2023: version repositories. + * When launching 2023 it's pinned to a specific version of the repository. +* AL1, AL2 and 2023 are different products: + * Realistically have only shipped 3 versions of cloud-init. diff -Nru cloud-init-23.3.3/integration-requirements.txt cloud-init-23.4.3/integration-requirements.txt --- cloud-init-23.3.3/integration-requirements.txt 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/integration-requirements.txt 2024-02-02 21:21:52.000000000 +0000 @@ -1,13 +1,13 @@ # PyPI requirements for cloud-init integration testing # https://cloudinit.readthedocs.io/en/latest/topics/integration_tests.html # -pycloudlib==1!5.3.0 +pycloudlib>=5.10.0,<1!6 -# Await pytest > 7.3.2. Breaking change in `testpaths` treatment forced +# Avoid breaking change in `testpaths` treatment forced # test/unittests/conftest.py to be loaded by our integration-tests tox env # resulting in an unmet dependency issue: # https://github.com/pytest-dev/pytest/issues/11104 -pytest<=7.3.1 +pytest!=7.3.2 packaging passlib diff -Nru cloud-init-23.3.3/packages/bddeb cloud-init-23.4.3/packages/bddeb --- cloud-init-23.3.3/packages/bddeb 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/packages/bddeb 2024-02-02 21:21:52.000000000 +0000 @@ -71,7 +71,7 @@ def run_helper(helper, args=None, strip=True): if args is None: args = [] - cmd = [util.abs_join(find_root(), "tools", helper)] + args + cmd = [os.path.abspath(os.path.join(find_root(), "tools", helper))] + args (stdout, _stderr) = subp.subp(cmd) if strip: stdout = stdout.strip() @@ -82,16 +82,18 @@ """Create a debian package directory with all rendered template files.""" print("Creating a debian/ folder in %r" % (root)) - deb_dir = util.abs_join(root, "debian") + deb_dir = os.path.abspath(os.path.join(root, "debian")) # Just copy debian/ dir and then update files - pdeb_d = util.abs_join(find_root(), "packages", "debian") + pdeb_d = os.path.abspath(os.path.join(find_root(), "packages", "debian")) subp.subp(["cp", "-a", pdeb_d, deb_dir]) # Fill in the change log template templater.render_to_file( - util.abs_join(find_root(), "packages", "debian", "changelog.in"), - util.abs_join(deb_dir, "changelog"), + os.path.abspath(os.path.join( + find_root(), "packages", "debian", "changelog.in") + ), + os.path.abspath(os.path.join(deb_dir, "changelog")), params=templ_data, ) @@ -128,8 +130,10 @@ else: build_deps += f",{debhelper_matches[-1]}" templater.render_to_file( - util.abs_join(find_root(), "packages", "debian", "control.in"), - util.abs_join(deb_dir, "control"), + os.path.abspath(os.path.join( + find_root(), "packages", "debian", "control.in") + ), + os.path.abspath(os.path.join(deb_dir, "control")), params={"build_depends": build_deps}, ) @@ -306,7 +310,7 @@ # since we will extract it then add in the debian # folder, then re-archive it for debian happiness tarball = "cloud-init_%s.orig.tar.gz" % ver_data["version_long"] - tarball_fp = util.abs_join(tdir, tarball) + tarball_fp = os.path.abspath(os.path.join(tdir, tarball)) path = None for pd in ("./", "../", "../dl/"): if os.path.exists(pd + tarball): @@ -329,7 +333,9 @@ cmd = ["tar", "-xvzf", tarball_fp, "-C", tdir] subp.subp(cmd, capture=capture) - xdir = util.abs_join(tdir, "cloud-init-%s" % ver_data["version_long"]) + xdir = os.path.abspath( + os.path.join(tdir, "cloud-init-%s" % ver_data["version_long"]) + ) templ_data.update(ver_data) if args.packaging_branch: diff -Nru cloud-init-23.3.3/packages/brpm cloud-init-23.4.3/packages/brpm --- cloud-init-23.3.3/packages/brpm 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/packages/brpm 2024-02-02 21:21:52.000000000 +0000 @@ -36,7 +36,7 @@ def run_helper(helper, args=None, strip=True): if args is None: args = [] - cmd = [util.abs_join(find_root(), 'tools', helper)] + args + cmd = [os.path.abspath(os.path.join(find_root(), 'tools', helper))] + args (stdout, _stderr) = subp.subp(cmd) if strip: stdout = stdout.strip() @@ -161,15 +161,15 @@ print("Archived the code in %r" % (real_archive_fn)) # Form the spec file to be used - tmpl_fn = util.abs_join(find_root(), 'packages', - args.distro, 'cloud-init.spec.in') + tmpl_fn = os.path.abspath(os.path.join(find_root(), 'packages', + args.distro, 'cloud-init.spec.in')) contents = generate_spec_contents(args, version_data, tmpl_fn, topdir, os.path.basename(archive_fn)) - spec_fn = util.abs_join(topdir, 'SPECS', 'cloud-init.spec') + spec_fn = os.path.abspath(os.path.join(topdir, 'SPECS', 'cloud-init.spec')) util.write_file(spec_fn, contents) print("Created spec file at %r" % (spec_fn)) for p in args.patches: - util.copy(p, util.abs_join(topdir, 'SOURCES', os.path.basename(p))) + util.copy(p, os.path.abspath(os.path.join(topdir, 'SOURCES', os.path.basename(p)))_ # Now build it! print("Running 'rpmbuild' in %r" % (topdir)) @@ -184,15 +184,19 @@ # Copy the items built to our local dir globs = [] globs.extend(glob.glob("%s/*.rpm" % - (util.abs_join(topdir, 'RPMS', 'noarch')))) + (os.path.abspath(os.path.join(topdir, 'RPMS', 'noarch'))))) globs.extend(glob.glob("%s/*.rpm" % - (util.abs_join(topdir, 'RPMS', 'x86_64')))) + (os.path.abspath(os.path.join(topdir, 'RPMS', + 'x86_64'))))) globs.extend(glob.glob("%s/*.rpm" % - (util.abs_join(topdir, 'RPMS')))) + (os.path.abspath(os.path.join(topdir, + 'RPMS'))))) globs.extend(glob.glob("%s/*.rpm" % - (util.abs_join(topdir, 'SRPMS')))) + (os.path.abspath(os.path.join(topdir, + 'SRPMS'))))) for rpm_fn in globs: - tgt_fn = util.abs_join(os.getcwd(), os.path.basename(rpm_fn)) + tgt_fn = os.path.abspath(os.path.join(os.getcwd(), + os.path.basename(rpm_fn))) shutil.move(rpm_fn, tgt_fn) print("Wrote out %s package %r" % (args.distro, tgt_fn)) finally: diff -Nru cloud-init-23.3.3/requirements.txt cloud-init-23.4.3/requirements.txt --- cloud-init-23.3.3/requirements.txt 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/requirements.txt 2024-02-02 21:21:52.000000000 +0000 @@ -28,7 +28,7 @@ jsonpatch # For validating cloud-config sections per schema definitions -jsonschema +jsonschema<=4.20.0 # Used by DataSourceVMware to inspect the host's network configuration during # the "setup()" function. diff -Nru cloud-init-23.3.3/setup.py cloud-init-23.4.3/setup.py --- cloud-init-23.3.3/setup.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/setup.py 2024-02-02 21:21:52.000000000 +0000 @@ -50,7 +50,7 @@ PREFIX = None -def render_tmpl(template, mode=None): +def render_tmpl(template, mode=None, is_yaml=False): """render template into a tmpdir under same dir as setup.py This is rendered to a temporary directory under the top level @@ -85,13 +85,11 @@ subprocess.run( [ sys.executable, - "./tools/render-cloudcfg", - ] - + cmd_prefix - + cmd_variant - + [ - template, - fpath, + "./tools/render-template", + *(["--is-yaml"] if is_yaml else []), + *cmd_prefix, + *cmd_variant, + *[template, fpath], ], check=True, ) @@ -123,20 +121,22 @@ PREFIX = sys.argv[idx + 1] INITSYS_FILES = { - "sysvinit": [f for f in glob("sysvinit/redhat/*") if is_f(f)], - "sysvinit_freebsd": [ + "sysvinit": lambda: [f for f in glob("sysvinit/redhat/*") if is_f(f)], + "sysvinit_freebsd": lambda: [ render_tmpl(f, mode=0o755) for f in glob("sysvinit/freebsd/*") if is_f(f) ], - "sysvinit_netbsd": [ + "sysvinit_netbsd": lambda: [ render_tmpl(f, mode=0o755) for f in glob("sysvinit/netbsd/*") if is_f(f) ], - "sysvinit_deb": [f for f in glob("sysvinit/debian/*") if is_f(f)], - "sysvinit_openrc": [f for f in glob("sysvinit/gentoo/*") if is_f(f)], - "systemd": [ + "sysvinit_deb": lambda: [f for f in glob("sysvinit/debian/*") if is_f(f)], + "sysvinit_openrc": lambda: [ + f for f in glob("sysvinit/gentoo/*") if is_f(f) + ], + "systemd": lambda: [ render_tmpl(f) for f in ( glob("systemd/*.tmpl") @@ -146,7 +146,7 @@ ) if (is_f(f) and not is_generator(f)) ], - "systemd.generators": [ + "systemd.generators": lambda: [ render_tmpl(f, mode=0o755) for f in glob("systemd/*") if is_f(f) and is_generator(f) @@ -250,11 +250,10 @@ k for k in INITSYS_ROOTS if k.partition(".")[0] == system ] for k in datakeys: - if not INITSYS_FILES[k]: + files = INITSYS_FILES[k]() + if not files: continue - self.distribution.data_files.append( - (INITSYS_ROOTS[k], INITSYS_FILES[k]) - ) + self.distribution.data_files.append((INITSYS_ROOTS[k], files)) # Force that command to reinitialize (with new file list) self.distribution.reinitialize_command("install_data", True) @@ -268,7 +267,7 @@ INITSYS_ROOTS[k] = "/" + INITSYS_ROOTS[k] data_files = [ - (ETC + "/cloud", [render_tmpl("config/cloud.cfg.tmpl")]), + (ETC + "/cloud", [render_tmpl("config/cloud.cfg.tmpl", is_yaml=True)]), (ETC + "/cloud/clean.d", glob("config/clean.d/*")), (ETC + "/cloud/cloud.cfg.d", glob("config/cloud.cfg.d/*")), (ETC + "/cloud/templates", glob("templates/*")), diff -Nru cloud-init-23.3.3/systemd/cloud-config.service.tmpl cloud-init-23.4.3/systemd/cloud-config.service.tmpl --- cloud-init-23.3.3/systemd/cloud-config.service.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-config.service.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -5,10 +5,9 @@ After=snapd.seeded.service Before=systemd-user-sessions.service Wants=network-online.target cloud-config.target -{% if variant == "rhel" %} ConditionPathExists=!/etc/cloud/cloud-init.disabled ConditionKernelCommandLine=!cloud-init=disabled -{% endif %} +ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled [Service] Type=oneshot diff -Nru cloud-init-23.3.3/systemd/cloud-final.service.tmpl cloud-init-23.4.3/systemd/cloud-final.service.tmpl --- cloud-init-23.3.3/systemd/cloud-final.service.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-final.service.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -7,10 +7,9 @@ Before=apt-daily.service {% endif %} Wants=network-online.target cloud-config.service -{% if variant == "rhel" %} ConditionPathExists=!/etc/cloud/cloud-init.disabled ConditionKernelCommandLine=!cloud-init=disabled -{% endif %} +ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled [Service] diff -Nru cloud-init-23.3.3/systemd/cloud-init-generator.tmpl cloud-init-23.4.3/systemd/cloud-init-generator.tmpl --- cloud-init-23.3.3/systemd/cloud-init-generator.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-init-generator.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -23,6 +23,8 @@ {% if variant in ["almalinux", "centos", "cloudlinux", "eurolinux", "fedora", "miraclelinux", "openeuler", "OpenCloudOS", "openmandriva", "rhel", "rocky", "TencentOS", "virtuozzo"] %} dsidentify="/usr/libexec/cloud-init/ds-identify" +{% elif variant == "benchmark" %} + dsidentify="/bin/true" {% else %} dsidentify="/usr/lib/cloud-init/ds-identify" {% endif %} @@ -40,105 +42,51 @@ echo "$@" >> "$LOG" } -etc_file() { - local pprefix="${1:-/etc/cloud/cloud-init.}" - _RET="unset" - [ -f "${pprefix}$ENABLE" ] && _RET="$ENABLE" && return 0 - [ -f "${pprefix}$DISABLE" ] && _RET="$DISABLE" && return 0 - return 0 -} - -read_proc_cmdline() { - # return /proc/cmdline for non-container, and /proc/1/cmdline for container - local ctname="systemd" - if [ -n "$CONTAINER" ] && ctname=$CONTAINER || - systemd-detect-virt --container --quiet; then - if { _RET=$(tr '\0' ' ' < /proc/1/cmdline); } 2>/dev/null; then - _RET_MSG="container[$ctname]: pid 1 cmdline" - return - fi - _RET="" - _RET_MSG="container[$ctname]: pid 1 cmdline not available" - return 0 - fi - - _RET_MSG="/proc/cmdline" - read _RET < /proc/cmdline -} - -kernel_cmdline() { - local cmdline="" tok="" - if [ -n "${KERNEL_CMDLINE+x}" ]; then - # use KERNEL_CMDLINE if present in environment even if empty - cmdline=${KERNEL_CMDLINE} - debug 1 "kernel command line from env KERNEL_CMDLINE: $cmdline" - else - read_proc_cmdline && cmdline="$_RET" && - debug 1 "kernel command line ($_RET_MSG): $cmdline" - fi - _RET="unset" - cmdline=" $cmdline " - tok=${cmdline##* cloud-init=} - [ "$tok" = "$cmdline" ] && _RET="unset" - tok=${tok%% *} - [ "$tok" = "$ENABLE" -o "$tok" = "$DISABLE" ] && _RET="$tok" - return 0 -} - -default() { - _RET="$ENABLE" -} - -check_for_datasource() { - local ds_rc="" - if [ ! -x "$dsidentify" ]; then - debug 1 "no ds-identify in $dsidentify" - return 0 - fi - $dsidentify - ds_rc=$? - debug 1 "ds-identify rc=$ds_rc" - if [ "$ds_rc" = "0" ]; then - return 0 - fi - return 1 -} - main() { local normal_d="$1" early_d="$2" late_d="$3" local target_name="multi-user.target" gen_d="$early_d" local link_path="$gen_d/${target_name}.wants/${CLOUD_TARGET_NAME}" - local ds="" + local ds="" ret="" debug 1 "$0 normal=$normal_d early=$early_d late=$late_d" debug 2 "$0 $*" - local search result="error" ret="" - for search in kernel_cmdline etc_file default; do - if $search; then - debug 1 "$search found $_RET" - [ "$_RET" = "$ENABLE" -o "$_RET" = "$DISABLE" ] && - result=$_RET && break - else - ret=$? - debug 0 "search $search returned $ret" - fi - done + # ds=found => enable + # ds=notfound => disable + # => disable + debug 1 "checking for datasource" - # enable AND ds=found == enable - # enable AND ds=notfound == disable - # disable || == disabled - if [ "$result" = "$ENABLE" ]; then - debug 1 "checking for datasource" - check_for_datasource - ds=$? + if [ ! -x "$dsidentify" ]; then + debug 1 "no ds-identify in $dsidentify" + ds=0 + fi + $dsidentify + ds=$? + debug 1 "ds-identify rc=$ds" + + if [ "$ds" = "1" -o "$ds" = "2" ]; then if [ "$ds" = "1" ]; then debug 1 "cloud-init is enabled but no datasource found, disabling" - result="$DISABLE" + else + debug 1 "cloud-init is disabled by kernel commandline or etc_file" fi - fi + if [ -f "$link_path" ]; then + if rm -f "$link_path"; then + debug 1 "disabled. removed existing $link_path" + else + ret=$? + debug 0 "[$ret] disable failed, remove $link_path" + fi + else + debug 1 "already disabled: no change needed [no $link_path]" + fi + if [ -e "$RUN_ENABLED_FILE" ]; then + debug 1 "removing $RUN_ENABLED_FILE and creating $RUN_DISABLED_FILE" + rm -f "$RUN_ENABLED_FILE" + fi + : > "$RUN_DISABLED_FILE" - if [ "$result" = "$ENABLE" ]; then + elif [ "$ds" = "0" ]; then if [ -e "$link_path" ]; then debug 1 "already enabled: no change needed" else @@ -157,22 +105,6 @@ rm -f $RUN_DISABLED_FILE fi : > "$RUN_ENABLED_FILE" - elif [ "$result" = "$DISABLE" ]; then - if [ -f "$link_path" ]; then - if rm -f "$link_path"; then - debug 1 "disabled. removed existing $link_path" - else - ret=$? - debug 0 "[$ret] disable failed, remove $link_path" - fi - else - debug 1 "already disabled: no change needed [no $link_path]" - fi - if [ -e "$RUN_ENABLED_FILE" ]; then - debug 1 "removing $RUN_ENABLED_FILE and creating $RUN_DISABLED_FILE" - rm -f "$RUN_ENABLED_FILE" - fi - : > "$RUN_DISABLED_FILE" else debug 0 "unexpected result '$result' 'ds=$ds'" ret=3 diff -Nru cloud-init-23.3.3/systemd/cloud-init-local.service.tmpl cloud-init-23.4.3/systemd/cloud-init-local.service.tmpl --- cloud-init-23.3.3/systemd/cloud-init-local.service.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-init-local.service.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -26,10 +26,9 @@ Conflicts=shutdown.target {% endif %} RequiresMountsFor=/var/lib/cloud -{% if variant == "rhel" %} ConditionPathExists=!/etc/cloud/cloud-init.disabled ConditionKernelCommandLine=!cloud-init=disabled -{% endif %} +ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled [Service] Type=oneshot diff -Nru cloud-init-23.3.3/systemd/cloud-init.service.tmpl cloud-init-23.4.3/systemd/cloud-init.service.tmpl --- cloud-init-23.3.3/systemd/cloud-init.service.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-init.service.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -38,10 +38,9 @@ Before=shutdown.target Conflicts=shutdown.target {% endif %} -{% if variant == "rhel" %} ConditionPathExists=!/etc/cloud/cloud-init.disabled ConditionKernelCommandLine=!cloud-init=disabled -{% endif %} +ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled [Service] Type=oneshot diff -Nru cloud-init-23.3.3/systemd/cloud-init.target cloud-init-23.4.3/systemd/cloud-init.target --- cloud-init-23.3.3/systemd/cloud-init.target 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/systemd/cloud-init.target 2024-02-02 21:21:52.000000000 +0000 @@ -10,3 +10,6 @@ [Unit] Description=Cloud-init target After=multi-user.target +ConditionPathExists=!/etc/cloud/cloud-init.disabled +ConditionKernelCommandLine=!cloud-init=disabled +ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled diff -Nru cloud-init-23.3.3/sysvinit/freebsd/dsidentify.tmpl cloud-init-23.4.3/sysvinit/freebsd/dsidentify.tmpl --- cloud-init-23.3.3/sysvinit/freebsd/dsidentify.tmpl 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/sysvinit/freebsd/dsidentify.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -23,18 +23,22 @@ dsidentify_start() { - echo "${command} starting" - if kenv -q kernel_options | grep -q 'cloud-init=disabled'; then - warn "cloud-init is disabled via kernel_options." - elif test -e {{ prefix }}/etc/cloud-init.disabled; then - warn "cloud-init is disabled via cloud-init.disabled file." - else - ${command} + if checkyesno cloudinit_dsidentify_enable; then + echo "${command} starting" + if kenv -q kernel_options | grep -q 'cloud-init=disabled'; then + warn "cloud-init is disabled via kernel_options." + elif test -e {{ prefix }}/etc/cloud-init.disabled; then + warn "cloud-init is disabled via cloud-init.disabled file." + else + ${command} + fi fi } load_rc_config 'cloudinit' : ${cloudinit_enable="NO"} +: ${cloudinit_dsidentify_enable="YES"} +: ${dsidentify_env="PATH_DI_CONFIG={{ prefix }}/etc/cloud/ds-identify.cfg PATH_ETC_CLOUD={{ prefix }}/etc/cloud"} run_rc_command "$1" diff -Nru cloud-init-23.3.3/templates/sources.list.debian.deb822.tmpl cloud-init-23.4.3/templates/sources.list.debian.deb822.tmpl --- cloud-init-23.3.3/templates/sources.list.debian.deb822.tmpl 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/templates/sources.list.debian.deb822.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,34 @@ +## template:jinja +## Note, this file is written by cloud-init on first boot of an instance +## modifications made here will not survive a re-bundle. +## if you wish to make changes you can: +## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg +## or do the same in user-data +## b.) add supplemental sources in /etc/apt/sources.list.d +## c.) make changes to template file +## /etc/cloud/templates/sources.list.debian.deb822.tmpl + +# For how to upgrade to newer versions of the distribution, see: +# http://www.debian.org/releases/stable/i386/release-notes/ch-upgrading.html + +## See the sources.list(5) manual page for further settings. + +## Comment any of the following Suites to avoid getting updates from the +## specific Suite. +## +## N.B. software from {{codename}}-backports repository may not have been +## tested as extensively as that contained in the main release, although it +## includes newer versions of some applications which may provide useful +## features. +Types: deb deb-src +URIs: {{mirror}} +Suites: {{codename}} {{codename}}-updates {{codename}}-backports +Components: main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg + +## Major bug fix updates produced after the final release of the distribution. +Types: deb deb-src +URIs: {{security}} +Suites: {{codename}}{% if codename in ('buster', 'stretch') %}/updates{% else %}-security{% endif %} +Components: main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg diff -Nru cloud-init-23.3.3/templates/sources.list.ubuntu.deb822.tmpl cloud-init-23.4.3/templates/sources.list.ubuntu.deb822.tmpl --- cloud-init-23.3.3/templates/sources.list.ubuntu.deb822.tmpl 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/templates/sources.list.ubuntu.deb822.tmpl 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,61 @@ +## template:jinja +## Note, this file is written by cloud-init on first boot of an instance +## modifications made here will not survive a re-bundle. +## if you wish to make changes you can: +## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg +## or do the same in user-data +## b.) add supplemental sources in /etc/apt/sources.list.d +## c.) make changes to template file +## /etc/cloud/templates/sources.list.ubuntu.deb822.tmpl +## +## See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to +## newer versions of the distribution. +## +## The following settings can be tweaked to configure which packages to use +## from Ubuntu. +## Mirror your choices (except for URIs and Suites) in the security section +## below to ensure timely security updates. +## +## Types: Append deb-src to enable the fetching of source package. +## URIs: A URL to the repository (you may add multiple URLs) +## Suites: The following additional suites can be configured +## -updates - Major bug fix updates produced after the final release +## of the distribution. +## -backports - software from this repository may not have been tested +## as extensively as that contained in the main release, +## although it includes newer versions of some +## applications which may provide useful features. +## Also, please note that software in backports WILL NOT +## receive any review or updates from the Ubuntu security +## team. +## Components: Aside from main, the following components can be added to the +## list: +## restricted - Software that may not be under a free license, or protected +## by patents. +## universe - Community maintained packages. Software from this repository +## is ENTIRELY UNSUPPORTED by the Ubuntu team. Also, please +## note that software in universe WILL NOT receive any +## review or updates from the Ubuntu security team. +## multiverse - Community maintained of restricted. Software from this +## repository is ENTIRELY UNSUPPORTED by the Ubuntu team, and +## may not be under a free licence. Please satisfy yourself as +## to your rights to use the software. +## Also, please note that software in multiverse WILL NOT +## receive any review or updates from the Ubuntu security team. +## +## See the sources.list(5) manual page for further settings. +# Types: deb deb-src +Types: deb +URIs: {{mirror}} +Suites: {{codename}} {{codename}}-updates {{codename}}-backports +Components: main restricted universe multiverse +Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg + +## Ubuntu security updates. Aside from URIs and Suites, +## this should mirror your choices in the previous section. +# Types: deb deb-src +Types: deb +URIs: {{security}} +Suites: {{codename}}-security +Components: main restricted universe multiverse +Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg diff -Nru cloud-init-23.3.3/test-requirements.txt cloud-init-23.4.3/test-requirements.txt --- cloud-init-23.3.3/test-requirements.txt 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/test-requirements.txt 2024-02-02 21:21:52.000000000 +0000 @@ -1,14 +1,14 @@ # Needed generally in tests -# Await pytest > 7.3.2. Breaking change in `testpaths` treatment forced +# Avoid breaking change in `testpaths` treatment forced # test/unittests/conftest.py to be loaded by our integration-tests tox env # resulting in an unmet dependency issue: # https://github.com/pytest-dev/pytest/issues/11104 -pytest<=7.3.1 +pytest!=7.3.2 pytest-cov pytest-mock setuptools -jsonschema +jsonschema<=4.20.0 responses passlib diff -Nru cloud-init-23.3.3/tests/integration_tests/bugs/test_lp1898997.py cloud-init-23.4.3/tests/integration_tests/bugs/test_lp1898997.py --- cloud-init-23.3.3/tests/integration_tests/bugs/test_lp1898997.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/bugs/test_lp1898997.py 2024-02-02 21:21:52.000000000 +0000 @@ -54,6 +54,9 @@ CURRENT_RELEASE < FOCAL, reason="Tested on Focal and above" ) @pytest.mark.lxd_use_exec +@pytest.mark.skip( + reason="Network online race. GH: #4350, GH: #4451, LP: #2036968" +) class TestInterfaceListingWithOpenvSwitch: def test_ovs_member_interfaces_not_excluded(self, client): # We need to install openvswitch for our provided network configuration diff -Nru cloud-init-23.3.3/tests/integration_tests/clouds.py cloud-init-23.4.3/tests/integration_tests/clouds.py --- cloud-init-23.3.3/tests/integration_tests/clouds.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/clouds.py 2024-02-02 21:21:52.000000000 +0000 @@ -19,6 +19,7 @@ LXDContainer, LXDVirtualMachine, Openstack, + Qemu, ) from pycloudlib.cloud import BaseCloud, ImageType from pycloudlib.ec2.instance import EC2Instance @@ -169,7 +170,13 @@ return IntegrationInstance(self, cloud_instance, settings) def destroy(self): - self.cloud_instance.clean() + if self.settings.KEEP_IMAGE or self.settings.KEEP_INSTANCE: + log.info( + "NOT cleaning cloud instance because KEEP_IMAGE or " + "KEEP_INSTANCE is True" + ) + else: + self.cloud_instance.clean() def snapshot(self, instance): return self.cloud_instance.snapshot(instance, clean=True) @@ -395,3 +402,11 @@ return IBM( tag="integration-test-ibm", ) + + +class QemuCloud(IntegrationCloud): + datasource = "qemu" + cloud_instance = Qemu + + def _get_cloud_instance(self): + return Qemu(tag="qemu-integration-test") diff -Nru cloud-init-23.3.3/tests/integration_tests/cmd/test_clean.py cloud-init-23.4.3/tests/integration_tests/cmd/test_clean.py --- cloud-init-23.3.3/tests/integration_tests/cmd/test_clean.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/cmd/test_clean.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,85 @@ +# This file is part of cloud-init. See LICENSE file for license information. +import re + +import pytest + +from tests.integration_tests.instances import IntegrationInstance + +USER_DATA = """\ +#cloud-config +write_files: +- path: /etc/network/interfaces.d/50-cloud-init.cfg + content: cloud-init test was here + permissions: '0644' + owner: root:root +- path: /etc/cloud/clean.d/runme.sh + encoding: b64 + content: IyEvYmluL3NoCmVjaG8gIi9ldGMvY2xvdWQvY2xlYW4uZC9ydW5tZS5zaCBSQU4iCg== + permissions: '0755' + owner: root:root +- path: /etc/systemd/network/10-cloud-init-eth0.network + content: cloud-init test was here + permissions: '0644' + owner: root:root +- path: /etc/cloud/clean.d/dontrunme.sh + content: '#!/bin/sh\necho DID NOT RUN BECAUSE NO EXEC PERMS' + permissions: '0644' + owner: root:root +""" + + +@pytest.mark.user_data(USER_DATA) +class TestCleanCommand: + def test_clean_by_param(self, class_client: IntegrationInstance): + """Clean with various params alters expected files without error""" + assert class_client.execute("cloud-init status --wait").ok + result = class_client.execute("cloud-init clean") + assert ( + result.ok + ), "non-zero exit on cloud-init clean runparts of /etc/cloud/clean.d" + # Log files are not removed without --logs + log_paths = ( + "/var/log/cloud-init.log", + "/var/log/cloud-init-output.log", + ) + net_cfg_paths = ( + "/etc/network/interfaces.d/50-cloud-init.cfg", + "/etc/netplan/50-cloud-init.yaml", + "/etc/systemd/network/10-cloud-init-eth0.network", + ) + for path in log_paths + net_cfg_paths: + assert class_client.execute( + f"test -f {path}" + ).ok, f"Missing expected file {path}" + # /etc/cloud/clean.d runparts scripts are run if executable + assert result.stdout == "/etc/cloud/clean.d/runme.sh RAN" + + # Log files removed with --logs + assert class_client.execute("cloud-init clean --logs").ok + for path in log_paths: + assert class_client.execute( + f"test -f {path}" + ).failed, f"Unexpected file found {path}" + for path in net_cfg_paths: + assert class_client.execute( + f"test -f {path}" + ).ok, f"Missing expected file {path}" + + prev_machine_id = class_client.read_from_file("/etc/machine-id") + assert re.match( + r"^[a-f0-9]{32}$", prev_machine_id + ), f"Unexpected machine-id format {prev_machine_id}" + + # --machine-id sets /etc/machine-id + assert class_client.execute("cloud-init clean --machine-id").ok + machine_id = class_client.read_from_file("/etc/machine-id") + assert machine_id != prev_machine_id + assert "uninitialized" == machine_id + + # --configs remove network scope + assert class_client.execute("cloud-init clean --configs network").ok + + for path in log_paths + net_cfg_paths: + assert class_client.execute( + f"test -f {path}" + ).failed, f"Unexpected file found {path}" diff -Nru cloud-init-23.3.3/tests/integration_tests/cmd/test_schema.py cloud-init-23.4.3/tests/integration_tests/cmd/test_schema.py --- cloud-init-23.3.3/tests/integration_tests/cmd/test_schema.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/cmd/test_schema.py 2024-02-02 21:21:52.000000000 +0000 @@ -70,6 +70,6 @@ # D3: Default: ``false``. Deprecated in version 22.2. Use ``package_reboot_if_required`` instead. - Valid cloud-config: /root/user-data""" # noqa: E501 + Valid schema /root/user-data""" # noqa: E501 ) assert expected_output in annotated_result.stdout diff -Nru cloud-init-23.3.3/tests/integration_tests/cmd/test_status.py cloud-init-23.4.3/tests/integration_tests/cmd/test_status.py --- cloud-init-23.3.3/tests/integration_tests/cmd/test_status.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/cmd/test_status.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,4 +1,6 @@ """Tests for `cloud-init status`""" +import json + import pytest from tests.integration_tests.clouds import IntegrationCloud @@ -47,3 +49,36 @@ status_out = wait_for_cloud_init(client).stdout.strip() assert "status: disabled" in status_out assert client.execute("cloud-init status --wait").ok + + +USER_DATA = """\ +#cloud-config +ca-certs: + remove_defaults: false + invalid_key: true +""" + + +@pytest.mark.user_data(USER_DATA) +def test_status_json_errors(client): + """Ensure that deprecated logs end up in the recoverable errors and that + machine readable status contains recoverable errors + """ + status_json = client.execute("cat /run/cloud-init/status.json").stdout + assert json.loads(status_json)["v1"]["init"]["recoverable_errors"].get( + "DEPRECATED" + ) + + status_json = client.execute("cloud-init status --format json").stdout + assert "Deprecated cloud-config provided:\nca-certs:" in json.loads( + status_json + )["init"]["recoverable_errors"].get("DEPRECATED").pop(0) + assert "Deprecated cloud-config provided:\nca-certs:" in json.loads( + status_json + )["recoverable_errors"].get("DEPRECATED").pop(0) + assert "Invalid cloud-config provided" in json.loads(status_json)["init"][ + "recoverable_errors" + ].get("WARNING").pop(0) + assert "Invalid cloud-config provided" in json.loads(status_json)[ + "recoverable_errors" + ].get("WARNING").pop(0) diff -Nru cloud-init-23.3.3/tests/integration_tests/conftest.py cloud-init-23.4.3/tests/integration_tests/conftest.py --- cloud-init-23.3.3/tests/integration_tests/conftest.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/conftest.py 2024-02-02 21:21:52.000000000 +0000 @@ -23,6 +23,7 @@ LxdVmCloud, OciCloud, OpenstackCloud, + QemuCloud, _LxdIntegrationCloud, ) from tests.integration_tests.instances import ( @@ -42,6 +43,7 @@ "ibm": IbmCloud, "lxd_container": LxdContainerCloud, "lxd_vm": LxdVmCloud, + "qemu": QemuCloud, "openstack": OpenstackCloud, } os_list = ["ubuntu"] diff -Nru cloud-init-23.3.3/tests/integration_tests/datasources/test_none.py cloud-init-23.4.3/tests/integration_tests/datasources/test_none.py --- cloud-init-23.3.3/tests/integration_tests/datasources/test_none.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/datasources/test_none.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,67 @@ +"""DataSourceNone integration tests on LXD.""" +import json + +from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.util import verify_clean_log + +DS_NONE_BASE_CFG = """\ +datasource_list: [None] +datasource: + None: + metadata: + instance-id: my-iid-uuid + userdata_raw: | + #cloud-config + runcmd: + - touch /var/tmp/success-with-datasource-none +""" + + +def test_datasource_none_discovery(client: IntegrationInstance): + """Integration test for #4635. + + Test that DataSourceNone detection (used by live installers) doesn't + generate errors or warnings. + """ + log = client.read_from_file("/var/log/cloud-init.log") + verify_clean_log(log) + # Limit datasource detection to DataSourceNone. + client.write_to_file( + "/etc/cloud/cloud.cfg.d/99-force-dsnone.cfg", DS_NONE_BASE_CFG + ) + if client.settings.PLATFORM in ["lxd_container"]: + # DataSourceNone provides no network_config. + # To avoid changing network config from platform desired net cfg + # to fallback config, copy out the rendered network config + # to /etc/cloud/cloud.cfg.d/99-orig-net.cfg so it is + # setup by the DataSourceNone case as well. + # Otherwise (LXD specifically) we'll have network torn down due + # to virtual NICs present which results in not network being + # brought up when we emit fallback config which attempts to + # match on PermanentMACAddress. LP:#2022947 + client.execute( + "cp /etc/netplan/50-cloud-init.yaml" + " /etc/cloud/cloud.cfg.d/99-orig-net.cfg" + ) + client.execute("cloud-init clean --logs") + client.restart() + status = json.loads(client.execute("cloud-init status --format=json")) + assert [] == status["errors"] + expected_warnings = [ + "Used fallback datasource", + "Falling back to a hard restart of systemd-networkd.service", + ] + unexpected_warnings = [] + for current_warning in status["recoverable_errors"].get("WARNING", []): + if [w for w in expected_warnings if w in current_warning]: + # Found a matching expected_warning substring in current_warning + continue + unexpected_warnings.append(current_warning) + + if unexpected_warnings: + raise AssertionError( + f"Unexpected recoverable errors: {list(unexpected_warnings)}" + ) + log = client.read_from_file("/var/log/cloud-init.log") + verify_clean_log(log) + assert client.execute("test -f /var/tmp/success-with-datasource-none").ok diff -Nru cloud-init-23.3.3/tests/integration_tests/instances.py cloud-init-23.4.3/tests/integration_tests/instances.py --- cloud-init-23.3.3/tests/integration_tests/instances.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/instances.py 2024-02-02 21:21:52.000000000 +0000 @@ -201,7 +201,12 @@ if self._ip: return self._ip try: - self._ip = self.instance.ip + # in some cases that ssh is not used, an address is not assigned + if ( + hasattr(self.instance, "execute_via_ssh") + and self.instance.execute_via_ssh + ): + self._ip = self.instance.ip except NotImplementedError: self._ip = "Unknown" return self._ip diff -Nru cloud-init-23.3.3/tests/integration_tests/integration_settings.py cloud-init-23.4.3/tests/integration_tests/integration_settings.py --- cloud-init-23.3.3/tests/integration_tests/integration_settings.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/integration_settings.py 2024-02-02 21:21:52.000000000 +0000 @@ -16,14 +16,15 @@ RUN_UNSTABLE = False # One of: -# lxd_container -# lxd_vm # azure # ec2 # gce # ibm +# lxd_container +# lxd_vm # oci # openstack +# qemu PLATFORM = "lxd_container" # The cloud-specific instance type to run. E.g., a1.medium on AWS diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_ansible.py cloud-init-23.4.3/tests/integration_tests/modules/test_ansible.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_ansible.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_ansible.py 2024-02-02 21:21:52.000000000 +0000 @@ -312,6 +312,7 @@ CURRENT_RELEASE < FOCAL, reason="Pip install is not supported for Ansible on release", ) +@pytest.mark.skip(reason="Need proxy support first. GH: #4527") def test_ansible_controller(client): log = client.read_from_file("/var/log/cloud-init.log") verify_clean_log(log) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_apt.py cloud-init-23.4.3/tests/integration_tests/modules/test_apt.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_apt.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_apt.py 1970-01-01 00:00:00.000000000 +0000 @@ -1,414 +0,0 @@ -"""Series of integration tests covering apt functionality.""" -import re - -import pytest - -from cloudinit import gpg -from cloudinit.config import cc_apt_configure -from tests.integration_tests.instances import IntegrationInstance -from tests.integration_tests.integration_settings import PLATFORM -from tests.integration_tests.releases import CURRENT_RELEASE, IS_UBUNTU - -USER_DATA = """\ -#cloud-config -apt: - conf: | - APT { - Get { - Assume-Yes "true"; - Fix-Broken "true"; - } - } - primary: - - arches: [default] - uri: http://badarchive.ubuntu.com/ubuntu - security: - - arches: [default] - uri: http://badsecurity.ubuntu.com/ubuntu - sources_list: | - deb $MIRROR $RELEASE main restricted - deb-src $MIRROR $RELEASE main restricted - deb $PRIMARY $RELEASE universe restricted - deb-src $PRIMARY $RELEASE universe restricted - deb $SECURITY $RELEASE-security multiverse - deb-src $SECURITY $RELEASE-security multiverse - sources: - test_keyserver: - keyid: 110E21D8B0E2A1F0243AF6820856F197B892ACEA - keyserver: keyserver.ubuntu.com - source: "deb http://ppa.launchpad.net/canonical-kernel-team/ppa/ubuntu $RELEASE main" - test_ppa: - keyid: 441614D8 - keyserver: keyserver.ubuntu.com - source: "ppa:simplestreams-dev/trunk" - test_signed_by: - keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 - keyserver: keyserver.ubuntu.com - source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" - test_bad_key: - key: "" - source: "deb $MIRROR $RELEASE main" - test_key: - source: "deb http://ppa.launchpad.net/cloud-init-dev/test-archive/ubuntu $RELEASE main" - key: | - -----BEGIN PGP PUBLIC KEY BLOCK----- - Version: SKS 1.1.6 - Comment: Hostname: keyserver.ubuntu.com - - mQINBFbZRUIBEAC+A0PIKYBP9kLC4hQtRrffRS11uLo8/BdtmOdrlW0hpPHzCfKnjR3tvSEI - lqPHG1QrrjAXKZDnZMRz+h/px7lUztvytGzHPSJd5ARUzAyjyRezUhoJ3VSCxrPqx62avuWf - RfoJaIeHfDehL5/dTVkyiWxfVZ369ZX6JN2AgLsQTeybTQ75+2z0xPrrhnGmgh6g0qTYcAaq - M5ONOGiqeSBX/Smjh6ALy5XkhUiFGLsI7Yluf6XSICY/x7gd6RAfgSIQrUTNMoS1sqhT4aot - +xvOfQy8ySkfAK4NddXql6E/+ZqTmBY/Lr0YklFBy8jGT+UysfiIznPMIwbmgq5Li7BtDDtX - b8Uyi4edPpjtextezfXYn4NVIpPL5dPZS/FXh4HpzyH0pYCfrH4QDGA7i52AGmhpiOFjJMo6 - N33sdjZHOH/2Vyp+QZaQnsdUAi1N4M6c33tQbpIScn1SY+El8z5JDA4PBzkw8HpLCi1gGoa6 - V4kfbWqXXbGAJFkLkP/vc4+pY9axOlmCkJg7xCPwhI75y1cONgovhz+BEXOzolh5KZuGbGbj - xe0wva5DLBeIg7EQFf+99pOS7Syby3Xpm6ZbswEFV0cllK4jf/QMjtfInxobuMoI0GV0bE5l - WlRtPCK5FnbHwxi0wPNzB/5fwzJ77r6HgPrR0OkT0lWmbUyoOQARAQABtC1MYXVuY2hwYWQg - UFBBIGZvciBjbG91ZCBpbml0IGRldmVsb3BtZW50IHRlYW2JAjgEEwECACIFAlbZRUICGwMG - CwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAg9Bvvk0wTfHfcP/REK5N2s1JYc69qEa9ZN - o6oi+A7l6AYw+ZY88O5TJe7F9otv5VXCIKSUT0Vsepjgf0mtXAgf/sb2lsJn/jp7tzgov3YH - vSrkTkRydz8xcA87gwQKePuvTLxQpftF4flrBxgSueIn5O/tPrBOxLz7EVYBc78SKg9aj9L2 - yUp+YuNevlwfZCTYeBb9r3FHaab2HcgkwqYch66+nKYfwiLuQ9NzXXm0Wn0JcEQ6pWvJscbj - C9BdawWovfvMK5/YLfI6Btm7F4mIpQBdhSOUp/YXKmdvHpmwxMCN2QhqYK49SM7qE9aUDbJL - arppSEBtlCLWhRBZYLTUna+BkuQ1bHz4St++XTR49Qd7vDERALpApDjB2dxPfMiBzCMwQQyq - uy13exU8o2ETLg+dZSLfDTzrBNsBFmXlw8WW17nTISYdKeGKL+QdlUjpzdwUMMzHhAO8SmMH - zjeSlDSRMXBJFAFSbCl7EwmMKa3yVX0zInT91fNllZ3iatAmtVdqVH/BFQfTIMH2ET7A8WzJ - ZzVSuMRhqoKdr5AMcHuJGPUoVkVJHQA+NNvEiXSysF3faL7jmKapmUwrhpYYX2H8pf+VMu2e - cLflKTI28dl+ZQ4Pl/aVsxrti/pzhdYy05Sn5ddtySyIkvo8L1cU5MWpbvSlFPkTstBUDLBf - pb0uBy+g0oxJQg15 - =uy53 - -----END PGP PUBLIC KEY BLOCK----- - test_write: - keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 - keyserver: keyserver.ubuntu.com - source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" - append: false - test_write.list: - keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 - keyserver: keyserver.ubuntu.com - source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/devel/ubuntu $RELEASE main" - append: false - test_append: - keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 - keyserver: keyserver.ubuntu.com - source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" - test_append.list: - keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 - keyserver: keyserver.ubuntu.com - source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/devel/ubuntu $RELEASE main" -apt_pipelining: os -""" # noqa: E501 - -EXPECTED_REGEXES = [ - r"deb http://badarchive.ubuntu.com/ubuntu [a-z]+ main restricted", - r"deb-src http://badarchive.ubuntu.com/ubuntu [a-z]+ main restricted", - r"deb http://badarchive.ubuntu.com/ubuntu [a-z]+ universe restricted", - r"deb-src http://badarchive.ubuntu.com/ubuntu [a-z]+ universe restricted", - r"deb http://badsecurity.ubuntu.com/ubuntu [a-z]+-security multiverse", - r"deb-src http://badsecurity.ubuntu.com/ubuntu [a-z]+-security multiverse", -] - -TEST_KEYSERVER_KEY = "110E 21D8 B0E2 A1F0 243A F682 0856 F197 B892 ACEA" -TEST_PPA_KEY = "3552 C902 B4DD F7BD 3842 1821 015D 28D7 4416 14D8" -TEST_KEY = "1FF0 D853 5EF7 E719 E5C8 1B9C 083D 06FB E4D3 04DF" -TEST_SIGNED_BY_KEY = "A2EB 2DEC 0BD7 519B 7B38 BE38 376A 290E C806 8B11" - - -@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") -@pytest.mark.user_data(USER_DATA) -class TestApt: - def get_keys(self, class_client: IntegrationInstance): - """Return all keys in /etc/apt/trusted.gpg.d/ and /etc/apt/trusted.gpg - in human readable format. Mimics the output of apt-key finger - """ - list_cmd = " ".join(gpg.GPG_LIST) + " " - keys = class_client.execute(list_cmd + cc_apt_configure.APT_LOCAL_KEYS) - print(keys) - files = class_client.execute( - "ls " + cc_apt_configure.APT_TRUSTED_GPG_DIR - ) - for file in files.split(): - path = cc_apt_configure.APT_TRUSTED_GPG_DIR + file - keys += class_client.execute(list_cmd + path) or "" - return keys - - def test_sources_list(self, class_client: IntegrationInstance): - """Integration test for the apt module's `sources_list` functionality. - - This test specifies a ``sources_list`` and then checks that (a) the - expected number of sources.list entries is present, and (b) that each - expected line appears in the file. - - (This is ported from - `tests/cloud_tests/testcases/modules/apt_configure_sources_list.yaml`.) - """ - sources_list = class_client.read_from_file("/etc/apt/sources.list") - assert 6 == len(sources_list.rstrip().split("\n")) - - for expected_re in EXPECTED_REGEXES: - assert re.search(expected_re, sources_list) is not None - - def test_apt_conf(self, class_client: IntegrationInstance): - """Test the apt conf functionality. - - Ported from tests/cloud_tests/testcases/modules/apt_configure_conf.py - """ - apt_config = class_client.read_from_file( - "/etc/apt/apt.conf.d/94cloud-init-config" - ) - assert 'Assume-Yes "true";' in apt_config - assert 'Fix-Broken "true";' in apt_config - - def test_ppa_source(self, class_client: IntegrationInstance): - """Test the apt ppa functionality. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_sources_ppa.py - """ - if CURRENT_RELEASE.series < "mantic": - ppa_path_contents = class_client.read_from_file( - "/etc/apt/sources.list.d/" - "simplestreams-dev-ubuntu-trunk-{}.list".format( - CURRENT_RELEASE.series - ) - ) - else: - # deb822 support present on mantic and later - ppa_path_contents = class_client.read_from_file( - "/etc/apt/sources.list.d/" - "simplestreams-dev-ubuntu-trunk-{}.sources".format( - CURRENT_RELEASE.series - ) - ) - - assert ( - "://ppa.launchpad.net/simplestreams-dev/trunk/ubuntu" - in ppa_path_contents - or "://ppa.launchpadcontent.net/simplestreams-dev/trunk/ubuntu" - in ppa_path_contents - ) - - assert TEST_PPA_KEY in self.get_keys(class_client) - - def test_signed_by(self, class_client: IntegrationInstance): - """Test the apt signed-by functionality.""" - source = ( - "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_signed_by.gpg] " - "http://ppa.launchpad.net/juju/stable/ubuntu" - " {} main".format(CURRENT_RELEASE.series) - ) - path_contents = class_client.read_from_file( - "/etc/apt/sources.list.d/test_signed_by.list" - ) - assert path_contents == source - - key = class_client.execute( - "gpg --no-default-keyring --with-fingerprint --list-keys " - "--keyring /etc/apt/cloud-init.gpg.d/test_signed_by.gpg" - ) - - assert TEST_SIGNED_BY_KEY in key - - def test_bad_key(self, class_client: IntegrationInstance): - """Test the apt signed-by functionality.""" - with pytest.raises(OSError): - class_client.read_from_file( - "/etc/apt/trusted.list.d/test_bad_key.gpg" - ) - - def test_key(self, class_client: IntegrationInstance): - """Test the apt key functionality. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_sources_key.py - """ - test_archive_contents = class_client.read_from_file( - "/etc/apt/sources.list.d/test_key.list" - ) - - assert ( - "http://ppa.launchpad.net/cloud-init-dev/test-archive/ubuntu" - in test_archive_contents - ) - assert TEST_KEY in self.get_keys(class_client) - - def test_keyserver(self, class_client: IntegrationInstance): - """Test the apt keyserver functionality. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_sources_keyserver.py - """ - test_keyserver_contents = class_client.read_from_file( - "/etc/apt/sources.list.d/test_keyserver.list" - ) - - assert ( - "http://ppa.launchpad.net/canonical-kernel-team/ppa/ubuntu" - in test_keyserver_contents - ) - - assert TEST_KEYSERVER_KEY in self.get_keys(class_client) - - def test_os_pipelining(self, class_client: IntegrationInstance): - """Test 'os' settings does not write apt config file. - - Ported from tests/cloud_tests/testcases/modules/apt_pipelining_os.py - """ - conf_exists = class_client.execute( - "test -f /etc/apt/apt.conf.d/90cloud-init-pipelining" - ).ok - assert conf_exists is False - - def test_sources_write(self, class_client: IntegrationInstance): - """Test overwrite or append to sources file""" - test_write_content = class_client.read_from_file( - "/etc/apt/sources.list.d/test_write.list" - ) - expected_contents = ( - "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_write.gpg] " - "http://ppa.launchpad.net/juju/devel/ubuntu " - f"{CURRENT_RELEASE.series} main" - ) - assert expected_contents.strip() == test_write_content.strip() - - def test_sources_append(self, class_client: IntegrationInstance): - series = CURRENT_RELEASE.series - test_append_content = class_client.read_from_file( - "/etc/apt/sources.list.d/test_append.list" - ) - - expected_contents = ( - "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_append.gpg] " - f"http://ppa.launchpad.net/juju/stable/ubuntu {series} main\n" - "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_append.gpg] " - f"http://ppa.launchpad.net/juju/devel/ubuntu {series} main" - ) - assert expected_contents.strip() == test_append_content.strip() - - -_DEFAULT_DATA = """\ -#cloud-config -apt: - primary: - - arches: - - default - {uri} - security: - - arches: - - default -""" -DEFAULT_DATA = _DEFAULT_DATA.format(uri="") - - -@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") -@pytest.mark.user_data(DEFAULT_DATA) -class TestDefaults: - @pytest.mark.skipif( - PLATFORM != "openstack", reason="Test is Openstack specific" - ) - def test_primary_on_openstack(self, class_client: IntegrationInstance): - """Test apt default primary source on openstack. - - When no uri is provided. - """ - zone = class_client.execute("cloud-init query v1.availability_zone") - sources_list = class_client.read_from_file("/etc/apt/sources.list") - assert "{}.clouds.archive.ubuntu.com".format(zone) in sources_list - - def test_security(self, class_client: IntegrationInstance): - """Test apt default security sources. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_security.py - """ - sources_list = class_client.read_from_file("/etc/apt/sources.list") - - # 3 lines from main, universe, and multiverse - series = CURRENT_RELEASE.series - sec_url = f"deb http://security.ubuntu.com/ubuntu {series}-security" - if class_client.settings.PLATFORM == "azure": - sec_url = ( - f"deb http://azure.archive.ubuntu.com/ubuntu/" - f" {series}-security" - ) - sec_src_url = sec_url.replace("deb ", "# deb-src ") - assert 3 == sources_list.count(sec_url) - assert 3 == sources_list.count(sec_src_url) - - -DEFAULT_DATA_WITH_URI = _DEFAULT_DATA.format( - uri='uri: "http://something.random.invalid/ubuntu"' -) - - -@pytest.mark.user_data(DEFAULT_DATA_WITH_URI) -def test_default_primary_with_uri(client: IntegrationInstance): - """Test apt default primary sources. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_primary.py - """ - sources_list = client.read_from_file("/etc/apt/sources.list") - assert "archive.ubuntu.com" not in sources_list - - assert "something.random.invalid" in sources_list - - -DISABLED_DATA = """\ -#cloud-config -apt: - disable_suites: - - $RELEASE - - $RELEASE-updates - - $RELEASE-backports - - $RELEASE-security -apt_pipelining: false -""" - - -@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") -@pytest.mark.user_data(DISABLED_DATA) -class TestDisabled: - def test_disable_suites(self, class_client: IntegrationInstance): - """Test disabling of apt suites. - - Ported from - tests/cloud_tests/testcases/modules/apt_configure_disable_suites.py - """ - sources_list = class_client.execute( - "cat /etc/apt/sources.list | grep -v '^#'" - ).strip() - assert "" == sources_list - - def test_disable_apt_pipelining(self, class_client: IntegrationInstance): - """Test disabling of apt pipelining. - - Ported from - tests/cloud_tests/testcases/modules/apt_pipelining_disable.py - """ - conf = class_client.read_from_file( - "/etc/apt/apt.conf.d/90cloud-init-pipelining" - ) - assert 'Acquire::http::Pipeline-Depth "0";' in conf - - -APT_PROXY_DATA = """\ -#cloud-config -apt: - proxy: "http://proxy.internal:3128" - http_proxy: "http://squid.internal:3128" - ftp_proxy: "ftp://squid.internal:3128" - https_proxy: "https://squid.internal:3128" -""" - - -@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") -@pytest.mark.user_data(APT_PROXY_DATA) -def test_apt_proxy(client: IntegrationInstance): - """Test the apt proxy data gets written correctly.""" - out = client.read_from_file("/etc/apt/apt.conf.d/90cloud-init-aptproxy") - assert 'Acquire::http::Proxy "http://proxy.internal:3128";' in out - assert 'Acquire::http::Proxy "http://squid.internal:3128";' in out - assert 'Acquire::ftp::Proxy "ftp://squid.internal:3128";' in out - assert 'Acquire::https::Proxy "https://squid.internal:3128";' in out diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_apt_functionality.py cloud-init-23.4.3/tests/integration_tests/modules/test_apt_functionality.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_apt_functionality.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_apt_functionality.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,447 @@ +# This file is part of cloud-init. See LICENSE file for license information. +"""Series of integration tests covering apt functionality.""" +import re +from textwrap import dedent + +import pytest + +from cloudinit import gpg +from cloudinit.config import cc_apt_configure +from cloudinit.util import is_true +from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.integration_settings import PLATFORM +from tests.integration_tests.releases import CURRENT_RELEASE, IS_UBUNTU +from tests.integration_tests.util import ( + get_feature_flag_value, + verify_clean_log, +) + +DEB822_SOURCES_FILE = "/etc/apt/sources.list.d/ubuntu.sources" +ORIG_SOURCES_FILE = "/etc/apt/sources.list" + +USER_DATA = """\ +#cloud-config +bootcmd: + - rm -f /etc/apt/sources.list /etc/apt/sources.list.d/ubuntu.sources + +apt: + conf: | + APT { + Get { + Assume-Yes "true"; + Fix-Broken "true"; + } + } + primary: + - arches: [default] + uri: http://badarchive.ubuntu.com/ubuntu + security: + - arches: [default] + uri: http://badsecurity.ubuntu.com/ubuntu + sources_list: | + deb $MIRROR $RELEASE main restricted + deb-src $MIRROR $RELEASE main restricted + deb $PRIMARY $RELEASE universe restricted + deb-src $PRIMARY $RELEASE universe restricted + deb $SECURITY $RELEASE-security multiverse + deb-src $SECURITY $RELEASE-security multiverse + sources: + test_keyserver: + keyid: 110E21D8B0E2A1F0243AF6820856F197B892ACEA + keyserver: keyserver.ubuntu.com + source: "deb http://ppa.launchpad.net/canonical-kernel-team/ppa/ubuntu $RELEASE main" + test_ppa: + keyid: 441614D8 + keyserver: keyserver.ubuntu.com + source: "ppa:simplestreams-dev/trunk" + test_signed_by: + keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 + keyserver: keyserver.ubuntu.com + source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" + test_bad_key: + key: "" + source: "deb $MIRROR $RELEASE main" + test_key: + source: "deb http://ppa.launchpad.net/cloud-init-dev/test-archive/ubuntu $RELEASE main" + key: | + -----BEGIN PGP PUBLIC KEY BLOCK----- + Version: SKS 1.1.6 + Comment: Hostname: keyserver.ubuntu.com + + mQINBFbZRUIBEAC+A0PIKYBP9kLC4hQtRrffRS11uLo8/BdtmOdrlW0hpPHzCfKnjR3tvSEI + lqPHG1QrrjAXKZDnZMRz+h/px7lUztvytGzHPSJd5ARUzAyjyRezUhoJ3VSCxrPqx62avuWf + RfoJaIeHfDehL5/dTVkyiWxfVZ369ZX6JN2AgLsQTeybTQ75+2z0xPrrhnGmgh6g0qTYcAaq + M5ONOGiqeSBX/Smjh6ALy5XkhUiFGLsI7Yluf6XSICY/x7gd6RAfgSIQrUTNMoS1sqhT4aot + +xvOfQy8ySkfAK4NddXql6E/+ZqTmBY/Lr0YklFBy8jGT+UysfiIznPMIwbmgq5Li7BtDDtX + b8Uyi4edPpjtextezfXYn4NVIpPL5dPZS/FXh4HpzyH0pYCfrH4QDGA7i52AGmhpiOFjJMo6 + N33sdjZHOH/2Vyp+QZaQnsdUAi1N4M6c33tQbpIScn1SY+El8z5JDA4PBzkw8HpLCi1gGoa6 + V4kfbWqXXbGAJFkLkP/vc4+pY9axOlmCkJg7xCPwhI75y1cONgovhz+BEXOzolh5KZuGbGbj + xe0wva5DLBeIg7EQFf+99pOS7Syby3Xpm6ZbswEFV0cllK4jf/QMjtfInxobuMoI0GV0bE5l + WlRtPCK5FnbHwxi0wPNzB/5fwzJ77r6HgPrR0OkT0lWmbUyoOQARAQABtC1MYXVuY2hwYWQg + UFBBIGZvciBjbG91ZCBpbml0IGRldmVsb3BtZW50IHRlYW2JAjgEEwECACIFAlbZRUICGwMG + CwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAg9Bvvk0wTfHfcP/REK5N2s1JYc69qEa9ZN + o6oi+A7l6AYw+ZY88O5TJe7F9otv5VXCIKSUT0Vsepjgf0mtXAgf/sb2lsJn/jp7tzgov3YH + vSrkTkRydz8xcA87gwQKePuvTLxQpftF4flrBxgSueIn5O/tPrBOxLz7EVYBc78SKg9aj9L2 + yUp+YuNevlwfZCTYeBb9r3FHaab2HcgkwqYch66+nKYfwiLuQ9NzXXm0Wn0JcEQ6pWvJscbj + C9BdawWovfvMK5/YLfI6Btm7F4mIpQBdhSOUp/YXKmdvHpmwxMCN2QhqYK49SM7qE9aUDbJL + arppSEBtlCLWhRBZYLTUna+BkuQ1bHz4St++XTR49Qd7vDERALpApDjB2dxPfMiBzCMwQQyq + uy13exU8o2ETLg+dZSLfDTzrBNsBFmXlw8WW17nTISYdKeGKL+QdlUjpzdwUMMzHhAO8SmMH + zjeSlDSRMXBJFAFSbCl7EwmMKa3yVX0zInT91fNllZ3iatAmtVdqVH/BFQfTIMH2ET7A8WzJ + ZzVSuMRhqoKdr5AMcHuJGPUoVkVJHQA+NNvEiXSysF3faL7jmKapmUwrhpYYX2H8pf+VMu2e + cLflKTI28dl+ZQ4Pl/aVsxrti/pzhdYy05Sn5ddtySyIkvo8L1cU5MWpbvSlFPkTstBUDLBf + pb0uBy+g0oxJQg15 + =uy53 + -----END PGP PUBLIC KEY BLOCK----- + test_write: + keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 + keyserver: keyserver.ubuntu.com + source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" + append: false + test_write.list: + keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 + keyserver: keyserver.ubuntu.com + source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/devel/ubuntu $RELEASE main" + append: false + test_append: + keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 + keyserver: keyserver.ubuntu.com + source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/stable/ubuntu $RELEASE main" + test_append.list: + keyid: A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 + keyserver: keyserver.ubuntu.com + source: "deb [signed-by=$KEY_FILE] http://ppa.launchpad.net/juju/devel/ubuntu $RELEASE main" +apt_pipelining: os +""" # noqa: E501 + +EXPECTED_REGEXES = [ + r"deb http://badarchive.ubuntu.com/ubuntu [a-z]+ main restricted", + r"deb-src http://badarchive.ubuntu.com/ubuntu [a-z]+ main restricted", + r"deb http://badarchive.ubuntu.com/ubuntu [a-z]+ universe restricted", + r"deb-src http://badarchive.ubuntu.com/ubuntu [a-z]+ universe restricted", + r"deb http://badsecurity.ubuntu.com/ubuntu [a-z]+-security multiverse", + r"deb-src http://badsecurity.ubuntu.com/ubuntu [a-z]+-security multiverse", +] + +TEST_KEYSERVER_KEY = "110E 21D8 B0E2 A1F0 243A F682 0856 F197 B892 ACEA" +TEST_PPA_KEY = "3552 C902 B4DD F7BD 3842 1821 015D 28D7 4416 14D8" +TEST_KEY = "1FF0 D853 5EF7 E719 E5C8 1B9C 083D 06FB E4D3 04DF" +TEST_SIGNED_BY_KEY = "A2EB 2DEC 0BD7 519B 7B38 BE38 376A 290E C806 8B11" + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") +@pytest.mark.user_data(USER_DATA) +class TestApt: + def get_keys(self, class_client: IntegrationInstance): + """Return all keys in /etc/apt/trusted.gpg.d/ and /etc/apt/trusted.gpg + in human readable format. Mimics the output of apt-key finger + """ + list_cmd = " ".join(gpg.GPG_LIST) + " " + keys = class_client.execute(list_cmd + cc_apt_configure.APT_LOCAL_KEYS) + print(keys) + files = class_client.execute( + "ls " + cc_apt_configure.APT_TRUSTED_GPG_DIR + ) + for file in files.split(): + path = cc_apt_configure.APT_TRUSTED_GPG_DIR + file + keys += class_client.execute(list_cmd + path) or "" + return keys + + def test_sources_list(self, class_client: IntegrationInstance): + """Integration test for the apt module's `sources_list` functionality. + + This test specifies a ``sources_list`` and then checks that (a) the + expected number of sources.list entries is present, and (b) that each + expected line appears in the file. + + Since sources_list is no deb822-compliant, ORIG_SOURCES_FILE will + always be written regardless of feature.APT_DEB822_SOURCE_LIST_FILE + """ + sources_list = class_client.read_from_file(ORIG_SOURCES_FILE) + assert 6 == len(sources_list.rstrip().split("\n")) + for expected_re in EXPECTED_REGEXES: + assert re.search(expected_re, sources_list) is not None + + def test_apt_conf(self, class_client: IntegrationInstance): + """Test the apt conf functionality.""" + apt_config = class_client.read_from_file( + "/etc/apt/apt.conf.d/94cloud-init-config" + ) + assert 'Assume-Yes "true";' in apt_config + assert 'Fix-Broken "true";' in apt_config + + def test_ppa_source(self, class_client: IntegrationInstance): + """Test the apt ppa functionality.""" + ppa_path = ( + "/etc/apt/sources.list.d/simplestreams-dev-ubuntu-trunk-{}".format( + CURRENT_RELEASE.series + ) + ) + if CURRENT_RELEASE.series < "mantic": + ppa_path += ".list" + else: + ppa_path += ".sources" + ppa_path_contents = class_client.read_from_file(ppa_path) + assert ( + "://ppa.launchpad.net/simplestreams-dev/trunk/ubuntu" + in ppa_path_contents + or "://ppa.launchpadcontent.net/simplestreams-dev/trunk/ubuntu" + in ppa_path_contents + ) + + assert TEST_PPA_KEY in self.get_keys(class_client) + + def test_signed_by(self, class_client: IntegrationInstance): + """Test the apt signed-by functionality.""" + source = ( + "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_signed_by.gpg] " + "http://ppa.launchpad.net/juju/stable/ubuntu" + " {} main".format(CURRENT_RELEASE.series) + ) + path_contents = class_client.read_from_file( + "/etc/apt/sources.list.d/test_signed_by.list" + ) + assert path_contents == source + + key = class_client.execute( + "gpg --no-default-keyring --with-fingerprint --list-keys " + "--keyring /etc/apt/cloud-init.gpg.d/test_signed_by.gpg" + ) + + assert TEST_SIGNED_BY_KEY in key + + def test_bad_key(self, class_client: IntegrationInstance): + """Test the apt signed-by functionality.""" + with pytest.raises(OSError): + class_client.read_from_file( + "/etc/apt/trusted.list.d/test_bad_key.gpg" + ) + + def test_key(self, class_client: IntegrationInstance): + """Test the apt key functionality.""" + test_archive_contents = class_client.read_from_file( + "/etc/apt/sources.list.d/test_key.list" + ) + + assert ( + "http://ppa.launchpad.net/cloud-init-dev/test-archive/ubuntu" + in test_archive_contents + ) + assert TEST_KEY in self.get_keys(class_client) + + def test_keyserver(self, class_client: IntegrationInstance): + """Test the apt keyserver functionality.""" + test_keyserver_contents = class_client.read_from_file( + "/etc/apt/sources.list.d/test_keyserver.list" + ) + + assert ( + "http://ppa.launchpad.net/canonical-kernel-team/ppa/ubuntu" + in test_keyserver_contents + ) + + assert TEST_KEYSERVER_KEY in self.get_keys(class_client) + + def test_os_pipelining(self, class_client: IntegrationInstance): + """Test 'os' settings does not write apt config file.""" + conf_exists = class_client.execute( + "test -f /etc/apt/apt.conf.d/90cloud-init-pipelining" + ).ok + assert conf_exists is False + + def test_sources_write(self, class_client: IntegrationInstance): + """Test overwrite or append to sources file""" + test_write_content = class_client.read_from_file( + "/etc/apt/sources.list.d/test_write.list" + ) + expected_contents = ( + "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_write.gpg] " + "http://ppa.launchpad.net/juju/devel/ubuntu " + f"{CURRENT_RELEASE.series} main" + ) + assert expected_contents.strip() == test_write_content.strip() + + def test_sources_append(self, class_client: IntegrationInstance): + series = CURRENT_RELEASE.series + test_append_content = class_client.read_from_file( + "/etc/apt/sources.list.d/test_append.list" + ) + + expected_contents = ( + "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_append.gpg] " + f"http://ppa.launchpad.net/juju/stable/ubuntu {series} main\n" + "deb [signed-by=/etc/apt/cloud-init.gpg.d/test_append.gpg] " + f"http://ppa.launchpad.net/juju/devel/ubuntu {series} main" + ) + assert expected_contents.strip() == test_append_content.strip() + + +_DEFAULT_DATA = """\ +#cloud-config +apt: + primary: + - arches: + - default + {uri} + security: + - arches: + - default +""" +DEFAULT_DATA = _DEFAULT_DATA.format(uri="") + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") +@pytest.mark.user_data(DEFAULT_DATA) +class TestDefaults: + @pytest.mark.skipif( + PLATFORM != "openstack", reason="Test is Openstack specific" + ) + def test_primary_on_openstack(self, class_client: IntegrationInstance): + """Test apt default primary source on openstack. + + When no uri is provided. + """ + zone = class_client.execute("cloud-init query v1.availability_zone") + feature_deb822 = is_true( + get_feature_flag_value(class_client, "APT_DEB822_SOURCE_LIST_FILE") + ) + src_file = DEB822_SOURCES_FILE if feature_deb822 else ORIG_SOURCES_FILE + sources_list = class_client.read_from_file(src_file) + assert "{}.clouds.archive.ubuntu.com".format(zone) in sources_list + + def test_security(self, class_client: IntegrationInstance): + """Test apt default security sources.""" + series = CURRENT_RELEASE.series + feature_deb822 = is_true( + get_feature_flag_value(class_client, "APT_DEB822_SOURCE_LIST_FILE") + ) + if class_client.settings.PLATFORM == "azure": + sec_url = "http://azure.archive.ubuntu.com/ubuntu/" + else: + sec_url = "http://security.ubuntu.com/ubuntu" + if feature_deb822: + expected_cfg = dedent( + f"""\ + Types: deb + URIs: {sec_url} + Suites: {series}-security + """ + ) + sources_list = class_client.read_from_file(DEB822_SOURCES_FILE) + assert expected_cfg in sources_list + else: + sources_list = class_client.read_from_file(ORIG_SOURCES_FILE) + # 3 lines from main, universe, and multiverse + sec_deb_line = f"deb {sec_url} {series}-security" + sec_src_deb_line = sec_deb_line.replace("deb ", "# deb-src ") + assert 3 == sources_list.count(sec_deb_line) + assert 3 == sources_list.count(sec_src_deb_line) + + +DEFAULT_DATA_WITH_URI = _DEFAULT_DATA.format( + uri='uri: "http://something.random.invalid/ubuntu"' +) + + +@pytest.mark.user_data(DEFAULT_DATA_WITH_URI) +def test_default_primary_with_uri(client: IntegrationInstance): + """Test apt default primary sources.""" + feature_deb822 = is_true( + get_feature_flag_value(client, "APT_DEB822_SOURCE_LIST_FILE") + ) + src_file = DEB822_SOURCES_FILE if feature_deb822 else ORIG_SOURCES_FILE + sources_list = client.read_from_file(src_file) + assert "archive.ubuntu.com" not in sources_list + assert "something.random.invalid" in sources_list + + +DISABLED_DATA = """\ +#cloud-config +bootcmd: [mkdir -p /etc/apt/sources.new.d] +apt: + conf: | + Dir::Etc::sourceparts "sources.new.d"; + disable_suites: + - $RELEASE + - $RELEASE-updates + - $RELEASE-backports + - $RELEASE-security +apt_pipelining: false +""" + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") +@pytest.mark.user_data(DISABLED_DATA) +class TestDisabled: + def test_disable_suites(self, class_client: IntegrationInstance): + """Test disabling of apt suites.""" + feature_deb822 = is_true( + get_feature_flag_value(class_client, "APT_DEB822_SOURCE_LIST_FILE") + ) + if feature_deb822: + # DISABLED_DATA changes Dir:Etc::sourceparts to sources.new.d + src_file = DEB822_SOURCES_FILE.replace("list", "new") + else: + src_file = ORIG_SOURCES_FILE + sources_list = class_client.execute( + f"cat {src_file} | grep -v '^#'" + ).strip() + assert "" == sources_list + + def test_disable_apt_pipelining(self, class_client: IntegrationInstance): + """Test disabling of apt pipelining.""" + conf = class_client.read_from_file( + "/etc/apt/apt.conf.d/90cloud-init-pipelining" + ) + assert 'Acquire::http::Pipeline-Depth "0";' in conf + + +APT_PROXY_DATA = """\ +#cloud-config +apt: + proxy: "http://proxy.internal:3128" + http_proxy: "http://squid.internal:3128" + ftp_proxy: "ftp://squid.internal:3128" + https_proxy: "https://squid.internal:3128" +""" + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") +@pytest.mark.user_data(APT_PROXY_DATA) +def test_apt_proxy(client: IntegrationInstance): + """Test the apt proxy data gets written correctly.""" + out = client.read_from_file("/etc/apt/apt.conf.d/90cloud-init-aptproxy") + assert 'Acquire::http::Proxy "http://proxy.internal:3128";' in out + assert 'Acquire::http::Proxy "http://squid.internal:3128";' in out + assert 'Acquire::ftp::Proxy "ftp://squid.internal:3128";' in out + assert 'Acquire::https::Proxy "https://squid.internal:3128";' in out + + +INSTALL_ANY_MISSING_RECOMMENDED_DEPENDENCIES = """\ +#cloud-config +bootcmd: + - apt-get remove gpg -y +apt: + sources: + test_keyserver: + keyid: 110E21D8B0E2A1F0243AF6820856F197B892ACEA + keyserver: keyserver.ubuntu.com + source: "deb http://ppa.launchpad.net/canonical-kernel-team/ppa/ubuntu $RELEASE main" + test_ppa: + keyid: 441614D8 + keyserver: keyserver.ubuntu.com + source: "ppa:simplestreams-dev/trunk" +""" # noqa: E501 + + +RE_GPG_SW_PROPERTIES_INSTALLED = ( + r"install" + r" (gnupg software-properties-common|software-properties-common gnupg)" +) + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Apt usage") +@pytest.mark.user_data(INSTALL_ANY_MISSING_RECOMMENDED_DEPENDENCIES) +def test_install_missing_deps(client: IntegrationInstance): + log = client.read_from_file("/var/log/cloud-init.log") + verify_clean_log(log) + assert re.search(RE_GPG_SW_PROPERTIES_INSTALLED, log) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_boothook.py cloud-init-23.4.3/tests/integration_tests/modules/test_boothook.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_boothook.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_boothook.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,39 @@ +# This file is part of cloud-init. See LICENSE file for license information. +import re + +import pytest + +from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.util import verify_clean_log + +USER_DATA = """\ +#cloud-boothook +#!/bin/sh +# Error below will generate stderr +BOOTHOOK/0 +echo BOOTHOOKstdout +echo "BOOTHOOK: $INSTANCE_ID: is called every boot." >> /boothook.txt +""" + + +@pytest.mark.user_data(USER_DATA) +def test_boothook_header_runs_part_per_instance(client: IntegrationInstance): + """Test boothook handling creates a script that runs per-boot. + Streams stderr and stdout are directed to /var/log/cloud-init-output.log. + """ + instance_id = client.instance.execute("cloud-init query instance-id") + RE_BOOTHOOK = f"BOOTHOOK: {instance_id}: is called every boot" + log = client.read_from_file("/var/log/cloud-init.log") + verify_clean_log(log) + output = client.read_from_file("/boothook.txt") + assert 1 == len(re.findall(RE_BOOTHOOK, output)) + client.restart() + output = client.read_from_file("/boothook.txt") + assert 2 == len(re.findall(RE_BOOTHOOK, output)) + output_log = client.read_from_file("/var/log/cloud-init-output.log") + expected_msgs = [ + "BOOTHOOKstdout", + "boothooks/part-001: 3: BOOTHOOK/0: not found", + ] + for msg in expected_msgs: + assert msg in output_log diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_cli.py cloud-init-23.4.3/tests/integration_tests/modules/test_cli.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_cli.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_cli.py 2024-02-02 21:21:52.000000000 +0000 @@ -6,6 +6,7 @@ import pytest from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.integration_settings import PLATFORM VALID_USER_DATA = """\ #cloud-config @@ -41,7 +42,7 @@ """ result = client.execute("cloud-init schema --system") assert result.ok - assert "Valid cloud-config: user-data" in result.stdout.strip() + assert "Valid schema user-data" in result.stdout.strip() result = client.execute("cloud-init status --long") if not result.ok: raise AssertionError( @@ -49,6 +50,9 @@ ) +@pytest.mark.skipif( + PLATFORM == "qemu", reason="QEMU only supports #cloud-config userdata" +) @pytest.mark.user_data(INVALID_USER_DATA_HEADER) def test_invalid_userdata(client: IntegrationInstance): """Test `cloud-init schema` with invalid userdata. @@ -63,10 +67,9 @@ " #cloud-boothook, #cloud-config" in result.stderr ) result = client.execute("cloud-init status --long") - if not result.ok: - raise AssertionError( - f"Unexpected error from cloud-init status: {result}" - ) + assert ( + 2 == result.return_code + ), f"Unexpected exit code {result.return_code}" @pytest.mark.user_data(INVALID_USER_DATA_SCHEMA) @@ -76,7 +79,9 @@ PR #1175 """ result = client.execute("cloud-init status --long") - assert result.ok + assert ( + 2 == result.return_code + ), f"Unexpected exit code {result.return_code}" log = client.read_from_file("/var/log/cloud-init.log") warning = ( "[WARNING]: Invalid cloud-config provided: Please run " @@ -84,9 +89,3 @@ ) assert warning in log assert "asdfasdf" not in log - - result = client.execute("cloud-init status --long") - if not result.ok: - raise AssertionError( - f"Unexpected error from cloud-init status: {result}" - ) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_combined.py cloud-init-23.4.3/tests/integration_tests/modules/test_combined.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_combined.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_combined.py 2024-02-02 21:21:52.000000000 +0000 @@ -78,7 +78,7 @@ ssh_import_id: - lp:smoser -timezone: US/Aleutian +timezone: Europe/Madrid """ @@ -207,7 +207,7 @@ timezone_output = client.execute( 'date "+%Z" --date="Thu, 03 Nov 2016 00:47:00 -0400"' ) - assert timezone_output.strip() == "HDT" + assert timezone_output.strip() == "CET" def test_no_problems(self, class_client: IntegrationInstance): """Test no errors, warnings, deprecations, tracebacks or @@ -264,6 +264,7 @@ "gce": "DataSourceGCELocal", "oci": "DataSourceOracle", "openstack": "DataSourceOpenStackLocal [net,ver=2]", + "qemu": "DataSourceNoCloud [seed=/dev/vda][dsmode=net]", } assert ( platform_datasources[client.settings.PLATFORM] @@ -307,7 +308,6 @@ assert data["sys_info"]["dist"][0] == CURRENT_RELEASE.os v1_data = data["v1"] - assert re.match(r"\d\.\d+\.\d+-\d+", v1_data["kernel_release"]) assert v1_data["variant"] == CURRENT_RELEASE.os assert v1_data["distro"] == CURRENT_RELEASE.os assert v1_data["distro_release"] == CURRENT_RELEASE.series @@ -327,6 +327,27 @@ assert data["system_info"]["default_user"]["name"] == "ubuntu" @pytest.mark.skipif( + PLATFORM not in ("lxd_vm", "lxd_container"), + reason="Test is LXD specific", + ) + def test_network_config_json(self, class_client: IntegrationInstance): + client = class_client + network_json = client.read_from_file( + "/run/cloud-init/network-config.json" + ) + devname = "eth0" if PLATFORM == "lxd_container" else "enp5s0" + assert { + "config": [ + { + "name": devname, + "subnets": [{"control": "auto", "type": "dhcp"}], + "type": "physical", + } + ], + "version": 1, + } == json.loads(network_json) + + @pytest.mark.skipif( PLATFORM != "lxd_container", reason="Test is LXD container specific", ) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_disk_setup.py cloud-init-23.4.3/tests/integration_tests/modules/test_disk_setup.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_disk_setup.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_disk_setup.py 2024-02-02 21:21:52.000000000 +0000 @@ -191,7 +191,7 @@ UPDATED_PARTPROBE_USERDATA, ) client.execute( - "sed -i 's/write_files/write_files\\n - mounts/' " + "sed -i 's/write_files$/write_files\\n - mounts/' " "/etc/cloud/cloud.cfg" ) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_jinja_templating.py cloud-init-23.4.3/tests/integration_tests/modules/test_jinja_templating.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_jinja_templating.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_jinja_templating.py 2024-02-02 21:21:52.000000000 +0000 @@ -2,6 +2,7 @@ import pytest from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.integration_settings import PLATFORM from tests.integration_tests.util import ( verify_clean_log, verify_ordered_items_in_text, @@ -22,6 +23,9 @@ """ +@pytest.mark.skipif( + PLATFORM == "qemu", reason="QEMU only supports #cloud-config header" +) @pytest.mark.user_data(USER_DATA) def test_runcmd_with_variable_substitution(client: IntegrationInstance): """Test jinja substitution. diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_package_update_upgrade_install.py cloud-init-23.4.3/tests/integration_tests/modules/test_package_update_upgrade_install.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_package_update_upgrade_install.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_package_update_upgrade_install.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,28 +1,28 @@ +# This file is part of cloud-init. See LICENSE file for license information. """Integration test for the package update upgrade install module. This test module asserts that packages are upgraded/updated during boot with the ``package_update_upgrade_install`` module. We are also testing if we can install new packages during boot too. - -(This is ported from -``tests/cloud_tests/testcases/modules/package_update_upgrade_install.yaml``.) - -NOTE: the testcase for this looks for the command in history.log as - /usr/bin/apt-get..., which is not how it always appears. it should - instead look for just apt-get... """ import re import pytest -from tests.integration_tests.releases import IS_UBUNTU +from tests.integration_tests.clouds import IntegrationCloud +from tests.integration_tests.releases import CURRENT_RELEASE, IS_UBUNTU +from tests.integration_tests.util import verify_clean_log USER_DATA = """\ #cloud-config packages: - sl - - tree + - apt: + - tree + - snap: + - curl + - postman package_update: true package_upgrade: true """ @@ -31,7 +31,7 @@ @pytest.mark.skipif(not IS_UBUNTU, reason="Uses Apt") @pytest.mark.user_data(USER_DATA) class TestPackageUpdateUpgradeInstall: - def assert_package_installed(self, pkg_out, name, version=None): + def assert_apt_package_installed(self, pkg_out, name, version=None): """Check dpkg-query --show output for matching package name. @param name: package base name @@ -52,25 +52,26 @@ version, installed_version, ) - raise AssertionError("Package not installed: %s" % name) + raise AssertionError(f"Package not installed: {name}") - def test_new_packages_are_installed(self, class_client): + def test_apt_packages_are_installed(self, class_client): pkg_out = class_client.execute("dpkg-query --show") - self.assert_package_installed(pkg_out, "sl") - self.assert_package_installed(pkg_out, "tree") + self.assert_apt_package_installed(pkg_out, "sl") + self.assert_apt_package_installed(pkg_out, "tree") - def test_packages_were_updated(self, class_client): + def test_apt_packages_were_updated(self, class_client): out = class_client.execute( "grep ^Commandline: /var/log/apt/history.log" ) - assert ( + assert re.search( "Commandline: /usr/bin/apt-get --option=Dpkg::Options" "::=--force-confold --option=Dpkg::options::=--force-unsafe-io " - "--assume-yes --quiet install sl tree" in out + r"--assume-yes --quiet install (sl|tree) (tree|sl)", + out, ) - def test_packages_were_upgraded(self, class_client): + def test_apt_packages_were_upgraded(self, class_client): """Test cloud-init-output for install & upgrade stuff.""" out = class_client.read_from_file("/var/log/cloud-init-output.log") assert "Setting up tree (" in out @@ -79,3 +80,39 @@ assert "Building dependency tree..." in out assert "Reading state information..." in out assert "Calculating upgrade..." in out + + def test_snap_packages_are_installed(self, class_client): + output = class_client.execute("snap list") + assert "curl" in output + assert "postman" in output + + +HELLO_VERSIONS_BY_RELEASE = { + "mantic": "2.10-3", + "lunar": "2.10-3", + "jammy": "2.10-2ubuntu4", + "focal": "2.10-2ubuntu2", +} + +VERSIONED_USER_DATA = """\ +#cloud-config +packages: +- [hello, {pkg_version}] +""" + + +@pytest.mark.skipif(not IS_UBUNTU, reason="Uses Apt") +def test_versioned_packages_are_installed(session_cloud: IntegrationCloud): + pkg_version = HELLO_VERSIONS_BY_RELEASE.get( + CURRENT_RELEASE.series, "2.10-3" + ) + with session_cloud.launch( + user_data=VERSIONED_USER_DATA.format(pkg_version=pkg_version) + ) as client: + verify_clean_log(client.read_from_file("/var/log/cloud-init.log")) + assert f"hello {pkg_version}" == client.execute( + "dpkg-query -W hello" + ), ( + "If this is failing for a new release, add it to " + "HELLO_VERSIONS_BY_RELEASE" + ) diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_ssh_keysfile.py cloud-init-23.4.3/tests/integration_tests/modules/test_ssh_keysfile.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_ssh_keysfile.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_ssh_keysfile.py 2024-02-02 21:21:52.000000000 +0000 @@ -5,6 +5,7 @@ from paramiko.ssh_exception import SSHException from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.integration_settings import PLATFORM from tests.integration_tests.releases import CURRENT_RELEASE, IS_UBUNTU from tests.integration_tests.util import get_test_rsa_keypair @@ -128,6 +129,10 @@ @pytest.mark.skipif( not IS_UBUNTU, reason="Tests permissions specific to Ubuntu releases" ) +@pytest.mark.skipif( + PLATFORM == "qemu", + reason="QEMU cloud manually adding key interferes with test", +) @pytest.mark.user_data(DEFAULT_KEYS_USERDATA) def test_authorized_keys_default(client: IntegrationInstance): expected_keys = [ @@ -159,6 +164,10 @@ @pytest.mark.skipif( not IS_UBUNTU, reason="Tests permissions specific to Ubuntu releases" ) +@pytest.mark.skipif( + PLATFORM == "qemu", + reason="QEMU cloud manually adding key interferes with test", +) @pytest.mark.user_data(AUTHORIZED_KEYS2_USERDATA) def test_authorized_keys2(client: IntegrationInstance): expected_keys = [ @@ -190,6 +199,10 @@ @pytest.mark.skipif( not IS_UBUNTU, reason="Tests permissions specific to Ubuntu releases" ) +@pytest.mark.skipif( + PLATFORM == "qemu", + reason="QEMU cloud manually adding key interferes with test", +) @pytest.mark.user_data(NESTED_KEYS_USERDATA) def test_nested_keys(client: IntegrationInstance): expected_keys = [ @@ -213,6 +226,10 @@ @pytest.mark.skipif( not IS_UBUNTU, reason="Tests permissions specific to Ubuntu releases" ) +@pytest.mark.skipif( + PLATFORM == "qemu", + reason="QEMU cloud manually adding key interferes with test", +) @pytest.mark.user_data(EXTERNAL_KEYS_USERDATA) def test_external_keys(client: IntegrationInstance): expected_keys = [ diff -Nru cloud-init-23.3.3/tests/integration_tests/modules/test_ubuntu_advantage.py cloud-init-23.4.3/tests/integration_tests/modules/test_ubuntu_advantage.py --- cloud-init-23.3.3/tests/integration_tests/modules/test_ubuntu_advantage.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/modules/test_ubuntu_advantage.py 2024-02-02 21:21:52.000000000 +0000 @@ -142,13 +142,26 @@ verify_clean_log(log) assert is_attached(client) + # Assert service-already-enabled handling for esm-infra. + # First totally destroy ubuntu-advantage-tools data and state. + # This is a hack but results in a system that thinks it + # is detached even though esm-infra is still enabled. + # When cloud-init runs again, it will successfully re-attach + # and then notice that esm-infra is already enabled. + client.execute("rm -rf /var/lib/ubuntu-advantage") + assert client.execute("cloud-init clean --logs").ok + client.restart() + log = client.read_from_file("/var/log/cloud-init.log") + verify_clean_log(log) + assert "Service `esm-infra` already enabled" in log + def maybe_install_cloud_init(session_cloud: IntegrationCloud): source = get_validated_source(session_cloud) launch_kwargs = { "image_id": session_cloud.cloud_instance.daily_image( - CURRENT_RELEASE.image_id, image_type=ImageType.PRO + CURRENT_RELEASE.series, image_type=ImageType.PRO ) } diff -Nru cloud-init-23.3.3/tests/integration_tests/releases.py cloud-init-23.4.3/tests/integration_tests/releases.py --- cloud-init-23.3.3/tests/integration_tests/releases.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/releases.py 2024-02-02 21:21:52.000000000 +0000 @@ -5,31 +5,24 @@ from packaging import version from cloudinit import subp -from cloudinit.subp import ProcessExecutionError from tests.integration_tests import integration_settings log = logging.getLogger("integration_testing") -def get_all_ubuntu_series() -> list: - """Use distro-info-data's ubuntu.csv to get a list of Ubuntu series""" - out = "" - try: - out, _err = subp.subp(["ubuntu-distro-info", "-a"]) - except ProcessExecutionError: - log.info( - "ubuntu-distro-info (from the distro-info package) must be" - " installed to guess Ubuntu os/release" - ) - return out.splitlines() - - def ubuntu_version_from_series(series) -> str: + running_ubuntu = subp.which("ubuntu-distro-info") try: - out, _err = subp.subp( + out = subp.subp( ["ubuntu-distro-info", "--release", "--series", series] - ) + ).stdout except subp.ProcessExecutionError as e: + if not running_ubuntu: + log.info( + "ubuntu-distro-info (from the distro-info package) not " + " installed, attempting pass through Ubuntu os/release" + ) + return series raise ValueError( f"'{series}' is not a recognized Ubuntu release" ) from e @@ -53,6 +46,21 @@ def __repr__(self): return f"Release({self.os}, {self.version})" + def __eq__(self, other: object): + if not isinstance(other, Release): + return False + if all( + [ + self.os == other.os, + self.series == other.series, + self.version == other.version, + ] + ): + if self.image_id and other.image_id: + return self.image_id == other.image_id + return True # If either image_id is None then ignore it + return False + def __lt__(self, other: "Release"): if self.os != other.os: raise ValueError(f"{self.os} cannot be compared to {other.os}!") diff -Nru cloud-init-23.3.3/tests/integration_tests/test_kernel_commandline_match.py cloud-init-23.4.3/tests/integration_tests/test_kernel_commandline_match.py --- cloud-init-23.3.3/tests/integration_tests/test_kernel_commandline_match.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/test_kernel_commandline_match.py 2024-02-02 21:21:52.000000000 +0000 @@ -11,7 +11,24 @@ log = logging.getLogger("integration_testing") -def override_kernel_cmdline(ds_str: str, c: IntegrationInstance) -> str: +def restart_cloud_init(c): + client = c + client.instance.shutdown(wait=False) + try: + client.instance.wait_for_state("STOPPED", num_retries=20) + except RuntimeError as e: + log.warning( + "Retrying shutdown due to timeout on initial shutdown request %s", + str(e), + ) + client.instance.shutdown() + + client.instance.execute_via_ssh = False + client.instance.start() + client.execute("cloud-init status --wait") + + +def override_kernel_cmdline(ds_str: str, c: IntegrationInstance): """ Configure grub's kernel command line to tell cloud-init to use OpenStack - even though LXD should naturally be detected. @@ -44,20 +61,7 @@ # most likely be as simple as updating the output path for grub-mkconfig client.execute("grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg") client.execute("cloud-init clean --logs") - client.instance.shutdown(wait=False) - try: - client.instance.wait_for_state("STOPPED", num_retries=20) - except RuntimeError as e: - log.warning( - "Retrying shutdown due to timeout on initial shutdown request %s", - str(e), - ) - client.instance.shutdown() - - client.instance.execute_via_ssh = False - client.instance.start() - client.execute("cloud-init status --wait") - return client.execute("cat /var/log/cloud-init.log") + restart_cloud_init(client) @pytest.mark.skipif(PLATFORM != "lxd_vm", reason="Modifies grub config") @@ -85,10 +89,11 @@ kernel commandline in Python code is required. """ + override_kernel_cmdline(ds_str, client) assert ( "Machine is configured by the kernel commandline to run on single " f"datasource {configured}" - ) in override_kernel_cmdline(ds_str, client) + ) in client.execute("cat /var/log/cloud-init.log") GH_REPO_PATH = "https://raw.githubusercontent.com/canonical/cloud-init/main/" @@ -129,7 +134,9 @@ client.install_new_cloud_init( source, take_snapshot=False, clean=False ) - logs = override_kernel_cmdline(ds_str, client) + override_kernel_cmdline(ds_str, client) + + logs = client.execute("cat /var/log/cloud-init.log") assert ( "nocloud" == client.execute("cloud-init query platform").stdout.strip() @@ -139,3 +146,40 @@ "Detected platform: DataSourceNoCloudNet [seed=None]" "[dsmode=net]. Checking for active instance data" ) in logs + + +@pytest.mark.skipif(PLATFORM != "lxd_vm", reason="Modifies grub config") +@pytest.mark.lxd_use_exec +def test_lxd_disable_cloud_init_cmdline(client: IntegrationInstance): + """Verify cloud-init disablement via kernel commandline works.""" + + override_kernel_cmdline("cloud-init=disabled", client) + assert "Active: inactive (dead)" in client.execute( + "systemctl status cloud-init" + ) + + +@pytest.mark.lxd_use_exec +def test_lxd_disable_cloud_init_file(client: IntegrationInstance): + """Verify cloud-init disablement via file works.""" + + client.execute("touch /etc/cloud/cloud-init.disabled") + client.execute("cloud-init --clean") + restart_cloud_init(client) + assert "Active: inactive (dead)" in client.execute( + "systemctl status cloud-init" + ) + + +@pytest.mark.lxd_use_exec +def test_lxd_disable_cloud_init_env(client: IntegrationInstance): + """Verify cloud-init disablement via environment variable works.""" + env = """DefaultEnvironment=KERNEL_CMDLINE=cloud-init=disabled""" + + client.execute(f'echo "{env}" >> /etc/systemd/system.conf') + + client.execute("cloud-init --clean") + restart_cloud_init(client) + assert "Active: inactive (dead)" in client.execute( + "systemctl status cloud-init" + ) diff -Nru cloud-init-23.3.3/tests/integration_tests/test_multi_part_user_data_handling.py cloud-init-23.4.3/tests/integration_tests/test_multi_part_user_data_handling.py --- cloud-init-23.3.3/tests/integration_tests/test_multi_part_user_data_handling.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/test_multi_part_user_data_handling.py 2024-02-02 21:21:52.000000000 +0000 @@ -7,6 +7,7 @@ from cloudinit.cmd.devel.make_mime import create_mime_message from tests.integration_tests.instances import IntegrationInstance +from tests.integration_tests.integration_settings import PLATFORM PER_FREQ_TEMPLATE = """\ #!/bin/bash @@ -63,7 +64,22 @@ ] MIME_WITH_ERRORS = create_mime_message(CLOUD_CONFIG_FILES)[0].as_string() +CLOUD_CONFIG_ARCHIVE = """ +#cloud-config-archive +- type: 'text/cloud-boothook' + content: | + #!/bin/sh + echo "BOOTHOOK: $(date -R): this is called every boot." | tee /run/boothook.txt +- type: 'text/cloud-config' + content: | + bootcmd: + - [sh, -c, 'echo "BOOTCMD: $(date -R): $INSTANCE_ID" | tee /run/bootcmd.txt'] +""" # noqa: E501 + +@pytest.mark.skipif( + PLATFORM == "qemu", reason="QEMU only supports #cloud-config userdata" +) @pytest.mark.ci @pytest.mark.user_data(USER_DATA) def test_per_freq(client: IntegrationInstance): @@ -119,6 +135,9 @@ # E2: \d+ is not of type 'string'""" +@pytest.mark.skipif( + PLATFORM == "qemu", reason="QEMU only supports #cloud-config userdata" +) @pytest.mark.ci @pytest.mark.user_data(MIME_WITH_ERRORS) def test_mime_with_error_parts(client: IntegrationInstance): @@ -132,3 +151,17 @@ assert ( "Failed at merging in cloud config part from cfg-invalid.yaml" in log ) + + +@pytest.mark.ci +@pytest.mark.user_data(CLOUD_CONFIG_ARCHIVE) +def test_cloud_config_archive_boot_hook_logging(client: IntegrationInstance): + """ + boot-hook and bootcmd scripts run per boot and log to cloud-init-outout.log + """ + cmd = "cloud-init schema --system" + assert "Valid schema user-data" in client.execute(cmd).stdout + client.restart() + log = client.read_from_file("/var/log/cloud-init-output.log") + assert 2 == len(re.findall("BOOTHOOK:.*", log)) + assert 2 == len(re.findall("BOOTCMD:.*", log)) diff -Nru cloud-init-23.3.3/tests/integration_tests/util.py cloud-init-23.4.3/tests/integration_tests/util.py --- cloud-init-23.3.3/tests/integration_tests/util.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/integration_tests/util.py 2024-02-02 21:21:52.000000000 +0000 @@ -55,7 +55,7 @@ "Found unexpected errors: %s" % "\n".join(error_logs) ) - warning_count = log.count("WARN") + warning_count = log.count("[WARNING]") expected_warnings = 0 traceback_count = log.count("Traceback") expected_tracebacks = 0 @@ -69,6 +69,17 @@ "WARNING]: Could not match supplied host pattern, ignoring:", ] traceback_texts = [] + if "install canonical-livepatch" in log: + # Ubuntu Pro Client emits a warning in between installing livepatch + # and enabling it + warning_texts.append( + "canonical-livepatch returned error when checking status" + ) + if "found network data from DataSourceNone" in log: + warning_texts.append("Used fallback datasource") + warning_texts.append( + "Falling back to a hard restart of systemd-networkd.service" + ) if "oracle" in log: # LP: #1842752 lease_exists_text = "Stderr: RTNETLINK answers: File exists" diff -Nru cloud-init-23.3.3/tests/unittests/analyze/test_boot.py cloud-init-23.4.3/tests/unittests/analyze/test_boot.py --- cloud-init-23.3.3/tests/unittests/analyze/test_boot.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/analyze/test_boot.py 2024-02-02 21:21:52.000000000 +0000 @@ -2,7 +2,7 @@ import pytest -from cloudinit.analyze.__main__ import analyze_boot, get_parser +from cloudinit.analyze import analyze_boot, get_parser from cloudinit.analyze.show import ( CONTAINER_CODE, FAIL_CODE, @@ -158,7 +158,7 @@ @mock.patch("cloudinit.util.is_container", return_value=True) @mock.patch("cloudinit.subp.subp", return_value=("U=1000000", None)) @mock.patch( - "cloudinit.analyze.__main__._get_events", + "cloudinit.analyze._get_events", return_value=[ { "name": "init-local", diff -Nru cloud-init-23.3.3/tests/unittests/cmd/devel/test_logs.py cloud-init-23.4.3/tests/unittests/cmd/devel/test_logs.py --- cloud-init-23.3.3/tests/unittests/cmd/devel/test_logs.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/devel/test_logs.py 2024-02-02 21:21:52.000000000 +0000 @@ -183,9 +183,22 @@ subp(cmd) # Pass through tar cmd so we can check output return SubpResult(expected_subp[cmd_tuple], "") + def fake_subprocess_call(cmd, stdout=None, stderr=None): + cmd_tuple = tuple(cmd) + if cmd_tuple not in expected_subp: + raise AssertionError( + "Unexpected command provided to subprocess: {0}".format( + cmd + ) + ) + stdout.write(expected_subp[cmd_tuple]) + fake_stderr = mock.MagicMock() mocker.patch(M_PATH + "subp", side_effect=fake_subp) + mocker.patch( + M_PATH + "subprocess.call", side_effect=fake_subprocess_call + ) mocker.patch(M_PATH + "sys.stderr", fake_stderr) mocker.patch(M_PATH + "CLOUDINIT_LOGS", [log1, log2]) mocker.patch(M_PATH + "CLOUDINIT_RUN_DIR", run_dir) diff -Nru cloud-init-23.3.3/tests/unittests/cmd/devel/test_net_convert.py cloud-init-23.4.3/tests/unittests/cmd/devel/test_net_convert.py --- cloud-init-23.3.3/tests/unittests/cmd/devel/test_net_convert.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/devel/test_net_convert.py 2024-02-02 21:21:52.000000000 +0000 @@ -58,7 +58,7 @@ """ SAMPLE_SYSCONFIG_CONTENT = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp DEVICE=eth0 @@ -225,6 +225,3 @@ net_convert.handle_args("somename", args) outfile = tmpdir.join("etc/netplan/50-cloud-init.yaml") assert yaml.load(content) == yaml.load(outfile.read()) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/cmd/test_clean.py cloud-init-23.4.3/tests/unittests/cmd/test_clean.py --- cloud-init-23.3.3/tests/unittests/cmd/test_clean.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/test_clean.py 2024-02-02 21:21:52.000000000 +0000 @@ -7,12 +7,13 @@ import cloudinit.settings from cloudinit.cmd import clean -from cloudinit.util import ensure_dir, sym_link +from cloudinit.util import ensure_dir, sym_link, write_file from tests.unittests.helpers import mock, wrap_and_call MyPaths = namedtuple("MyPaths", "cloud_dir") CleanPaths = namedtuple( - "CleanPaths", ["tmpdir", "cloud_dir", "clean_dir", "log", "output_log"] + "CleanPaths", + ["tmpdir", "cloud_dir", "clean_dir", "log", "output_log"], ) @@ -69,6 +70,148 @@ ), f"Unexpected file {clean_paths.output_log}" assert 0 == retcode + def test_remove_net_conf(self, clean_paths, init_class): + """remove_config removes config files when True.""" + + TEST_GEN_NET_CONFIG_FILES = [ + clean_paths.tmpdir.join(conf_file) + for conf_file in clean.GEN_NET_CONFIG_FILES + ] + for conf_path in TEST_GEN_NET_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected {conf_path}" + ensure_dir(os.path.dirname(conf_path)) + if "*" in conf_path.strpath: + # Expand glob path to interface-specific eth0 filename + eth0_path = conf_path.strpath.replace("*", "eth0") + write_file( + eth0_path, f"#generated by cloud-init test {eth0_path}" + ) + else: + conf_path.write(f"#generated by cloud-init test {conf_path}") + + with mock.patch( + "cloudinit.cmd.clean.GEN_NET_CONFIG_FILES", + [f.strpath for f in TEST_GEN_NET_CONFIG_FILES], + ): + retcode = wrap_and_call( + "cloudinit.cmd.clean", + {"Init": {"side_effect": init_class}}, + clean.remove_artifacts, + remove_logs=False, + remove_config=["network"], + ) + + for conf_path in TEST_GEN_NET_CONFIG_FILES: + if "*" in conf_path.strpath: + # Expand glob path to interface-specific eth0 filename + assert ( + os.path.exists(conf_path.strpath.replace("*", "eth0")) + is False + ), f"Unexpected file {conf_path.strpath.replace('*', 'eth0')}" + else: + assert ( + conf_path.exists() is False + ), f"Unexpected file {conf_path}" + assert 0 == retcode + + def test_remove_ssh_conf(self, clean_paths, init_class): + """remove_config removes config files when True.""" + + TEST_GEN_SSH_CONFIG_FILES = [ + clean_paths.tmpdir.join(conf_file) + for conf_file in clean.GEN_SSH_CONFIG_FILES + ] + for conf_path in TEST_GEN_SSH_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected {conf_path}" + ensure_dir(os.path.dirname(conf_path)) + conf_path.write(f"#generated by cloud-init\ntouch {conf_path}\n") + + with mock.patch( + "cloudinit.cmd.clean.GEN_SSH_CONFIG_FILES", + TEST_GEN_SSH_CONFIG_FILES, + ): + retcode = wrap_and_call( + "cloudinit.cmd.clean", + {"Init": {"side_effect": init_class}}, + clean.remove_artifacts, + remove_logs=False, + remove_config=["ssh_config"], + ) + + for conf_path in TEST_GEN_SSH_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected file {conf_path}" + assert 0 == retcode + + def test_remove_all_conf(self, clean_paths, init_class): + """remove_config removes config files when True.""" + + TEST_GEN_NET_CONFIG_FILES = [ + clean_paths.tmpdir.join(conf_file) + for conf_file in clean.GEN_NET_CONFIG_FILES + ] + for conf_path in TEST_GEN_NET_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected {conf_path}" + ensure_dir(os.path.dirname(conf_path)) + conf_path.write(f"#generated by cloud-init\ntouch {conf_path}\n") + + TEST_GEN_SSH_CONFIG_FILES = [ + clean_paths.tmpdir.join(conf_file) + for conf_file in clean.GEN_SSH_CONFIG_FILES + ] + for conf_path in TEST_GEN_SSH_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected {conf_path}" + ensure_dir(os.path.dirname(conf_path)) + conf_path.write(f"#generated by cloud-init\ntouch {conf_path}\n") + + with mock.patch( + "cloudinit.cmd.clean.GEN_NET_CONFIG_FILES", + [f.strpath for f in TEST_GEN_NET_CONFIG_FILES], + ), mock.patch( + "cloudinit.cmd.clean.GEN_SSH_CONFIG_FILES", + TEST_GEN_SSH_CONFIG_FILES, + ): + retcode = wrap_and_call( + "cloudinit.cmd.clean", + {"Init": {"side_effect": init_class}}, + clean.remove_artifacts, + remove_logs=False, + remove_config=["all"], + ) + + for conf_path in TEST_GEN_NET_CONFIG_FILES: + assert conf_path.exists() is False, f"file {conf_path} exists!" + for conf_path in TEST_GEN_SSH_CONFIG_FILES: + assert conf_path.exists() is False, f"file {conf_path} exists!" + assert 0 == retcode + + def test_keep_net_conf(self, clean_paths, init_class): + """remove_config removes config files when True.""" + + TEST_GEN_NET_CONFIG_FILES = [ + clean_paths.tmpdir.join(conf_file) + for conf_file in clean.GEN_NET_CONFIG_FILES + ] + for conf_path in TEST_GEN_NET_CONFIG_FILES: + assert conf_path.exists() is False, f"Unexpected {conf_path}" + ensure_dir(os.path.dirname(conf_path)) + conf_path.write(f"#generated by cloud-init\ntouch {conf_path}\n") + + with mock.patch( + "cloudinit.cmd.clean.GEN_NET_CONFIG_FILES", + TEST_GEN_NET_CONFIG_FILES, + ): + retcode = wrap_and_call( + "cloudinit.cmd.clean", + {"Init": {"side_effect": init_class}}, + clean.remove_artifacts, + remove_logs=False, + remove_config=[], + ) + + for conf_path in TEST_GEN_NET_CONFIG_FILES: + assert conf_path.exists() is True, f"file {conf_path} removed!" + assert 0 == retcode + @pytest.mark.allow_all_subp def test_remove_artifacts_runparts_clean_d(self, clean_paths, init_class): """remove_artifacts performs runparts on CLEAN_RUNPARTS_DIR""" @@ -229,10 +372,14 @@ return "", "" myargs = namedtuple( - "MyArgs", "remove_logs remove_seed reboot machine_id" + "MyArgs", "remove_logs remove_seed remove_config reboot machine_id" ) cmdargs = myargs( - remove_logs=False, remove_seed=False, reboot=True, machine_id=False + remove_logs=False, + remove_seed=False, + remove_config=[], + reboot=True, + machine_id=False, ) retcode = wrap_and_call( "cloudinit.cmd.clean", @@ -264,11 +411,12 @@ """handle_clean_args removes /etc/machine-id when arg is True.""" uses_systemd.return_value = systemd_val myargs = namedtuple( - "MyArgs", "remove_logs remove_seed reboot machine_id" + "MyArgs", "remove_logs remove_seed remove_config reboot machine_id" ) cmdargs = myargs( remove_logs=False, remove_seed=False, + remove_config=[], reboot=False, machine_id=machine_id, ) @@ -314,6 +462,3 @@ assert ( clean_paths.log.exists() is False ), f"Unexpected log {clean_paths.log}" - - -# vi: ts=4 expandtab syntax=python diff -Nru cloud-init-23.3.3/tests/unittests/cmd/test_cloud_id.py cloud-init-23.4.3/tests/unittests/cmd/test_cloud_id.py --- cloud-init-23.3.3/tests/unittests/cmd/test_cloud_id.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/test_cloud_id.py 2024-02-02 21:21:52.000000000 +0000 @@ -16,32 +16,40 @@ status.UXAppBootStatusCode.UNKNOWN, "DataSourceNoCloud somedetail", [], + {}, "", "nocloud", + {}, ) STATUS_DETAILS_DISABLED = status.StatusDetails( status.UXAppStatus.DISABLED, status.UXAppBootStatusCode.DISABLED_BY_GENERATOR, "DataSourceNoCloud somedetail", [], + {}, "", "", + {}, ) STATUS_DETAILS_NOT_RUN = status.StatusDetails( status.UXAppStatus.NOT_RUN, status.UXAppBootStatusCode.UNKNOWN, "", [], + {}, "", "", + {}, ) STATUS_DETAILS_RUNNING = status.StatusDetails( status.UXAppStatus.RUNNING, status.UXAppBootStatusCode.UNKNOWN, "", [], + {}, "", "", + {}, ) @@ -50,8 +58,10 @@ status.UXAppBootStatusCode.UNKNOWN, "", [], + {}, "", None, + {}, ) @@ -230,6 +240,3 @@ with pytest.raises(SystemExit) as context_manager: cloud_id.main() assert exit_code == context_manager.value.code - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/cmd/test_main.py cloud-init-23.4.3/tests/unittests/cmd/test_main.py --- cloud-init-23.3.3/tests/unittests/cmd/test_main.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/test_main.py 2024-02-02 21:21:52.000000000 +0000 @@ -188,6 +188,3 @@ result = main._should_bring_up_interfaces(init, args) assert result == expected - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/cmd/test_query.py cloud-init-23.4.3/tests/unittests/cmd/test_query.py --- cloud-init-23.3.3/tests/unittests/cmd/test_query.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/test_query.py 2024-02-02 21:21:52.000000000 +0000 @@ -33,7 +33,6 @@ mocker.patch("cloudinit.cmd.query.read_cfg_paths", return_value=Paths({})) -@mock.patch(M_PATH + "addLogHandlerCLI", lambda *args: "") class TestQuery: Args = namedtuple( "Args", @@ -84,10 +83,7 @@ vendor_data=None, varname=None, ) - with mock.patch( - M_PATH + "addLogHandlerCLI", return_value="" - ) as m_cli_log: - assert 1 == query.handle_args("anyname", args) + assert 1 == query.handle_args("anyname", args) expected_error = ( "Expected one of the options: --all, --format, --list-keys" " or varname\n" @@ -95,7 +91,6 @@ assert expected_error in caplog.text out, _err = capsys.readouterr() assert "usage: query" in out - assert 1 == m_cli_log.call_count @pytest.mark.parametrize( "inst_data,varname,expected_error", @@ -131,10 +126,9 @@ paths, _, _, _ = self._setup_paths(tmpdir) with mock.patch(M_PATH + "read_cfg_paths") as m_paths: m_paths.return_value = paths - with mock.patch(M_PATH + "addLogHandlerCLI", return_value=""): - with mock.patch(M_PATH + "load_userdata") as m_lud: - m_lud.return_value = "ud" - assert 1 == query.handle_args("anyname", args) + with mock.patch(M_PATH + "load_userdata") as m_lud: + m_lud.return_value = "ud" + assert 1 == query.handle_args("anyname", args) assert expected_error in caplog.text def test_handle_args_error_on_missing_instance_data(self, caplog, tmpdir): @@ -573,6 +567,3 @@ m_getuid.return_value = 100 assert 1 == query.handle_args("anyname", args) assert expected_error in caplog.text - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/cmd/test_status.py cloud-init-23.4.3/tests/unittests/cmd/test_status.py --- cloud-init-23.3.3/tests/unittests/cmd/test_status.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/cmd/test_status.py 2024-02-02 21:21:52.000000000 +0000 @@ -9,9 +9,13 @@ import pytest +from cloudinit import subp from cloudinit.atomic_helper import write_json from cloudinit.cmd import status -from cloudinit.cmd.status import UXAppStatus, _get_systemd_status +from cloudinit.cmd.status import ( + UXAppStatus, + _get_error_or_running_from_systemd, +) from cloudinit.subp import SubpResult from cloudinit.util import ensure_file from tests.unittests.helpers import wrap_and_call @@ -48,8 +52,7 @@ 'null, "start": 1669231096.9621563}, "modules-config": ' '{"errors": [], "finished": null, "start": null},' '"modules-final": {"errors": [], "finished": null, ' - '"start": null}, "modules-init": {"errors": [], "finished": ' - 'null, "start": null}, "stage": "init-local"} }' + '"start": null}, "stage": "init-local"} }' ), ) @mock.patch(M_PATH + "os.path.exists", return_value=True) @@ -60,7 +63,10 @@ "Cloud-init enabled by systemd cloud-init-generator", ), ) - @mock.patch(f"{M_PATH}_get_systemd_status", return_value=None) + @mock.patch( + f"{M_PATH}_get_error_or_running_from_systemd", + return_value=None, + ) def test_get_status_details_ds_none( self, m_get_systemd_status, @@ -76,8 +82,28 @@ status.UXAppBootStatusCode.ENABLED_BY_GENERATOR, "Running in stage: init-local", [], + {}, "Wed, 23 Nov 2022 19:18:16 +0000", None, # datasource + { + "init": {"errors": [], "finished": None, "start": None}, + "init-local": { + "errors": [], + "finished": None, + "start": 1669231096.9621563, + }, + "modules-config": { + "errors": [], + "finished": None, + "start": None, + }, + "modules-final": { + "errors": [], + "finished": None, + "start": None, + }, + "stage": "init-local", + }, ) == status.get_status_details(paths) @pytest.mark.parametrize( @@ -161,19 +187,25 @@ failure_msg: str, expected_reason: Union[str, Callable], config: Config, + mocker, ): if ensured_file is not None: ensure_file(ensured_file(config)) - (code, reason) = wrap_and_call( - M_NAME, - { - "uses_systemd": uses_systemd, - "get_cmdline": get_cmdline, - }, - status.get_bootstatus, - config.disable_file, - config.paths, - ) + with mock.patch( + f"{M_PATH}subp.subp", + return_value=SubpResult( + """\ +LANG=en_US.UTF-8 +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin +""", + stderr=None, + ), + ): + mocker.patch(f"{M_PATH}uses_systemd", return_value=uses_systemd) + mocker.patch(f"{M_PATH}get_cmdline", return_value=get_cmdline) + code, reason = status.get_bootstatus( + config.disable_file, config.paths, False + ) assert code == expected_bootstatus, failure_msg if isinstance(expected_reason, str): assert reason == expected_reason @@ -235,9 +267,12 @@ expected = dedent( """\ status: disabled + extended_status: disabled boot_status_code: disabled-by-kernel-cmdline detail: disabled for some reason + errors: [] + recoverable_errors: {} """ ) out, _err = capsys.readouterr() @@ -325,11 +360,14 @@ dedent( """\ status: done + extended_status: done boot_status_code: enabled-by-generator last_update: Thu, 01 Jan 1970 00:02:05 +0000 detail: DataSourceNoCloud [seed=/var/.../seed/nocloud-net]\ [dsmode=net] + errors: [] + recoverable_errors: {} """ ), id="returns_done_long", @@ -385,14 +423,18 @@ 1, dedent( """\ - status: error - boot_status_code: enabled-by-kernel-cmdline - last_update: Thu, 01 Jan 1970 00:02:05 +0000 - detail: - error1 - error2 - error3 - """ + status: error + extended_status: error + boot_status_code: enabled-by-kernel-cmdline + last_update: Thu, 01 Jan 1970 00:02:05 +0000 + detail: + DataSourceNoCloud [seed=/var/.../seed/nocloud-net][dsmode=net] + errors: + \t- error1 + \t- error2 + \t- error3 + recoverable_errors: {} + """ ), id="on_errors_long", ), @@ -413,10 +455,13 @@ dedent( """\ status: running + extended_status: running boot_status_code: enabled-by-kernel-cmdline last_update: Thu, 01 Jan 1970 00:02:04 +0000 detail: Running in stage: init + errors: [] + recoverable_errors: {} """ ), id="running_long_format", @@ -437,20 +482,20 @@ dedent( """\ --- - _schema_version: '1' boot_status_code: enabled-by-kernel-cmdline datasource: '' detail: 'Running in stage: init' errors: [] + extended_status: running + init: + finished: null + start: 124.456 + init-local: + finished: 123.46 + start: 123.45 last_update: Thu, 01 Jan 1970 00:02:04 +0000 - schemas: - '1': - boot_status_code: enabled-by-kernel-cmdline - datasource: '' - detail: 'Running in stage: init' - errors: [] - last_update: Thu, 01 Jan 1970 00:02:04 +0000 - status: running + recoverable_errors: {} + stage: init status: running ... @@ -472,30 +517,200 @@ MyArgs(long=False, wait=False, format="json"), 0, { - "_schema_version": "1", "boot_status_code": "enabled-by-kernel-cmdline", "datasource": "", "detail": "Running in stage: init", "errors": [], + "status": "running", + "extended_status": "running", + "init": {"finished": None, "start": 124.456}, + "init-local": {"finished": 123.46, "start": 123.45}, "last_update": "Thu, 01 Jan 1970 00:02:04 +0000", - "schemas": { - "1": { - "boot_status_code": "enabled-by-kernel-cmdline", - "datasource": "", - "detail": "Running in stage: init", + "recoverable_errors": {}, + "stage": "init", + }, + id="running_json_format", + ), + pytest.param( + None, + status.UXAppBootStatusCode.ENABLED_BY_KERNEL_CMDLINE, + { + "v1": { + "stage": None, + "datasource": ( + "DataSourceNoCloud " + "[seed=/var/.../seed/nocloud-net]" + "[dsmode=net]" + ), + "init": { + "errors": ["error1"], + "start": 124.567, + "finished": 125.678, + }, + "init-local": { + "errors": ["error2", "error3"], + "start": 123.45, + "finished": 123.46, + }, + } + }, + None, + MyArgs(long=False, wait=False, format="json"), + 1, + { + "boot_status_code": "enabled-by-kernel-cmdline", + "datasource": "nocloud", + "detail": ( + "DataSourceNoCloud [seed=/var/.../seed/" + "nocloud-net][dsmode=net]" + ), + "errors": ["error1", "error2", "error3"], + "status": "error", + "extended_status": "error", + "init": { + "finished": 125.678, + "start": 124.567, + "errors": ["error1"], + }, + "init-local": { + "finished": 123.46, + "start": 123.45, + "errors": ["error2", "error3"], + }, + "last_update": "Thu, 01 Jan 1970 00:02:05 +0000", + "recoverable_errors": {}, + "stage": None, + }, + id="running_json_format_with_errors", + ), + pytest.param( + lambda config: config.result_file, + status.UXAppBootStatusCode.ENABLED_BY_KERNEL_CMDLINE, + { + "v1": { + "stage": None, + "datasource": ( + "DataSourceNoCloud " + "[seed=/var/.../seed/nocloud-net]" + "[dsmode=net]" + ), + "modules-final": { "errors": [], - "last_update": "Thu, 01 Jan 1970 00:02:04 +0000", - "status": "running", - } + "recoverable_errors": { + "DEPRECATED": [ + ( + "don't try to open the hatch " + "or we'll all be soup" + ) + ] + }, + "start": 127.567, + "finished": 128.678, + }, + "modules-config": { + "errors": [], + "recoverable_errors": { + "CRITICAL": ["Power lost! Prepare to"] + }, + "start": 125.567, + "finished": 126.678, + }, + "init": { + "errors": [], + "recoverable_errors": { + "WARNINGS": [ + "the prime omega transfuser borkeded!" + ] + }, + "start": 124.567, + "finished": 125.678, + }, + "init-local": { + "errors": [], + "recoverable_errors": { + "ERROR": [ + "the ion field reactor just transmutated" + ] + }, + "start": 123.45, + "finished": 123.46, + }, + } + }, + None, + MyArgs(long=False, wait=False, format="json"), + 2, + { + "boot_status_code": "enabled-by-kernel-cmdline", + "datasource": "nocloud", + "detail": ( + "DataSourceNoCloud [seed=/var/.../" + "seed/nocloud-net][dsmode=net]" + ), + "errors": [], + "status": "done", + "extended_status": "degraded done", + "modules-final": { + "errors": [], + "recoverable_errors": { + "DEPRECATED": [ + ( + "don't try to open the " + "hatch or we'll all be soup" + ) + ] + }, + "start": 127.567, + "finished": 128.678, }, - "status": "running", + "modules-config": { + "errors": [], + "recoverable_errors": { + "CRITICAL": ["Power lost! Prepare to"] + }, + "start": 125.567, + "finished": 126.678, + }, + "init": { + "errors": [], + "recoverable_errors": { + "WARNINGS": [ + "the prime omega transfuser borkeded!" + ] + }, + "start": 124.567, + "finished": 125.678, + }, + "init-local": { + "errors": [], + "recoverable_errors": { + "ERROR": [ + "the ion field reactor just transmutated" + ] + }, + "start": 123.45, + "finished": 123.46, + }, + "last_update": "Thu, 01 Jan 1970 00:02:08 +0000", + "recoverable_errors": { + "ERROR": ["the ion field reactor just transmutated"], + "WARNINGS": ["the prime omega transfuser borkeded!"], + "CRITICAL": ["Power lost! Prepare to"], + "DEPRECATED": [ + "don't try to open the hatch or we'll all be soup" + ], + }, + "stage": None, }, - id="running_json_format", + id="running_json_format_with_recoverable_errors", ), ], ) @mock.patch(M_PATH + "read_cfg_paths") - @mock.patch(f"{M_PATH}_get_systemd_status", return_value=None) + @mock.patch( + f"{M_PATH}_get_error_or_running_from_systemd", + return_value=None, + ) def test_status_output( self, m_get_systemd_status, @@ -536,7 +751,10 @@ assert out == expected_status @mock.patch(M_PATH + "read_cfg_paths") - @mock.patch(f"{M_PATH}_get_systemd_status", return_value=None) + @mock.patch( + f"{M_PATH}_get_error_or_running_from_systemd", + return_value=None, + ) def test_status_wait_blocks_until_done( self, m_get_systemd_status, m_read_cfg_paths, config: Config, capsys ): @@ -587,7 +805,10 @@ assert out == "....\nstatus: done\n" @mock.patch(M_PATH + "read_cfg_paths") - @mock.patch(f"{M_PATH}_get_systemd_status", return_value=None) + @mock.patch( + f"{M_PATH}_get_error_or_running_from_systemd", + return_value=None, + ) def test_status_wait_blocks_until_error( self, m_get_systemd_status, m_read_cfg_paths, config: Config, capsys ): @@ -640,7 +861,10 @@ assert out == "....\nstatus: error\n" @mock.patch(M_PATH + "read_cfg_paths") - @mock.patch(f"{M_PATH}_get_systemd_status", return_value=None) + @mock.patch( + f"{M_PATH}_get_error_or_running_from_systemd", + return_value=None, + ) def test_status_main( self, m_get_systemd_status, m_read_cfg_paths, config: Config, capsys ): @@ -664,7 +888,12 @@ assert out == "status: running\n" -class TestSystemdStatusDetails: +class TestGetErrorOrRunningFromSystemd: + @pytest.fixture(autouse=True) + def common_mocks(self, mocker): + mocker.patch("cloudinit.cmd.status.sleep") + yield + @pytest.mark.parametrize( ["active_state", "unit_file_state", "sub_state", "main_pid", "status"], [ @@ -672,7 +901,7 @@ # enabled, enabled-runtime, and static into an "enabled" state # and everything else functionally disabled. # Additionally, SubStates are undocumented and may mean something - # different depending on the ActiveState they are mapped too. + # different depending on the ActiveState they are mapped to. # Because of this I'm only testing SubState combinations seen # in real-world testing (or using "any" string if we dont care). ("activating", "enabled", "start", "123", UXAppStatus.RUNNING), @@ -701,7 +930,7 @@ ("failed", "invalid", "failed", "0", UXAppStatus.ERROR), ], ) - def test_get_systemd_status( + def test_get_error_or_running_from_systemd( self, active_state, unit_file_state, sub_state, main_pid, status ): with mock.patch( @@ -714,4 +943,124 @@ stderr=None, ), ): - assert _get_systemd_status() == status + assert ( + _get_error_or_running_from_systemd(UXAppStatus.RUNNING, False) + == status + ) + + def test_exception_while_running(self, mocker, capsys): + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=subp.ProcessExecutionError( + "Message recipient disconnected from message bus without" + " replying" + ), + ) + assert ( + _get_error_or_running_from_systemd(UXAppStatus.RUNNING, wait=True) + is None + ) + assert 1 == m_subp.call_count + assert "Failed to get status" not in capsys.readouterr().err + + def test_retry(self, mocker, capsys): + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=[ + subp.ProcessExecutionError( + "Message recipient disconnected from message bus without" + " replying" + ), + subp.ProcessExecutionError( + "Message recipient disconnected from message bus without" + " replying" + ), + SubpResult( + "ActiveState=activating\nUnitFileState=enabled\n" + "SubState=start\nMainPID=123\n", + stderr=None, + ), + ], + ) + assert ( + _get_error_or_running_from_systemd(UXAppStatus.ERROR, wait=True) + is UXAppStatus.RUNNING + ) + assert 3 == m_subp.call_count + assert "Failed to get status" not in capsys.readouterr().err + + def test_retry_no_wait(self, mocker, capsys): + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=subp.ProcessExecutionError( + "Message recipient disconnected from message bus without" + " replying" + ), + ) + mocker.patch("time.time", side_effect=[1, 2, 50]) + assert ( + _get_error_or_running_from_systemd(UXAppStatus.ERROR, wait=False) + is None + ) + assert 1 == m_subp.call_count + assert ( + "Failed to get status from systemd. " + "Cloud-init status may be inaccurate." + ) in capsys.readouterr().err + + +class TestQuerySystemctl: + def test_query_systemctl(self, mocker): + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + return_value=SubpResult(stdout="hello", stderr=None), + ) + assert status.query_systemctl(["some", "args"], wait=False) == "hello" + m_subp.assert_called_once_with(["systemctl", "some", "args"]) + + def test_query_systemctl_with_exception(self, mocker, capsys): + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=subp.ProcessExecutionError( + "Message recipient disconnected", stderr="oh noes!" + ), + ) + assert status.query_systemctl(["some", "args"], wait=False) == "" + m_subp.assert_called_once_with(["systemctl", "some", "args"]) + assert "Error from systemctl: oh noes!" in capsys.readouterr().err + + def test_query_systemctl_wait_with_exception(self, mocker): + m_sleep = mocker.patch(f"{M_PATH}sleep") + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=[ + subp.ProcessExecutionError("Message recipient disconnected"), + subp.ProcessExecutionError("Message recipient disconnected"), + subp.ProcessExecutionError("Message recipient disconnected"), + SubpResult(stdout="hello", stderr=None), + ], + ) + + assert status.query_systemctl(["some", "args"], wait=True) == "hello" + assert m_subp.call_count == 4 + assert m_sleep.call_count == 3 + + def test_query_systemctl_wait_with_exception_status(self, mocker): + m_sleep = mocker.patch(f"{M_PATH}sleep") + m_subp = mocker.patch( + f"{M_PATH}subp.subp", + side_effect=subp.ProcessExecutionError( + "Message recipient disconnected" + ), + ) + + assert ( + status.query_systemctl( + ["some", "args"], + wait=True, + existing_status=UXAppStatus.RUNNING, + ) + == "" + ) + assert m_subp.call_count == 1 + assert m_sleep.call_count == 0 diff -Nru cloud-init-23.3.3/tests/unittests/config/test_apt_conf_v1.py cloud-init-23.4.3/tests/unittests/config/test_apt_conf_v1.py --- cloud-init-23.3.3/tests/unittests/config/test_apt_conf_v1.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_apt_conf_v1.py 2024-02-02 21:21:52.000000000 +0000 @@ -132,6 +132,3 @@ self.assertEqual( expected, cc_apt_configure.convert_to_v3_apt_format(mydata) ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_apt_configure_sources_list_v1.py cloud-init-23.4.3/tests/unittests/config/test_apt_configure_sources_list_v1.py --- cloud-init-23.3.3/tests/unittests/config/test_apt_configure_sources_list_v1.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_apt_configure_sources_list_v1.py 2024-02-02 21:21:52.000000000 +0000 @@ -3,73 +3,109 @@ """ test_handler_apt_configure_sources_list Test templating of sources list """ -import os -import shutil -import tempfile -from unittest import mock +import stat -from cloudinit import subp, templater, util +import pytest + +from cloudinit import subp, util from cloudinit.config import cc_apt_configure -from cloudinit.distros.debian import Distro -from tests.unittests import helpers as t_help from tests.unittests.util import get_cloud +EXAMPLE_TMPL = """\ +## template:jinja +## Note, this file is written by cloud-init on first boot of an instance +deb {{mirror}} {{codename}} main restricted +deb-src {{mirror}} {{codename}} main restricted +""" + YAML_TEXT_CUSTOM_SL = """ apt_mirror: http://archive.ubuntu.com/ubuntu/ apt_custom_sources_list: | - ## template:jinja ## Note, this file is written by cloud-init on first boot of an instance - ## modifications made here will not survive a re-bundle. - ## if you wish to make changes you can: - ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg - ## or do the same in user-data - ## b.) add sources in /etc/apt/sources.list.d - ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl - - # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to - # newer versions of the distribution. - deb {{mirror}} {{codename}} main restricted - deb-src {{mirror}} {{codename}} main restricted + deb $MIRROR $RELEASE main restricted + deb-src $MIRROR $RELEASE main restricted # FIND_SOMETHING_SPECIAL """ EXPECTED_CONVERTED_CONTENT = """## Note, this file is written by cloud-init on first boot of an instance -## modifications made here will not survive a re-bundle. -## if you wish to make changes you can: -## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg -## or do the same in user-data -## b.) add sources in /etc/apt/sources.list.d -## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl - -# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to -# newer versions of the distribution. deb http://archive.ubuntu.com/ubuntu/ fakerelease main restricted deb-src http://archive.ubuntu.com/ubuntu/ fakerelease main restricted -# FIND_SOMETHING_SPECIAL """ # noqa: E501 +EXAMPLE_TMPL_DEB822 = """\ +## template:jinja +Types: deb deb-src +URIs: {{mirror}} +Suites: {{codename}} {{codename}}-updates +Components: main restricted + +# Security section +Types: deb deb-src +URIs: {{security}} +Suites: {{codename}}-security +Components: main restricted +""" + +YAML_TEXT_CUSTOM_SL_DEB822 = """ +apt_mirror: http://archive.ubuntu.com/ubuntu/ +apt_custom_sources_list: | + ## template:jinja + Types: deb deb-src + URIs: {{mirror}} + Suites: {{codename}} {{codename}}-updates + Components: main restricted + + # Security section + Types: deb deb-src + URIs: {{security}} + Suites: {{codename}}-security + Components: main restricted + # custom_sources_list +""" + +EXPECTED_CONVERTED_CONTENT_DEB822 = """\ +Types: deb deb-src +URIs: http://archive.ubuntu.com/ubuntu/ +Suites: fakerelease fakerelease-updates +Components: main restricted + +# Security section +Types: deb deb-src +URIs: http://archive.ubuntu.com/ubuntu/ +Suites: fakerelease-security +Components: main restricted +""" -class TestAptSourceConfigSourceList(t_help.FilesystemMockingTestCase): + +@pytest.mark.usefixtures("fake_filesystem") +class TestAptSourceConfigSourceList: """TestAptSourceConfigSourceList Main Class to test sources list rendering """ - def setUp(self): - super(TestAptSourceConfigSourceList, self).setUp() - self.subp = subp.subp - self.new_root = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, self.new_root) - - rpatcher = mock.patch("cloudinit.util.lsb_release") - get_rel = rpatcher.start() - get_rel.return_value = {"codename": "fakerelease"} - self.addCleanup(rpatcher.stop) - apatcher = mock.patch("cloudinit.util.get_dpkg_architecture") - get_arch = apatcher.start() - get_arch.return_value = "amd64" - self.addCleanup(apatcher.stop) + @pytest.fixture(autouse=True) + def common_mocks(self, mocker): + self.subp = mocker.patch.object( + subp, "subp", return_value=("PPID PID", "") + ) + mocker.patch("cloudinit.config.cc_apt_configure._ensure_dependencies") + lsb = mocker.patch("cloudinit.util.lsb_release") + lsb.return_value = {"codename": "fakerelease"} + m_arch = mocker.patch("cloudinit.util.get_dpkg_architecture") + m_arch.return_value = "amd64" + self.deb822 = mocker.patch.object( + cc_apt_configure.features, "APT_DEB822_SOURCE_LIST_FILE", True + ) + mocker.patch.object( + cc_apt_configure, + "get_apt_cfg", + return_value={ + "sourcelist": "/etc/apt/sources.list", + "sourceparts": "/etc/apt/sources.list.d/", + }, + ) - def apt_source_list(self, distro, mirror, mirrorcheck=None): + def apt_source_list(self, distro, mirror, tmpdir, mirrorcheck=None): """apt_source_list Test rendering of a source.list from template for a given distro """ @@ -78,69 +114,53 @@ if isinstance(mirror, list): cfg = {"apt_mirror_search": mirror} + expected = EXPECTED_CONVERTED_CONTENT.replace( + "http://archive.ubuntu.com/ubuntu/", mirror[-1] + ) else: cfg = {"apt_mirror": mirror} + expected = EXPECTED_CONVERTED_CONTENT.replace( + "http://archive.ubuntu.com/ubuntu/", mirror + ) mycloud = get_cloud(distro) + tmpl_file = f"/etc/cloud/templates/sources.list.{distro}.tmpl" + util.write_file(tmpl_file, EXAMPLE_TMPL) - with mock.patch.object(util, "write_file") as mockwf: - with mock.patch.object( - util, "load_file", return_value="faketmpl" - ) as mocklf: - with mock.patch.object( - os.path, "isfile", return_value=True - ) as mockisfile: - with mock.patch.object( - templater, "render_string", return_value="fake" - ) as mockrnd: - with mock.patch.object(util, "rename"): - cc_apt_configure.handle("test", cfg, mycloud, None) - - mockisfile.assert_any_call( - "/etc/cloud/templates/sources.list.%s.tmpl" % distro - ) - mocklf.assert_any_call( - "/etc/cloud/templates/sources.list.%s.tmpl" % distro - ) - mockrnd.assert_called_once_with( - "faketmpl", - { - "RELEASE": "fakerelease", - "PRIMARY": mirrorcheck, - "MIRROR": mirrorcheck, - "SECURITY": mirrorcheck, - "codename": "fakerelease", - "primary": mirrorcheck, - "mirror": mirrorcheck, - "security": mirrorcheck, - }, - ) - mockwf.assert_called_once_with( - "/etc/apt/sources.list", "fake", mode=0o644 - ) - - def test_apt_v1_source_list_debian(self): - """Test rendering of a source.list from template for debian""" - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - self.apt_source_list( - "debian", "http://httpredir.debian.org/debian" + cc_apt_configure.handle("test", cfg, mycloud, None) + sources_file = tmpdir.join("/etc/apt/sources.list") + assert expected == sources_file.read() + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) + + @pytest.mark.parametrize( + "distro,mirror", + ( + ( + "ubuntu", + "http://archive.ubuntu.com/ubuntu/", + ), + ( + "debian", + "http://httpredir.debian.org/debian/", + ), + ), + ) + def test_apt_v1_source_list_by_distro(self, distro, mirror, tmpdir): + """Test rendering of a source.list from template for each distro""" + mycloud = get_cloud(distro) + tmpl_file = f"/etc/cloud/templates/sources.list.{distro}.deb822.tmpl" + util.write_file(tmpl_file, EXAMPLE_TMPL_DEB822) + cc_apt_configure.handle("test", {"apt_mirror": mirror}, mycloud, None) + sources_file = tmpdir.join(f"/etc/apt/sources.list.d/{distro}.sources") + assert ( + EXPECTED_CONVERTED_CONTENT_DEB822.replace( + "http://archive.ubuntu.com/ubuntu/", mirror ) - mocksubp.assert_called_once_with( - ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], - capture=True, - target=None, - rcs=[0, 1], + == sources_file.read() ) + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) - def test_apt_v1_source_list_ubuntu(self): - """Test rendering of a source.list from template for ubuntu""" - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - self.apt_source_list("ubuntu", "http://archive.ubuntu.com/ubuntu/") - mocksubp.assert_called_once_with( + self.subp.assert_called_once_with( ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], capture=True, target=None, @@ -157,80 +177,115 @@ print("Faking SUCCESS for '%s'" % name) return True - def test_apt_v1_srcl_debian_mirrorfail(self): - """Test rendering of a source.list from template for debian""" - with mock.patch.object( + @pytest.mark.parametrize( + "distro,mirrorlist,mirrorcheck", + ( + ( + "ubuntu", + ["http://does.not.exist", "http://archive.ubuntu.com/ubuntu/"], + "http://archive.ubuntu.com/ubuntu/", + ), + ( + "debian", + [ + "http://does.not.exist", + "http://httpredir.debian.org/debian/", + ], + "http://httpredir.debian.org/debian/", + ), + ), + ) + def test_apt_v1_srcl_distro_mirrorfail( + self, distro, mirrorlist, mirrorcheck, mocker, tmpdir + ): + """Test rendering of a source.list from template for each distro""" + mycloud = get_cloud(distro) + tmpl_file = f"/etc/cloud/templates/sources.list.{distro}.deb822.tmpl" + util.write_file(tmpl_file, EXAMPLE_TMPL_DEB822) + + mockresolve = mocker.patch.object( util, "is_resolvable", side_effect=self.myresolve - ) as mockresolve: - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - self.apt_source_list( - "debian", - [ - "http://does.not.exist", - "http://httpredir.debian.org/debian", - ], - "http://httpredir.debian.org/debian", - ) - mockresolve.assert_any_call("http://does.not.exist") - mockresolve.assert_any_call("http://httpredir.debian.org/debian") - mocksubp.assert_called_once_with( - ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], - capture=True, - target=None, - rcs=[0, 1], ) + cc_apt_configure.handle( + "test", {"apt_mirror_search": mirrorlist}, mycloud, None + ) + sources_file = tmpdir.join(f"/etc/apt/sources.list.d/{distro}.sources") + assert ( + EXPECTED_CONVERTED_CONTENT_DEB822.replace( + "http://archive.ubuntu.com/ubuntu/", mirrorcheck + ) + == sources_file.read() + ) + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) - def test_apt_v1_srcl_ubuntu_mirrorfail(self): - """Test rendering of a source.list from template for ubuntu""" - with mock.patch.object( - util, "is_resolvable", side_effect=self.myresolve - ) as mockresolve: - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - self.apt_source_list( - "ubuntu", - [ - "http://does.not.exist", - "http://archive.ubuntu.com/ubuntu/", - ], - "http://archive.ubuntu.com/ubuntu/", - ) mockresolve.assert_any_call("http://does.not.exist") - mockresolve.assert_any_call("http://archive.ubuntu.com/ubuntu/") - mocksubp.assert_called_once_with( + mockresolve.assert_any_call(mirrorcheck) + self.subp.assert_called_once_with( ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], capture=True, target=None, rcs=[0, 1], ) - def test_apt_v1_srcl_custom(self): + @pytest.mark.parametrize( + "deb822,cfg,apt_file,expected", + ( + pytest.param( + True, + util.load_yaml(YAML_TEXT_CUSTOM_SL_DEB822), + "/etc/apt/sources.list.d/ubuntu.sources", + EXPECTED_CONVERTED_CONTENT_DEB822 + "# custom_sources_list\n", + id="deb822_and_deb822_sources_list_writes_deb822_source_file", + ), + pytest.param( + True, + util.load_yaml(YAML_TEXT_CUSTOM_SL), + "/etc/apt/sources.list", + EXPECTED_CONVERTED_CONTENT + "# FIND_SOMETHING_SPECIAL\n", + id="deb822_and_non_deb822_sources_list_writes_apt_list_file", + ), + pytest.param( + False, + util.load_yaml(YAML_TEXT_CUSTOM_SL), + "/etc/apt/sources.list", + EXPECTED_CONVERTED_CONTENT + "# FIND_SOMETHING_SPECIAL\n", + id="nodeb822_and_nondeb822_sources_list_writes_list_file", + ), + pytest.param( + True, + util.load_yaml(YAML_TEXT_CUSTOM_SL_DEB822), + "/etc/apt/sources.list.d/ubuntu.sources", + EXPECTED_CONVERTED_CONTENT_DEB822 + "# custom_sources_list\n", + id="nodeb822_and_deb822_sources_list_writes_sources_file", + ), + ), + ) + def test_apt_v1_srcl_custom( + self, deb822, cfg, apt_file, expected, mocker, tmpdir + ): """Test rendering from a custom source.list template""" - cfg = util.load_yaml(YAML_TEXT_CUSTOM_SL) - mycloud = get_cloud() + self.deb822 = mocker.patch.object( + cc_apt_configure.features, + "APT_DEB822_SOURCE_LIST_FILE", + deb822, + ) + mycloud = get_cloud("ubuntu") + if deb822: + tmpl_file = "/etc/cloud/templates/sources.list.ubuntu.deb822.tmpl" + tmpl_content = EXAMPLE_TMPL_DEB822 + else: + tmpl_content = EXAMPLE_TMPL + tmpl_file = "/etc/cloud/templates/sources.list.ubuntu.tmpl" + util.write_file(tmpl_file, tmpl_content) # the second mock restores the original subp - with mock.patch.object(util, "write_file") as mockwrite: - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - with mock.patch.object( - Distro, "get_primary_arch", return_value="amd64" - ): - cc_apt_configure.handle("notimportant", cfg, mycloud, None) - - mockwrite.assert_called_once_with( - "/etc/apt/sources.list", EXPECTED_CONVERTED_CONTENT, mode=420 - ) - mocksubp.assert_called_once_with( + cc_apt_configure.handle("notimportant", cfg, mycloud, None) + sources_file = tmpdir.join(apt_file) + assert expected == sources_file.read() + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) + self.subp.assert_called_once_with( ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], capture=True, target=None, rcs=[0, 1], ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_apt_configure_sources_list_v3.py cloud-init-23.4.3/tests/unittests/config/test_apt_configure_sources_list_v3.py --- cloud-init-23.3.3/tests/unittests/config/test_apt_configure_sources_list_v3.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_apt_configure_sources_list_v3.py 2024-02-02 21:21:52.000000000 +0000 @@ -3,22 +3,41 @@ """ test_apt_custom_sources_list Test templating of custom sources list """ -import os -import shutil -import tempfile -from contextlib import ExitStack -from unittest import mock -from unittest.mock import call +import stat + +import pytest from cloudinit import subp, util from cloudinit.config import cc_apt_configure from cloudinit.distros.debian import Distro -from tests.unittests import helpers as t_help from tests.unittests.util import get_cloud TARGET = "/" # Input and expected output for the custom template +EXAMPLE_TMPL = """\ +## template:jinja +deb {{mirror}} {{codename}} main restricted +deb-src {{mirror}} {{codename}} main restricted +deb {{mirror}} {{codename}}-updates universe restricted +deb {{security}} {{codename}}-security multiverse +""" + +EXAMPLE_TMPL_DEB822 = """\ +## template:jinja +# Generated by cloud-init +Types: deb deb-src +URIs: {{mirror}} +Suites: {{codename}} {{codename}}-updates +Components: main restricted + +# Security section +Types: deb deb-src +URIs: {{security}} +Suites: {{codename}}-security +Components: main restricted +""" + YAML_TEXT_CUSTOM_SL = """ apt: primary: @@ -29,10 +48,7 @@ uri: http://testsec.ubuntu.com/ubuntu/ sources_list: | - # Note, this file is written by cloud-init at install time. It should not - # end up on the installed system itself. - # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to - # newer versions of the distribution. + # Note, this file is written by cloud-init at install time. deb $MIRROR $RELEASE main restricted deb-src $MIRROR $RELEASE main restricted deb $PRIMARY $RELEASE universe restricted @@ -41,173 +57,207 @@ """ EXPECTED_CONVERTED_CONTENT = """ -# Note, this file is written by cloud-init at install time. It should not -# end up on the installed system itself. -# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to -# newer versions of the distribution. +# Note, this file is written by cloud-init at install time. deb http://test.ubuntu.com/ubuntu/ fakerel main restricted deb-src http://test.ubuntu.com/ubuntu/ fakerel main restricted deb http://test.ubuntu.com/ubuntu/ fakerel universe restricted deb http://testsec.ubuntu.com/ubuntu/ fakerel-security multiverse -# FIND_SOMETHING_SPECIAL """ -# mocked to be independent to the unittest system -MOCKED_APT_SRC_LIST = """ -deb http://test.ubuntu.com/ubuntu/ notouched main restricted -deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted -deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted -deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted -""" +YAML_TEXT_CUSTOM_SL_DEB822 = """ +apt: + primary: + - arches: [default] + uri: http://test.ubuntu.com/ubuntu/ + security: + - arches: [default] + uri: http://testsec.ubuntu.com/ubuntu/ + sources_list: | -EXPECTED_BASE_CONTENT = """ -deb http://test.ubuntu.com/ubuntu/ notouched main restricted -deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted -deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted -deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted + # custom deb822 sources_list deb822 + Types: deb deb-src + URIs: $PRIMARY + Suites: $RELEASE $RELEASE-updates + Components: main restricted + + Types: deb deb-src + URIs: $SECURITY + Suites: $RELEASE-security + Components: main restricted """ -EXPECTED_MIRROR_CONTENT = """ -deb http://test.ubuntu.com/ubuntu/ notouched main restricted -deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted -deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted -deb http://test.ubuntu.com/ubuntu/ notouched-security main restricted +EXPECTED_CONVERTED_CONTENT_DEB822 = """ +# custom deb822 sources_list deb822 +Types: deb deb-src +URIs: http://test.ubuntu.com/ubuntu/ +Suites: fakerel fakerel-updates +Components: main restricted + +Types: deb deb-src +URIs: http://testsec.ubuntu.com/ubuntu/ +Suites: fakerel-security +Components: main restricted """ -EXPECTED_PRIMSEC_CONTENT = """ -deb http://test.ubuntu.com/ubuntu/ notouched main restricted -deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted -deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted -deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted +# mocked to be independent to the unittest system + +EXPECTED_BASE_CONTENT = """\ +deb http://archive.ubuntu.com/ubuntu/ fakerel main restricted +deb-src http://archive.ubuntu.com/ubuntu/ fakerel main restricted +deb http://archive.ubuntu.com/ubuntu/ fakerel-updates universe restricted +deb http://security.ubuntu.com/ubuntu/ fakerel-security multiverse """ +EXPECTED_MIRROR_CONTENT = """\ +deb http://test.ubuntu.com/ubuntu/ fakerel main restricted +deb-src http://test.ubuntu.com/ubuntu/ fakerel main restricted +deb http://test.ubuntu.com/ubuntu/ fakerel-updates main restricted +deb http://test.ubuntu.com/ubuntu/ fakerel-security main restricted +""" -class TestAptSourceConfigSourceList(t_help.FilesystemMockingTestCase): - """TestAptSourceConfigSourceList - Class to test sources list rendering""" +EXPECTED_BASE_CONTENT_DEB822 = """\ +Types: deb deb-src +URIs: http://archive.ubuntu.com/ubuntu/ +Suites: fakerel fakerel-updates +Components: main restricted + +# Security section +Types: deb deb-src +URIs: http://security.ubuntu.com/ubuntu/ +Suites: fakerel-security +Components: main restricted +""" - def setUp(self): - super(TestAptSourceConfigSourceList, self).setUp() - self.new_root = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, self.new_root) - - rpatcher = mock.patch("cloudinit.util.lsb_release") - get_rel = rpatcher.start() - get_rel.return_value = {"codename": "fakerel"} - self.addCleanup(rpatcher.stop) - apatcher = mock.patch("cloudinit.util.get_dpkg_architecture") - get_arch = apatcher.start() - get_arch.return_value = "amd64" - self.addCleanup(apatcher.stop) - - def _apt_source_list(self, distro, cfg, cfg_on_empty=False): - """_apt_source_list - Test rendering from template (generic)""" - # entry at top level now, wrap in 'apt' key - cfg = {"apt": cfg} - mycloud = get_cloud(distro) +EXPECTED_PRIMSEC_CONTENT = """\ +deb http://test.ubuntu.com/ubuntu/ fakerel main restricted +deb-src http://test.ubuntu.com/ubuntu/ fakerel main restricted +deb http://test.ubuntu.com/ubuntu/ fakerel-updates universe restricted +deb http://testsec.ubuntu.com/ubuntu/ fakerel-security multiverse +""" - with ExitStack() as stack: - mock_writefile = stack.enter_context( - mock.patch.object(util, "write_file") - ) - mock_loadfile = stack.enter_context( - mock.patch.object( - util, "load_file", return_value=MOCKED_APT_SRC_LIST - ) - ) - mock_isfile = stack.enter_context( - mock.patch.object(os.path, "isfile", return_value=True) - ) - stack.enter_context(mock.patch.object(util, "del_file")) - cfg_func = ( - "cloudinit.config.cc_apt_configure." - "_should_configure_on_empty_apt" - ) - mock_shouldcfg = stack.enter_context( - mock.patch(cfg_func, return_value=(cfg_on_empty, "test")) - ) - mock_subp = stack.enter_context( - mock.patch.object(subp, "subp", return_value=("PPID PID", "")) - ) - cc_apt_configure.handle("test", cfg, mycloud, None) +EXPECTED_PRIMSEC_CONTENT_DEB822 = """# Generated by cloud-init +Types: deb deb-src +URIs: http://test.ubuntu.com/ubuntu/ +Suites: fakerel fakerel-updates +Components: main restricted + +# Security section +Types: deb deb-src +URIs: http://testsec.ubuntu.com/ubuntu/ +Suites: fakerel-security +Components: main restricted +""" - return ( - mock_writefile, - mock_loadfile, - mock_isfile, - mock_shouldcfg, - mock_subp, - ) - def test_apt_v3_source_list_debian(self): - """test_apt_v3_source_list_debian - without custom sources or parms""" - cfg = {} - distro = "debian" - expected = EXPECTED_BASE_CONTENT +@pytest.mark.usefixtures("fake_filesystem") +class TestAptSourceConfigSourceList: + """TestAptSourceConfigSourceList - Class to test sources list rendering""" - ( - mock_writefile, - mock_load_file, - mock_isfile, - mock_shouldcfg, - _mock_subp, - ) = self._apt_source_list(distro, cfg, cfg_on_empty=True) - - template = "/etc/cloud/templates/sources.list.%s.tmpl" % distro - mock_writefile.assert_called_once_with( - "/etc/apt/sources.list", expected, mode=0o644 + @pytest.fixture(autouse=True) + def common_mocks(self, mocker): + self.subp = mocker.patch.object( + subp, + "subp", + return_value=("PPID PID", ""), + ) + lsb = mocker.patch("cloudinit.util.lsb_release") + lsb.return_value = {"codename": "fakerel"} + m_arch = mocker.patch("cloudinit.util.get_dpkg_architecture") + m_arch.return_value = "amd64" + mocker.patch("cloudinit.config.cc_apt_configure._ensure_dependencies") + self.deb822 = mocker.patch.object( + cc_apt_configure.features, "APT_DEB822_SOURCE_LIST_FILE", True + ) + mocker.patch.object( + cc_apt_configure, + "get_apt_cfg", + return_value={ + "sourcelist": "/etc/apt/sources.list", + "sourceparts": "/etc/apt/sources.list.d/", + }, ) - mock_load_file.assert_called_with(template) - mock_isfile.assert_any_call(template) - self.assertEqual(1, mock_shouldcfg.call_count) - - def test_apt_v3_source_list_ubuntu(self): - """test_apt_v3_source_list_ubuntu - without custom sources or parms""" - cfg = {} - distro = "ubuntu" - expected = EXPECTED_BASE_CONTENT - ( - mock_writefile, - mock_load_file, - mock_isfile, - mock_shouldcfg, - _mock_subp, - ) = self._apt_source_list(distro, cfg, cfg_on_empty=True) - - template = "/etc/cloud/templates/sources.list.%s.tmpl" % distro - mock_writefile.assert_called_once_with( - "/etc/apt/sources.list", expected, mode=0o644 + @pytest.mark.parametrize( + "distro,template_present", + (("ubuntu", True), ("debian", True), ("rhel", False)), + ) + def test_apt_v3_empty_cfg_source_list_by_distro( + self, distro, template_present, mocker, tmpdir + ): + """Template based on distro, empty config relies on mirror default.""" + template = f"/etc/cloud/templates/sources.list.{distro}.deb822.tmpl" + if template_present: + util.write_file(template, EXAMPLE_TMPL_DEB822) + + mycloud = get_cloud(distro) + mock_shouldcfg = mocker.patch.object( + cc_apt_configure, + "_should_configure_on_empty_apt", + return_value=(True, "test"), ) - mock_load_file.assert_called_with(template) - mock_isfile.assert_any_call(template) - self.assertEqual(1, mock_shouldcfg.call_count) + cc_apt_configure.handle("test", {"apt": {}}, mycloud, None) - def test_apt_v3_source_list_ubuntu_snappy(self): + sources_file = tmpdir.join(f"/etc/apt/sources.list.d/{distro}.sources") + if template_present: + assert ( + "# Generated by cloud-init\n" + EXPECTED_BASE_CONTENT_DEB822 + == sources_file.read() + ) + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) + else: + assert ( + sources_file.exists() is False + ), f"Unexpected file found: {sources_file}" + + assert 1 == mock_shouldcfg.call_count + + def test_apt_v3_source_list_ubuntu_snappy(self, mocker): """test_apt_v3_source_list_ubuntu_snappy - without custom sources or parms""" cfg = {"apt": {}} mycloud = get_cloud() - with mock.patch.object(util, "write_file") as mock_writefile: - with mock.patch.object( - util, "system_is_snappy", return_value=True - ) as mock_issnappy: - cc_apt_configure.handle("test", cfg, mycloud, None) - - self.assertEqual(0, mock_writefile.call_count) - self.assertEqual(1, mock_issnappy.call_count) - - def test_apt_v3_source_list_centos(self): - """test_apt_v3_source_list_centos - without custom sources or parms""" - cfg = {} - distro = "rhel" - - mock_writefile, _, _, _, _ = self._apt_source_list(distro, cfg) + mock_writefile = mocker.patch.object(util, "write_file") + mock_issnappy = mocker.patch.object(util, "system_is_snappy") + mock_issnappy.return_value = True + cc_apt_configure.handle("test", cfg, mycloud, None) + mock_writefile.assert_not_called() + assert 1 == mock_issnappy.call_count - self.assertEqual(0, mock_writefile.call_count) - - def test_apt_v3_source_list_psm(self): - """test_apt_v3_source_list_psm - Test specifying prim+sec mirrors""" + @pytest.mark.parametrize( + "deb822,tmpl_file,tmpl_content,apt_file,expected", + ( + ( + True, + "/etc/cloud/templates/sources.list.ubuntu.deb822.tmpl", + EXAMPLE_TMPL_DEB822, + "/etc/apt/sources.list.d/ubuntu.sources", + EXPECTED_PRIMSEC_CONTENT_DEB822, + ), + ( + False, + "/etc/cloud/templates/sources.list.ubuntu.tmpl", + EXAMPLE_TMPL, + "/etc/apt/sources.list", + EXPECTED_PRIMSEC_CONTENT, + ), + ), + ) + def test_apt_v3_source_list_psm_deb822_feature_aware( + self, + deb822, + tmpl_file, + tmpl_content, + apt_file, + expected, + tmpdir, + mocker, + ): + """test_apt_v3_source_list_psm - Test specifying prim+sec mirrors + + Assert APT_DEB822_SOURCE_LIST_FILE is taken into account when + determining which sources.list.tmpl source file to read. + """ pm = "http://test.ubuntu.com/ubuntu/" sm = "http://testsec.ubuntu.com/ubuntu/" cfg = { @@ -215,50 +265,74 @@ "primary": [{"arches": ["default"], "uri": pm}], "security": [{"arches": ["default"], "uri": sm}], } - distro = "ubuntu" - expected = EXPECTED_PRIMSEC_CONTENT - ( - mock_writefile, - mock_load_file, - mock_isfile, - _, - _, - ) = self._apt_source_list(distro, cfg, cfg_on_empty=True) - - template = "/etc/cloud/templates/sources.list.%s.tmpl" % distro - mock_writefile.assert_called_once_with( - "/etc/apt/sources.list", expected, mode=0o644 + self.deb822 = mocker.patch.object( + cc_apt_configure.features, "APT_DEB822_SOURCE_LIST_FILE", deb822 ) - mock_load_file.assert_called_with(template) - mock_isfile.assert_any_call(template) - def test_apt_v3_srcl_custom(self): - """test_apt_v3_srcl_custom - Test rendering a custom source template""" - cfg = util.load_yaml(YAML_TEXT_CUSTOM_SL) - mycloud = get_cloud() + util.write_file(tmpl_file, tmpl_content) + mycloud = get_cloud("ubuntu") + cc_apt_configure.handle("test", {"apt": cfg}, mycloud, None) + + sources_file = tmpdir.join(apt_file) + assert expected == sources_file.read() + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) - with mock.patch.object(util, "write_file") as mockwrite: - with mock.patch.object( - subp, "subp", return_value=("PPID PID", "") - ) as mocksubp: - with mock.patch.object( - Distro, "get_primary_arch", return_value="amd64" - ): - cc_apt_configure.handle("notimportant", cfg, mycloud, None) - - calls = [ - call( - "/etc/apt/sources.list", EXPECTED_CONVERTED_CONTENT, mode=0o644 - ) - ] - mockwrite.assert_has_calls(calls) - mocksubp.assert_called_once_with( + @pytest.mark.parametrize( + "deb822,cfg,apt_file,expected", + ( + pytest.param( + True, + util.load_yaml(YAML_TEXT_CUSTOM_SL_DEB822), + "/etc/apt/sources.list.d/debian.sources", + EXPECTED_CONVERTED_CONTENT_DEB822, + id="deb822_feature_deb822_sources_list_writes_deb822_source", + ), + pytest.param( + True, + util.load_yaml(YAML_TEXT_CUSTOM_SL), + "/etc/apt/sources.list", + EXPECTED_CONVERTED_CONTENT + "# FIND_SOMETHING_SPECIAL\n", + id="deb822_feature_nondeb822_sources_list_writes_list_file", + ), + pytest.param( + False, + util.load_yaml(YAML_TEXT_CUSTOM_SL), + "/etc/apt/sources.list", + EXPECTED_CONVERTED_CONTENT + "# FIND_SOMETHING_SPECIAL\n", + id="nodeb822_feature_nondeb822_sources_list_writes_list_file", + ), + pytest.param( + False, + util.load_yaml(YAML_TEXT_CUSTOM_SL_DEB822), + "/etc/apt/sources.list.d/debian.sources", + EXPECTED_CONVERTED_CONTENT_DEB822, + id="nodeb822_feature_deb822_sources_list_writes_deb822_source", + ), + ), + ) + def test_apt_v3_srcl_custom_deb822_feature_aware( + self, deb822, cfg, apt_file, expected, mocker, tmpdir + ): + """test_apt_v3_srcl_custom - Test rendering a custom source template + + Also take into account deb822 feature flag to assert writing the + appropriate deb822 /etc/apt/sources.list.d/*list or *source file based + on content and deb822 feature flag. + """ + mycloud = get_cloud("debian") + + self.deb822 = mocker.patch.object( + cc_apt_configure.features, "APT_DEB822_SOURCE_LIST_FILE", deb822 + ) + mocker.patch.object(Distro, "get_primary_arch", return_value="amd64") + cc_apt_configure.handle("notimportant", cfg, mycloud, None) + sources_file = tmpdir.join(apt_file) + assert expected == sources_file.read() + assert 0o644 == stat.S_IMODE(sources_file.stat().mode) + self.subp.assert_called_once_with( ["ps", "-o", "ppid,pid", "-C", "dirmngr", "-C", "gpg-agent"], capture=True, target=None, rcs=[0, 1], ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_apt_source_v1.py cloud-init-23.4.3/tests/unittests/config/test_apt_source_v1.py --- cloud-init-23.3.3/tests/unittests/config/test_apt_source_v1.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_apt_source_v1.py 2024-02-02 21:21:52.000000000 +0000 @@ -8,17 +8,19 @@ import os import pathlib import re -import shutil import signal -import tempfile +from functools import partial from unittest import mock from unittest.mock import call +import pytest + from cloudinit import gpg, subp, util from cloudinit.config import cc_apt_configure -from tests.unittests.helpers import TestCase from tests.unittests.util import get_cloud +original_join = os.path.join + EXPECTEDKEY = """-----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1 @@ -44,51 +46,47 @@ return -class TestAptSourceConfig(TestCase): +class TestAptSourceConfig: """TestAptSourceConfig Main Class to test apt_source configs """ release = "fantastic" + matcher = re.compile(ADD_APT_REPO_MATCH).search - def setUp(self): - super(TestAptSourceConfig, self).setUp() - self.tmp = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, self.tmp) - self.aptlistfile = os.path.join(self.tmp, "single-deb.list") - self.aptlistfile2 = os.path.join(self.tmp, "single-deb2.list") - self.aptlistfile3 = os.path.join(self.tmp, "single-deb3.list") - self.join = os.path.join - self.matcher = re.compile(ADD_APT_REPO_MATCH).search - # mock fallback filename into writable tmp dir - self.fallbackfn = os.path.join( - self.tmp, "etc/apt/sources.list.d/", "cloud_config_sources.list" - ) - - self.cloud = get_cloud() - - rpatcher = mock.patch("cloudinit.util.lsb_release") - get_rel = rpatcher.start() - get_rel.return_value = {"codename": self.release} - self.addCleanup(rpatcher.stop) - apatcher = mock.patch("cloudinit.util.get_dpkg_architecture") - get_arch = apatcher.start() - get_arch.return_value = "amd64" - self.addCleanup(apatcher.stop) - subp_patcher = mock.patch.object( - subp, "subp", return_value=("PPID PID", "") + @pytest.fixture + def apt_lists(self, tmpdir): + p1 = os.path.join(tmpdir, "single-deb.list") + p2 = os.path.join(tmpdir, "single-deb2.list") + p3 = os.path.join(tmpdir, "single-deb3.list") + return p1, p2, p3 + + @pytest.fixture + def fallback_path(self, tmpdir): + return os.path.join( + tmpdir, "etc/apt/sources.list.d/", "cloud_config_sources.list" + ) + + @pytest.fixture(autouse=True) + def common_mocks(self, mocker): + mocker.patch( + "cloudinit.util.lsb_release", + return_value={"codename": self.release}, + ) + mocker.patch( + "cloudinit.util.get_dpkg_architecture", return_value="amd64" ) - self.m_subp = subp_patcher.start() - self.addCleanup(subp_patcher.stop) + mocker.patch.object(subp, "subp", return_value=("PPID PID", "")) + mocker.patch("cloudinit.config.cc_apt_configure._ensure_dependencies") def _get_default_params(self): """get_default_params Get the most basic default mrror and release info to be used in tests """ - params = {} - params["RELEASE"] = self.release - params["MIRROR"] = "http://archive.ubuntu.com/ubuntu" - return params + return { + "RELEASE": self.release, + "MIRROR": "http://archive.ubuntu.com/ubuntu", + } def wrapv1conf(self, cfg): params = self._get_default_params() @@ -100,16 +98,16 @@ "apt_sources": cfg, } - def myjoin(self, *args, **kwargs): + def myjoin(self, tmpfile, *args, **kwargs): """myjoin - redir into writable tmpdir""" if ( args[0] == "/etc/apt/sources.list.d/" and args[1] == "cloud_config_sources.list" and len(args) == 2 ): - return self.join(self.tmp, args[0].lstrip("/"), args[1]) + return original_join(tmpfile, args[0].lstrip("/"), args[1]) else: - return self.join(*args, **kwargs) + return original_join(*args, **kwargs) def apt_src_basic(self, filename, cfg): """apt_src_basic @@ -117,26 +115,24 @@ """ cfg = self.wrapv1conf(cfg) - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://archive.ubuntu.com/ubuntu", - "karmic-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://archive.ubuntu.com/ubuntu", + "karmic-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_basic(self): + def test_apt_src_basic(self, apt_lists): """Test deb source string, overwrite mirror and filename""" cfg = { "source": ( @@ -144,14 +140,14 @@ " karmic-backports" " main universe multiverse restricted" ), - "filename": self.aptlistfile, + "filename": apt_lists[0], } - self.apt_src_basic(self.aptlistfile, [cfg]) + self.apt_src_basic(apt_lists[0], [cfg]) - def test_apt_src_basic_dict(self): + def test_apt_src_basic_dict(self, apt_lists): """Test deb source string, overwrite mirror and filename (dict)""" cfg = { - self.aptlistfile: { + apt_lists[0]: { "source": ( "deb http://archive.ubuntu.com/ubuntu" " karmic-backports" @@ -159,47 +155,43 @@ ) } } - self.apt_src_basic(self.aptlistfile, cfg) + self.apt_src_basic(apt_lists[0], cfg) - def apt_src_basic_tri(self, cfg): + def apt_src_basic_tri(self, cfg, apt_lists): """apt_src_basic_tri Test Fix three deb source string, has to overwrite mirror conf in params. Test with filenames provided in config. generic part to check three files with different content """ - self.apt_src_basic(self.aptlistfile, cfg) + self.apt_src_basic(apt_lists[0], cfg) # extra verify on two extra files of this test - contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://archive.ubuntu.com/ubuntu", - "precise-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) - ) - contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://archive.ubuntu.com/ubuntu", - "lucid-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) + contents = util.load_file(apt_lists[1]) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://archive.ubuntu.com/ubuntu", + "precise-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, + ) + contents = util.load_file(apt_lists[2]) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://archive.ubuntu.com/ubuntu", + "lucid-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_basic_tri(self): + def test_apt_src_basic_tri(self, apt_lists): """Test Fix three deb source string with filenames""" cfg1 = { "source": ( @@ -207,7 +199,7 @@ " karmic-backports" " main universe multiverse restricted" ), - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": ( @@ -215,7 +207,7 @@ " precise-backports" " main universe multiverse restricted" ), - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": ( @@ -223,28 +215,28 @@ " lucid-backports" " main universe multiverse restricted" ), - "filename": self.aptlistfile3, + "filename": apt_lists[2], } - self.apt_src_basic_tri([cfg1, cfg2, cfg3]) + self.apt_src_basic_tri([cfg1, cfg2, cfg3], apt_lists) - def test_apt_src_basic_dict_tri(self): + def test_apt_src_basic_dict_tri(self, apt_lists): """Test Fix three deb source string with filenames (dict)""" cfg = { - self.aptlistfile: { + apt_lists[0]: { "source": ( "deb http://archive.ubuntu.com/ubuntu" " karmic-backports" " main universe multiverse restricted" ) }, - self.aptlistfile2: { + apt_lists[1]: { "source": ( "deb http://archive.ubuntu.com/ubuntu" " precise-backports" " main universe multiverse restricted" ) }, - self.aptlistfile3: { + apt_lists[2]: { "source": ( "deb http://archive.ubuntu.com/ubuntu" " lucid-backports" @@ -252,9 +244,9 @@ ) }, } - self.apt_src_basic_tri(cfg) + self.apt_src_basic_tri(cfg, apt_lists) - def test_apt_src_basic_nofn(self): + def test_apt_src_basic_nofn(self, fallback_path, tmpdir): """Test Fix three deb source string without filenames (dict)""" cfg = { "source": ( @@ -263,8 +255,10 @@ " main universe multiverse restricted" ) } - with mock.patch.object(os.path, "join", side_effect=self.myjoin): - self.apt_src_basic(self.fallbackfn, [cfg]) + with mock.patch.object( + os.path, "join", side_effect=partial(self.myjoin, tmpdir) + ): + self.apt_src_basic(fallback_path, [cfg]) def apt_src_replacement(self, filename, cfg): """apt_src_replace @@ -272,126 +266,122 @@ """ cfg = self.wrapv1conf(cfg) params = self._get_default_params() - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "multiverse"), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "multiverse"), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_replace(self): + def test_apt_src_replace(self, apt_lists): """Test Autoreplacement of MIRROR and RELEASE in source specs""" cfg = { "source": "deb $MIRROR $RELEASE multiverse", - "filename": self.aptlistfile, + "filename": apt_lists[0], } - self.apt_src_replacement(self.aptlistfile, [cfg]) + self.apt_src_replacement(apt_lists[0], [cfg]) - def apt_src_replace_tri(self, cfg): + def apt_src_replace_tri(self, cfg, apt_lists): """apt_src_replace_tri Test three autoreplacements of MIRROR and RELEASE in source specs with generic part """ - self.apt_src_replacement(self.aptlistfile, cfg) + self.apt_src_replacement(apt_lists[0], cfg) # extra verify on two extra files of this test params = self._get_default_params() - contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "main"), - contents, - flags=re.IGNORECASE, - ) - ) - contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "universe"), - contents, - flags=re.IGNORECASE, - ) + contents = util.load_file(apt_lists[1]) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "main"), + contents, + flags=re.IGNORECASE, + ) + + contents = util.load_file(apt_lists[2]) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "universe"), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_replace_tri(self): + def test_apt_src_replace_tri(self, apt_lists): """Test triple Autoreplacement of MIRROR and RELEASE in source specs""" cfg1 = { "source": "deb $MIRROR $RELEASE multiverse", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": "deb $MIRROR $RELEASE main", - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": "deb $MIRROR $RELEASE universe", - "filename": self.aptlistfile3, + "filename": apt_lists[2], } - self.apt_src_replace_tri([cfg1, cfg2, cfg3]) + self.apt_src_replace_tri([cfg1, cfg2, cfg3], apt_lists) - def test_apt_src_replace_dict_tri(self): + def test_apt_src_replace_dict_tri(self, apt_lists): """Test triple Autoreplacement in source specs (dict)""" cfg = { - self.aptlistfile: {"source": "deb $MIRROR $RELEASE multiverse"}, + apt_lists[0]: {"source": "deb $MIRROR $RELEASE multiverse"}, "notused": { "source": "deb $MIRROR $RELEASE main", - "filename": self.aptlistfile2, + "filename": apt_lists[1], }, - self.aptlistfile3: {"source": "deb $MIRROR $RELEASE universe"}, + apt_lists[2]: {"source": "deb $MIRROR $RELEASE universe"}, } - self.apt_src_replace_tri(cfg) + self.apt_src_replace_tri(cfg, apt_lists) - def test_apt_src_replace_nofn(self): + def test_apt_src_replace_nofn(self, fallback_path, tmpdir): """Test Autoreplacement of MIRROR and RELEASE in source specs nofile""" cfg = {"source": "deb $MIRROR $RELEASE multiverse"} - with mock.patch.object(os.path, "join", side_effect=self.myjoin): - self.apt_src_replacement(self.fallbackfn, [cfg]) + with mock.patch.object( + os.path, "join", side_effect=partial(self.myjoin, tmpdir) + ): + self.apt_src_replacement(fallback_path, [cfg]) def apt_src_keyid(self, filename, cfg, keynum): """apt_src_keyid Test specification of a source + keyid """ cfg = self.wrapv1conf(cfg) + cloud = get_cloud() with mock.patch.object(cc_apt_configure, "add_apt_key") as mockobj: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, cloud, []) # check if it added the right number of keys calls = [] sources = cfg["apt"]["sources"] for src in sources: print(sources[src]) - calls.append(call(sources[src], None)) + calls.append(call(sources[src], cloud, None)) mockobj.assert_has_calls(calls, any_order=True) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "main", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "main", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_keyid(self): + def test_apt_src_keyid(self, apt_lists): """Test specification of a source + keyid with filename being set""" cfg = { "source": ( @@ -401,11 +391,11 @@ " xenial main" ), "keyid": "03683F77", - "filename": self.aptlistfile, + "filename": apt_lists[0], } - self.apt_src_keyid(self.aptlistfile, [cfg], 1) + self.apt_src_keyid(apt_lists[0], [cfg], 1) - def test_apt_src_keyid_tri(self): + def test_apt_src_keyid_tri(self, apt_lists): """Test 3x specification of a source + keyid with filename being set""" cfg1 = { "source": ( @@ -415,7 +405,7 @@ " xenial main" ), "keyid": "03683F77", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": ( @@ -425,7 +415,7 @@ " xenial universe" ), "keyid": "03683F77", - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": ( @@ -435,40 +425,36 @@ " xenial multiverse" ), "keyid": "03683F77", - "filename": self.aptlistfile3, + "filename": apt_lists[2], } - self.apt_src_keyid(self.aptlistfile, [cfg1, cfg2, cfg3], 3) - contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "universe", - ), - contents, - flags=re.IGNORECASE, - ) - ) - contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "multiverse", - ), - contents, - flags=re.IGNORECASE, - ) + self.apt_src_keyid(apt_lists[0], [cfg1, cfg2, cfg3], 3) + contents = util.load_file(apt_lists[1]) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "universe", + ), + contents, + flags=re.IGNORECASE, + ) + contents = util.load_file(apt_lists[2]) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "multiverse", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_keyid_nofn(self): + def test_apt_src_keyid_nofn(self, fallback_path, tmpdir): """Test specification of a source + keyid without filename being set""" cfg = { "source": ( @@ -479,45 +465,46 @@ ), "keyid": "03683F77", } - with mock.patch.object(os.path, "join", side_effect=self.myjoin): - self.apt_src_keyid(self.fallbackfn, [cfg], 1) + with mock.patch.object( + os.path, "join", side_effect=partial(self.myjoin, tmpdir) + ): + self.apt_src_keyid(fallback_path, [cfg], 1) def apt_src_key(self, filename, cfg): """apt_src_key Test specification of a source + key """ cfg = self.wrapv1conf([cfg]) + cloud = get_cloud() with mock.patch.object(cc_apt_configure, "add_apt_key") as mockobj: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, cloud, []) # check if it added the right amount of keys sources = cfg["apt"]["sources"] calls = [] for src in sources: print(sources[src]) - calls.append(call(sources[src], None)) + calls.append(call(sources[src], cloud, None)) mockobj.assert_has_calls(calls, any_order=True) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "main", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "main", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_src_key(self): + def test_apt_src_key(self, apt_lists): """Test specification of a source + key with filename being set""" cfg = { "source": ( @@ -527,11 +514,11 @@ " xenial main" ), "key": "fakekey 4321", - "filename": self.aptlistfile, + "filename": apt_lists[0], } - self.apt_src_key(self.aptlistfile, cfg) + self.apt_src_key(apt_lists[0], cfg) - def test_apt_src_key_nofn(self): + def test_apt_src_key_nofn(self, fallback_path, tmpdir): """Test specification of a source + key without filename being set""" cfg = { "source": ( @@ -542,20 +529,22 @@ ), "key": "fakekey 4321", } - with mock.patch.object(os.path, "join", side_effect=self.myjoin): - self.apt_src_key(self.fallbackfn, cfg) + with mock.patch.object( + os.path, "join", side_effect=partial(self.myjoin, tmpdir) + ): + self.apt_src_key(fallback_path, cfg) - def test_apt_src_keyonly(self): + def test_apt_src_keyonly(self, apt_lists): """Test specifying key without source""" - cfg = {"key": "fakekey 4242", "filename": self.aptlistfile} + cfg = {"key": "fakekey 4242", "filename": apt_lists[0]} cfg = self.wrapv1conf([cfg]) with mock.patch.object(cc_apt_configure, "apt_key") as mockobj: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) calls = ( call( "add", - output_file=pathlib.Path(self.aptlistfile).stem, + output_file=pathlib.Path(apt_lists[0]).stem, data="fakekey 4242", hardened=False, ), @@ -563,11 +552,11 @@ mockobj.assert_has_calls(calls, any_order=True) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert not os.path.isfile(apt_lists[0]) - def test_apt_src_keyidonly(self): + def test_apt_src_keyidonly(self, apt_lists): """Test specification of a keyid without source""" - cfg = {"keyid": "03683F77", "filename": self.aptlistfile} + cfg = {"keyid": "03683F77", "filename": apt_lists[0]} cfg = self.wrapv1conf([cfg]) SAMPLE_GPG_AGENT_DIRMNGR_PIDS = """\ PPID PID @@ -586,26 +575,27 @@ ): with mock.patch.object(cc_apt_configure, "apt_key") as mockobj: with mock.patch.object(cc_apt_configure.os, "kill") as m_kill: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) calls = ( call( "add", - output_file=pathlib.Path(self.aptlistfile).stem, + output_file=pathlib.Path(apt_lists[0]).stem, data="fakekey 1212", hardened=False, ), ) mockobj.assert_has_calls(calls, any_order=True) - self.assertEqual( - ([call(1057, signal.SIGKILL), call(1095, signal.SIGKILL)]), - m_kill.call_args_list, - ) + assert ( + [call(1057, signal.SIGKILL), call(1095, signal.SIGKILL)] + ) == m_kill.call_args_list # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert not os.path.isfile(apt_lists[0]) - def apt_src_keyid_real(self, cfg, expectedkey, is_hardened=None): + def apt_src_keyid_real( + self, apt_lists, cfg, expectedkey, is_hardened=None + ): """apt_src_keyid_real Test specification of a keyid without source including up to addition of the key (add_apt_key_raw mocked to keep the @@ -619,102 +609,102 @@ with mock.patch.object( gpg, "getkeybyid", return_value=expectedkey ) as mockgetkey: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) if is_hardened is not None: mockkey.assert_called_with( - expectedkey, self.aptlistfile, hardened=is_hardened + expectedkey, apt_lists[0], hardened=is_hardened ) else: - mockkey.assert_called_with(expectedkey, self.aptlistfile) + mockkey.assert_called_with(expectedkey, apt_lists[0]) mockgetkey.assert_called_with(key, keyserver) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert not os.path.isfile(apt_lists[0]) - def test_apt_src_keyid_real(self): + def test_apt_src_keyid_real(self, apt_lists): """test_apt_src_keyid_real - Test keyid including key add""" keyid = "03683F77" - cfg = {"keyid": keyid, "filename": self.aptlistfile} + cfg = {"keyid": keyid, "filename": apt_lists[0]} - self.apt_src_keyid_real(cfg, EXPECTEDKEY, is_hardened=False) + self.apt_src_keyid_real(apt_lists, cfg, EXPECTEDKEY, is_hardened=False) - def test_apt_src_longkeyid_real(self): + def test_apt_src_longkeyid_real(self, apt_lists): """test_apt_src_longkeyid_real - Test long keyid including key add""" keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77" - cfg = {"keyid": keyid, "filename": self.aptlistfile} + cfg = {"keyid": keyid, "filename": apt_lists[0]} - self.apt_src_keyid_real(cfg, EXPECTEDKEY, is_hardened=False) + self.apt_src_keyid_real(apt_lists, cfg, EXPECTEDKEY, is_hardened=False) - def test_apt_src_longkeyid_ks_real(self): + def test_apt_src_longkeyid_ks_real(self, apt_lists): """test_apt_src_longkeyid_ks_real - Test long keyid from other ks""" keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77" cfg = { "keyid": keyid, "keyserver": "keys.gnupg.net", - "filename": self.aptlistfile, + "filename": apt_lists[0], } - self.apt_src_keyid_real(cfg, EXPECTEDKEY, is_hardened=False) + self.apt_src_keyid_real(apt_lists, cfg, EXPECTEDKEY, is_hardened=False) - def test_apt_src_ppa(self): + def test_apt_src_ppa(self, apt_lists, mocker): """Test adding a ppa""" + m_subp = mocker.patch.object( + subp, "subp", return_value=("PPID PID", "") + ) cfg = { "source": "ppa:smoser/cloud-init-test", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg = self.wrapv1conf([cfg]) - cc_apt_configure.handle("test", cfg, self.cloud, None) - self.assertEqual( - self.m_subp.call_args_list, - [ - mock.call( - [ - "add-apt-repository", - "--no-update", - "ppa:smoser/cloud-init-test", - ], - target=None, - ), - mock.call( - [ - "ps", - "-o", - "ppid,pid", - "-C", - "dirmngr", - "-C", - "gpg-agent", - ], - capture=True, - target=None, - rcs=[0, 1], - ), - ], - ) + cc_apt_configure.handle("test", cfg, get_cloud(), []) + assert m_subp.call_args_list == [ + mock.call( + [ + "add-apt-repository", + "--no-update", + "ppa:smoser/cloud-init-test", + ], + target=None, + ), + mock.call( + [ + "ps", + "-o", + "ppid,pid", + "-C", + "dirmngr", + "-C", + "gpg-agent", + ], + capture=True, + target=None, + rcs=[0, 1], + ), + ] # adding ppa should ignore filename (uses add-apt-repository) - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert not os.path.isfile(apt_lists[0]) - def test_apt_src_ppa_tri(self): + def test_apt_src_ppa_tri(self, apt_lists): """Test adding three ppa's""" cfg1 = { "source": "ppa:smoser/cloud-init-test", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": "ppa:smoser/cloud-init-test2", - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": "ppa:smoser/cloud-init-test3", - "filename": self.aptlistfile3, + "filename": apt_lists[2], } cfg = self.wrapv1conf([cfg1, cfg2, cfg3]) with mock.patch.object( subp, "subp", return_value=("PPID PID", "") ) as mockobj: - cc_apt_configure.handle("test", cfg, self.cloud, None) + cc_apt_configure.handle("test", cfg, get_cloud(), []) calls = [ call( [ @@ -744,51 +734,51 @@ mockobj.assert_has_calls(calls, any_order=True) # adding ppa should ignore all filenames (uses add-apt-repository) - self.assertFalse(os.path.isfile(self.aptlistfile)) - self.assertFalse(os.path.isfile(self.aptlistfile2)) - self.assertFalse(os.path.isfile(self.aptlistfile3)) + assert not os.path.isfile(apt_lists[0]) + assert not os.path.isfile(apt_lists[1]) + assert not os.path.isfile(apt_lists[2]) - def test_convert_to_new_format(self): + def test_convert_to_new_format(self, apt_lists): """Test the conversion of old to new format""" cfg1 = { "source": "deb $MIRROR $RELEASE multiverse", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": "deb $MIRROR $RELEASE main", - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": "deb $MIRROR $RELEASE universe", - "filename": self.aptlistfile3, + "filename": apt_lists[2], } cfg = {"apt_sources": [cfg1, cfg2, cfg3]} checkcfg = { - self.aptlistfile: { - "filename": self.aptlistfile, + apt_lists[0]: { + "filename": apt_lists[0], "source": "deb $MIRROR $RELEASE multiverse", }, - self.aptlistfile2: { - "filename": self.aptlistfile2, + apt_lists[1]: { + "filename": apt_lists[1], "source": "deb $MIRROR $RELEASE main", }, - self.aptlistfile3: { - "filename": self.aptlistfile3, + apt_lists[2]: { + "filename": apt_lists[2], "source": "deb $MIRROR $RELEASE universe", }, } newcfg = cc_apt_configure.convert_to_v3_apt_format(cfg) - self.assertEqual(newcfg["apt"]["sources"], checkcfg) + assert newcfg["apt"]["sources"] == checkcfg # convert again, should stay the same newcfg2 = cc_apt_configure.convert_to_v3_apt_format(newcfg) - self.assertEqual(newcfg2["apt"]["sources"], checkcfg) + assert newcfg2["apt"]["sources"] == checkcfg # should work without raising an exception cc_apt_configure.convert_to_v3_apt_format({}) - with self.assertRaises(ValueError): + with pytest.raises(ValueError): cc_apt_configure.convert_to_v3_apt_format({"apt_sources": 5}) def test_convert_to_new_format_collision(self): @@ -806,39 +796,39 @@ # collision (equal) newcfg = cc_apt_configure.convert_to_v3_apt_format(cfg_1_and_3) - self.assertEqual(newcfg, cfg_3_only) + assert newcfg == cfg_3_only # collision (equal, so ok to remove) newcfg = cc_apt_configure.convert_to_v3_apt_format(cfg_3_only) - self.assertEqual(newcfg, cfg_3_only) + assert newcfg == cfg_3_only # collision (unequal) match = "Old and New.*unequal.*apt_proxy" - with self.assertRaisesRegex(ValueError, match): + with pytest.raises(ValueError, match=match): cc_apt_configure.convert_to_v3_apt_format(cfgconflict) - def test_convert_to_new_format_dict_collision(self): + def test_convert_to_new_format_dict_collision(self, apt_lists): cfg1 = { "source": "deb $MIRROR $RELEASE multiverse", - "filename": self.aptlistfile, + "filename": apt_lists[0], } cfg2 = { "source": "deb $MIRROR $RELEASE main", - "filename": self.aptlistfile2, + "filename": apt_lists[1], } cfg3 = { "source": "deb $MIRROR $RELEASE universe", - "filename": self.aptlistfile3, + "filename": apt_lists[2], } fullv3 = { - self.aptlistfile: { - "filename": self.aptlistfile, + apt_lists[0]: { + "filename": apt_lists[0], "source": "deb $MIRROR $RELEASE multiverse", }, - self.aptlistfile2: { - "filename": self.aptlistfile2, + apt_lists[1]: { + "filename": apt_lists[1], "source": "deb $MIRROR $RELEASE main", }, - self.aptlistfile3: { - "filename": self.aptlistfile3, + apt_lists[2]: { + "filename": apt_lists[2], "source": "deb $MIRROR $RELEASE universe", }, } @@ -848,22 +838,22 @@ # collision (equal, so ok to remove) newcfg = cc_apt_configure.convert_to_v3_apt_format(cfg_1_and_3) - self.assertEqual(newcfg, cfg_3_only) + assert newcfg == cfg_3_only # no old spec (same result) newcfg = cc_apt_configure.convert_to_v3_apt_format(cfg_3_only) - self.assertEqual(newcfg, cfg_3_only) + assert newcfg == cfg_3_only diff = { - self.aptlistfile: { - "filename": self.aptlistfile, + apt_lists[0]: { + "filename": apt_lists[0], "source": "deb $MIRROR $RELEASE DIFFERENTVERSE", }, - self.aptlistfile2: { - "filename": self.aptlistfile2, + apt_lists[1]: { + "filename": apt_lists[1], "source": "deb $MIRROR $RELEASE main", }, - self.aptlistfile3: { - "filename": self.aptlistfile3, + apt_lists[2]: { + "filename": apt_lists[2], "source": "deb $MIRROR $RELEASE universe", }, } @@ -872,12 +862,12 @@ cfg_1_and_3_different.update(cfg_3_only) # collision (unequal by dict having a different entry) - with self.assertRaises(ValueError): + with pytest.raises(ValueError): cc_apt_configure.convert_to_v3_apt_format(cfg_1_and_3_different) missing = { - self.aptlistfile: { - "filename": self.aptlistfile, + apt_lists[0]: { + "filename": apt_lists[0], "source": "deb $MIRROR $RELEASE multiverse", } } @@ -885,8 +875,5 @@ cfg_1_and_3_missing = {"apt_sources": [cfg1, cfg2, cfg3]} cfg_1_and_3_missing.update(cfg_3_only) # collision (unequal by dict missing an entry) - with self.assertRaises(ValueError): + with pytest.raises(ValueError): cc_apt_configure.convert_to_v3_apt_format(cfg_1_and_3_missing) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_apt_source_v3.py cloud-init-23.4.3/tests/unittests/config/test_apt_source_v3.py --- cloud-init-23.3.3/tests/unittests/config/test_apt_source_v3.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_apt_source_v3.py 2024-02-02 21:21:52.000000000 +0000 @@ -5,20 +5,19 @@ This tries to call all in the new v3 format and cares about new features """ import glob +import logging import os import pathlib import re -import shutil import socket -import tempfile -from unittest import TestCase, mock +from unittest import mock from unittest.mock import call import pytest from cloudinit import gpg, subp, util from cloudinit.config import cc_apt_configure -from tests.unittests import helpers as t_help +from tests.unittests.helpers import skipIfAptPkg from tests.unittests.util import get_cloud EXPECTEDKEY = """-----BEGIN PGP PUBLIC KEY BLOCK----- @@ -37,8 +36,6 @@ ADD_APT_REPO_MATCH = r"^[\w-]+:\w" -TARGET = None - MOCK_LSB_RELEASE_DATA = { "id": "Ubuntu", "description": "Ubuntu 18.04.1 LTS", @@ -46,28 +43,25 @@ "codename": "bionic", } +M_PATH = "cloudinit.config.cc_apt_configure." + -class TestAptSourceConfig(t_help.FilesystemMockingTestCase): +class TestAptSourceConfig: """TestAptSourceConfig Main Class to test apt configs """ - def setUp(self): - super(TestAptSourceConfig, self).setUp() - self.tmp = tempfile.mkdtemp() - self.new_root = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, self.tmp) - self.addCleanup(shutil.rmtree, self.new_root) - self.aptlistfile = os.path.join(self.tmp, "single-deb.list") - self.aptlistfile2 = os.path.join(self.tmp, "single-deb2.list") - self.aptlistfile3 = os.path.join(self.tmp, "single-deb3.list") - self.join = os.path.join - self.matcher = re.compile(ADD_APT_REPO_MATCH).search - self.add_patch( - "cloudinit.config.cc_apt_configure.util.lsb_release", - "m_lsb_release", + @pytest.fixture(autouse=True) + def setup(self, mocker, tmpdir): + mocker.patch( + f"{M_PATH}util.lsb_release", return_value=MOCK_LSB_RELEASE_DATA.copy(), ) + mocker.patch(f"{M_PATH}_ensure_dependencies") + self.aptlistfile = tmpdir.join("src1.list").strpath + self.aptlistfile2 = tmpdir.join("src2.list").strpath + self.aptlistfile3 = tmpdir.join("src3.list").strpath + self.matcher = re.compile(ADD_APT_REPO_MATCH).search @staticmethod def _add_apt_sources(*args, **kwargs): @@ -87,45 +81,38 @@ ] return params - def _myjoin(self, *args, **kwargs): - """_myjoin - redir into writable tmpdir""" - if ( - args[0] == "/etc/apt/sources.list.d/" - and args[1] == "cloud_config_sources.list" - and len(args) == 2 - ): - return self.join(self.tmp, args[0].lstrip("/"), args[1]) - else: - return self.join(*args, **kwargs) - - def _apt_src_basic(self, filename, cfg): + def _apt_src_basic(self, filename, cfg, tmpdir): """_apt_src_basic Test Fix deb source string, has to overwrite mirror conf in params """ params = self._get_default_params() self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, ) - self.assertTrue(os.path.isfile(filename)) + assert ( + os.path.isfile(filename) is True + ), f"Missing expected file {filename}" contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://test.ubuntu.com/ubuntu", - "karmic-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://test.ubuntu.com/ubuntu", + "karmic-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, + ), f"Unexpected APT config in {filename}: {contents}" - def test_apt_v3_src_basic(self): + def test_apt_v3_src_basic(self, tmpdir): """test_apt_v3_src_basic - Test fix deb source string""" cfg = { self.aptlistfile: { @@ -136,9 +123,9 @@ ) } } - self._apt_src_basic(self.aptlistfile, cfg) + self._apt_src_basic(self.aptlistfile, cfg, tmpdir) - def test_apt_v3_src_basic_tri(self): + def test_apt_v3_src_basic_tri(self, tmpdir): """test_apt_v3_src_basic_tri - Test multiple fix deb source strings""" cfg = { self.aptlistfile: { @@ -163,116 +150,110 @@ ) }, } - self._apt_src_basic(self.aptlistfile, cfg) + self._apt_src_basic(self.aptlistfile, cfg, tmpdir) # extra verify on two extra files of this test contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://test.ubuntu.com/ubuntu", - "precise-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://test.ubuntu.com/ubuntu", + "precise-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, + ), f"Unexpected APT format of {self.aptlistfile2}: contents" contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://test.ubuntu.com/ubuntu", - "lucid-backports", - "main universe multiverse restricted", - ), - contents, - flags=re.IGNORECASE, - ) - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://test.ubuntu.com/ubuntu", + "lucid-backports", + "main universe multiverse restricted", + ), + contents, + flags=re.IGNORECASE, + ), f"Unexpected APT format of {self.aptlistfile3}: contents" - def _apt_src_replacement(self, filename, cfg): + def _apt_src_replacement(self, filename, cfg, tmpdir): """apt_src_replace Test Autoreplacement of MIRROR and RELEASE in source specs """ params = self._get_default_params() self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, ) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) is True, f"Unexpected file {filename}" contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "multiverse"), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "multiverse"), + contents, + flags=re.IGNORECASE, ) - def test_apt_v3_src_replace(self): + def test_apt_v3_src_replace(self, tmpdir): """test_apt_v3_src_replace - Test replacement of MIRROR & RELEASE""" cfg = {self.aptlistfile: {"source": "deb $MIRROR $RELEASE multiverse"}} - self._apt_src_replacement(self.aptlistfile, cfg) + self._apt_src_replacement(self.aptlistfile, cfg, tmpdir) - def test_apt_v3_src_replace_fn(self): + def test_apt_v3_src_replace_fn(self, tmpdir): """test_apt_v3_src_replace_fn - Test filename overwritten in dict""" cfg = { "ignored": { "source": "deb $MIRROR $RELEASE multiverse", - "filename": self.aptlistfile, + "filename": self.aptlistfile.replace(tmpdir.strpath, ""), } } # second file should overwrite the dict key - self._apt_src_replacement(self.aptlistfile, cfg) + self._apt_src_replacement(self.aptlistfile, cfg, tmpdir) - def _apt_src_replace_tri(self, cfg): + def _apt_src_replace_tri(self, cfg, tmpdir): """_apt_src_replace_tri Test three autoreplacements of MIRROR and RELEASE in source specs with generic part """ - self._apt_src_replacement(self.aptlistfile, cfg) + self._apt_src_replacement(self.aptlistfile, cfg, tmpdir) # extra verify on two extra files of this test params = self._get_default_params() contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "main"), - contents, - flags=re.IGNORECASE, - ) - ) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "main"), + contents, + flags=re.IGNORECASE, + ), f"Unexpected APT format {self.aptlistfile2}: {contents}" contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ("deb", params["MIRROR"], params["RELEASE"], "universe"), - contents, - flags=re.IGNORECASE, - ) - ) + assert re.search( + r"%s %s %s %s\n" + % ("deb", params["MIRROR"], params["RELEASE"], "universe"), + contents, + flags=re.IGNORECASE, + ), f"Unexpected APT format {self.aptlistfile3}: {contents}" - def test_apt_v3_src_replace_tri(self): + def test_apt_v3_src_replace_tri(self, tmpdir): """test_apt_v3_src_replace_tri - Test multiple replace/overwrites""" cfg = { self.aptlistfile: {"source": "deb $MIRROR $RELEASE multiverse"}, "notused": { "source": "deb $MIRROR $RELEASE main", - "filename": self.aptlistfile2, + "filename": self.aptlistfile2.replace(tmpdir.strpath, ""), }, self.aptlistfile3: {"source": "deb $MIRROR $RELEASE universe"}, } - self._apt_src_replace_tri(cfg) + self._apt_src_replace_tri(cfg, tmpdir) - def _apt_src_keyid(self, filename, cfg, keynum, is_hardened=None): + def _apt_src_keyid(self, filename, cfg, keynum, tmpdir, is_hardened=None): """_apt_src_keyid Test specification of a source + keyid """ @@ -280,37 +261,39 @@ with mock.patch.object(cc_apt_configure, "add_apt_key") as mockobj: self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, ) # check if it added the right number of keys calls = [] for key in cfg: if is_hardened is not None: - calls.append(call(cfg[key], hardened=is_hardened)) + calls.append(call(cfg[key], None, hardened=is_hardened)) else: - calls.append(call(cfg[key], TARGET)) + calls.append(call(cfg[key], None, tmpdir.strpath)) mockobj.assert_has_calls(calls, any_order=True) - self.assertTrue(os.path.isfile(filename)) + assert os.path.isfile(filename) is True contents = util.load_file(filename) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "main", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "main", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_v3_src_keyid(self): + def test_apt_v3_src_keyid(self, tmpdir): """test_apt_v3_src_keyid - Test source + keyid with filename""" cfg = { self.aptlistfile: { @@ -320,14 +303,14 @@ "smoser/cloud-init-test/ubuntu" " xenial main" ), - "filename": self.aptlistfile, + "filename": self.aptlistfile.replace(tmpdir.strpath, ""), "keyid": "03683F77", } } - self._apt_src_keyid(self.aptlistfile, cfg, 1) + self._apt_src_keyid(self.aptlistfile, cfg, 1, tmpdir) - def test_apt_v3_src_keyid_tri(self): - """test_apt_v3_src_keyid_tri - Test multiple src+key+filen writes""" + def test_apt_v3_src_keyid_tri(self, tmpdir): + """test_apt_v3_src_keyid_tri - Test multiple src+key+file writes""" cfg = { self.aptlistfile: { "source": ( @@ -346,7 +329,7 @@ " xenial universe" ), "keyid": "03683F77", - "filename": self.aptlistfile2, + "filename": self.aptlistfile2.replace(tmpdir.strpath, ""), }, self.aptlistfile3: { "source": ( @@ -355,42 +338,38 @@ "smoser/cloud-init-test/ubuntu" " xenial multiverse" ), - "filename": self.aptlistfile3, + "filename": self.aptlistfile3.replace(tmpdir.strpath, ""), "keyid": "03683F77", }, } - self._apt_src_keyid(self.aptlistfile, cfg, 3) + self._apt_src_keyid(self.aptlistfile, cfg, 3, tmpdir) contents = util.load_file(self.aptlistfile2) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "universe", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "universe", + ), + contents, + flags=re.IGNORECASE, ) contents = util.load_file(self.aptlistfile3) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "multiverse", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "multiverse", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_v3_src_key(self): + def test_apt_v3_src_key(self, mocker, tmpdir): """test_apt_v3_src_key - Test source + key""" params = self._get_default_params() cfg = { @@ -401,15 +380,18 @@ "smoser/cloud-init-test/ubuntu" " xenial main" ), - "filename": self.aptlistfile, + "filename": self.aptlistfile.replace(tmpdir.strpath, ""), "key": "fakekey 4321", } } - - with mock.patch.object(cc_apt_configure, "apt_key") as mockobj: - self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher - ) + mockobj = mocker.patch.object(cc_apt_configure, "apt_key") + self._add_apt_sources( + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, + ) calls = ( call( @@ -420,32 +402,32 @@ ), ) mockobj.assert_has_calls(calls, any_order=True) - self.assertTrue(os.path.isfile(self.aptlistfile)) - contents = util.load_file(self.aptlistfile) - self.assertTrue( - re.search( - r"%s %s %s %s\n" - % ( - "deb", - "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", - "xenial", - "main", - ), - contents, - flags=re.IGNORECASE, - ) + assert re.search( + r"%s %s %s %s\n" + % ( + "deb", + "http://ppa.launchpad.net/smoser/cloud-init-test/ubuntu", + "xenial", + "main", + ), + contents, + flags=re.IGNORECASE, ) - def test_apt_v3_src_keyonly(self): + def test_apt_v3_src_keyonly(self, tmpdir, mocker): """test_apt_v3_src_keyonly - Test key without source""" params = self._get_default_params() cfg = {self.aptlistfile: {"key": "fakekey 4242"}} - with mock.patch.object(cc_apt_configure, "apt_key") as mockobj: - self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher - ) + mockobj = mocker.patch.object(cc_apt_configure, "apt_key") + self._add_apt_sources( + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, + ) calls = ( call( @@ -458,9 +440,9 @@ mockobj.assert_has_calls(calls, any_order=True) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert os.path.isfile(self.aptlistfile) is False - def test_apt_v3_src_keyidonly(self): + def test_apt_v3_src_keyidonly(self, tmpdir): """test_apt_v3_src_keyidonly - Test keyid without source""" params = self._get_default_params() cfg = {self.aptlistfile: {"keyid": "03683F77"}} @@ -470,7 +452,8 @@ with mock.patch.object(cc_apt_configure, "apt_key") as mockobj: self._add_apt_sources( cfg, - TARGET, + cloud=None, + target=tmpdir.strpath, template_params=params, aa_repo_match=self.matcher, ) @@ -486,9 +469,11 @@ mockobj.assert_has_calls(calls, any_order=True) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert ( + os.path.isfile(self.aptlistfile) is False + ), f"Unexpected file {self.aptlistfile} found" - def apt_src_keyid_real(self, cfg, expectedkey, is_hardened=None): + def apt_src_keyid_real(self, cfg, expectedkey, tmpdir, is_hardened=None): """apt_src_keyid_real Test specification of a keyid without source including up to addition of the key (add_apt_key_raw mocked to keep the @@ -502,7 +487,8 @@ ) as mockgetkey: self._add_apt_sources( cfg, - TARGET, + cloud=None, + target=tmpdir.strpath, template_params=params, aa_repo_match=self.matcher, ) @@ -513,27 +499,29 @@ ) if is_hardened is not None: mockkey.assert_called_with( - expectedkey, keycfg["keyfile"], hardened=is_hardened + expectedkey, + keycfg["keyfile"].replace(tmpdir.strpath, ""), + hardened=is_hardened, ) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert os.path.isfile(self.aptlistfile) is False - def test_apt_v3_src_keyid_real(self): + def test_apt_v3_src_keyid_real(self, tmpdir): """test_apt_v3_src_keyid_real - Test keyid including key add""" keyid = "03683F77" cfg = {self.aptlistfile: {"keyid": keyid, "keyfile": self.aptlistfile}} - self.apt_src_keyid_real(cfg, EXPECTEDKEY, is_hardened=False) + self.apt_src_keyid_real(cfg, EXPECTEDKEY, tmpdir, is_hardened=False) - def test_apt_v3_src_longkeyid_real(self): + def test_apt_v3_src_longkeyid_real(self, tmpdir): """test_apt_v3_src_longkeyid_real Test long keyid including key add""" keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77" cfg = {self.aptlistfile: {"keyid": keyid, "keyfile": self.aptlistfile}} - self.apt_src_keyid_real(cfg, EXPECTEDKEY, is_hardened=False) + self.apt_src_keyid_real(cfg, EXPECTEDKEY, tmpdir, is_hardened=False) - def test_apt_v3_src_longkeyid_ks_real(self): + def test_apt_v3_src_longkeyid_ks_real(self, tmpdir): """test_apt_v3_src_longkeyid_ks_real Test long keyid from other ks""" keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77" cfg = { @@ -544,9 +532,9 @@ } } - self.apt_src_keyid_real(cfg, EXPECTEDKEY) + self.apt_src_keyid_real(cfg, EXPECTEDKEY, tmpdir) - def test_apt_v3_src_keyid_keyserver(self): + def test_apt_v3_src_keyid_keyserver(self, tmpdir): """test_apt_v3_src_keyid_keyserver - Test custom keyserver""" keyid = "03683F77" params = self._get_default_params() @@ -568,25 +556,34 @@ ) as mockadd: self._add_apt_sources( cfg, - TARGET, + cloud=None, + target=tmpdir.strpath, template_params=params, aa_repo_match=self.matcher, ) mockgetkey.assert_called_with("03683F77", "test.random.com") - mockadd.assert_called_with("fakekey", self.aptlistfile, hardened=False) + mockadd.assert_called_with( + "fakekey", + self.aptlistfile.replace(tmpdir.strpath, ""), + hardened=False, + ) # filename should be ignored on key only - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert os.path.isfile(self.aptlistfile) is False - def test_apt_v3_src_ppa(self): + def test_apt_v3_src_ppa(self, tmpdir): """test_apt_v3_src_ppa - Test specification of a ppa""" params = self._get_default_params() cfg = {self.aptlistfile: {"source": "ppa:smoser/cloud-init-test"}} with mock.patch("cloudinit.subp.subp") as mockobj: self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, ) mockobj.assert_any_call( [ @@ -594,13 +591,15 @@ "--no-update", "ppa:smoser/cloud-init-test", ], - target=TARGET, + target=tmpdir.strpath, ) # adding ppa should ignore filename (uses add-apt-repository) - self.assertFalse(os.path.isfile(self.aptlistfile)) + assert ( + os.path.isfile(self.aptlistfile) is False + ), f"Unexpected file found {self.aptlistfile}" - def test_apt_v3_src_ppa_tri(self): + def test_apt_v3_src_ppa_tri(self, tmpdir): """test_apt_v3_src_ppa_tri - Test specification of multiple ppa's""" params = self._get_default_params() cfg = { @@ -611,7 +610,11 @@ with mock.patch("cloudinit.subp.subp") as mockobj: self._add_apt_sources( - cfg, TARGET, template_params=params, aa_repo_match=self.matcher + cfg, + cloud=None, + target=tmpdir.strpath, + template_params=params, + aa_repo_match=self.matcher, ) calls = [ call( @@ -620,7 +623,7 @@ "--no-update", "ppa:smoser/cloud-init-test", ], - target=TARGET, + target=tmpdir.strpath, ), call( [ @@ -628,7 +631,7 @@ "--no-update", "ppa:smoser/cloud-init-test2", ], - target=TARGET, + target=tmpdir.strpath, ), call( [ @@ -636,20 +639,21 @@ "--no-update", "ppa:smoser/cloud-init-test3", ], - target=TARGET, + target=tmpdir.strpath, ), ] mockobj.assert_has_calls(calls, any_order=True) # adding ppa should ignore all filenames (uses add-apt-repository) - self.assertFalse(os.path.isfile(self.aptlistfile)) - self.assertFalse(os.path.isfile(self.aptlistfile2)) - self.assertFalse(os.path.isfile(self.aptlistfile3)) + for path in [self.aptlistfile, self.aptlistfile2, self.aptlistfile3]: + assert not os.path.isfile( + self.aptlistfile + ), f"Unexpected file {path}" @mock.patch("cloudinit.config.cc_apt_configure.util.get_dpkg_architecture") - def test_apt_v3_list_rename(self, m_get_dpkg_architecture): + def test_apt_v3_list_rename(self, m_get_dpkg_architecture, tmpdir): """test_apt_v3_list_rename - Test find mirror and apt list renaming""" - pre = "/var/lib/apt/lists" + pre = tmpdir.join("/var/lib/apt/lists") # filenames are archive dependent arch = "s390x" @@ -680,26 +684,27 @@ mirrors = cc_apt_configure.find_apt_mirror_info(cfg, get_cloud(), arch) - self.assertEqual( - mirrors["MIRROR"], "http://test.ubuntu.com/%s/" % component - ) - self.assertEqual( - mirrors["PRIMARY"], "http://test.ubuntu.com/%s/" % component - ) - self.assertEqual( - mirrors["SECURITY"], "http://testsec.ubuntu.com/%s/" % component + assert mirrors["MIRROR"] == "http://test.ubuntu.com/%s/" % component + assert mirrors["PRIMARY"] == "http://test.ubuntu.com/%s/" % component + assert ( + mirrors["SECURITY"] == "http://testsec.ubuntu.com/%s/" % component ) with mock.patch.object(os, "rename") as mockren: with mock.patch.object(glob, "glob", return_value=[fromfn]): - cc_apt_configure.rename_apt_lists(mirrors, TARGET, arch) + cc_apt_configure.rename_apt_lists( + mirrors, tmpdir.strpath, arch + ) mockren.assert_any_call(fromfn, tofn) @mock.patch("cloudinit.config.cc_apt_configure.util.get_dpkg_architecture") - def test_apt_v3_list_rename_non_slash(self, m_get_dpkg_architecture): - target = os.path.join(self.tmp, "rename_non_slash") - apt_lists_d = os.path.join(target, "./" + cc_apt_configure.APT_LISTS) + def test_apt_v3_list_rename_non_slash( + self, m_get_dpkg_architecture, tmpdir + ): + target = tmpdir.join("rename_non_slash") + + apt_lists_d = target.join(cc_apt_configure.APT_LISTS).strpath arch = "amd64" m_get_dpkg_architecture.return_value = arch @@ -734,13 +739,12 @@ expected = sorted([npre + suff for opre, npre, suff in files]) # create files - for (opre, _npre, suff) in files: + for opre, _npre, suff in files: fpath = os.path.join(apt_lists_d, opre + suff) util.write_file(fpath, content=fpath) - cc_apt_configure.rename_apt_lists(mirrors, target, arch) - found = sorted(os.listdir(apt_lists_d)) - self.assertEqual(expected, found) + cc_apt_configure.rename_apt_lists(mirrors, target.strpath, arch) + assert expected == sorted(os.listdir(apt_lists_d)) @staticmethod def test_apt_v3_proxy(): @@ -776,9 +780,9 @@ cfg, get_cloud(), "amd64" ) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir def test_apt_v3_mirror_default(self): """test_apt_v3_mirror_default - Test without defining a mirror""" @@ -789,9 +793,9 @@ mycloud = get_cloud() mirrors = cc_apt_configure.find_apt_mirror_info({}, mycloud, arch) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir def test_apt_v3_mirror_arches(self): """test_apt_v3_mirror_arches - Test arches selection of mirror""" @@ -811,9 +815,9 @@ mirrors = cc_apt_configure.find_apt_mirror_info(cfg, get_cloud(), arch) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir def test_apt_v3_mirror_arches_default(self): """test_apt_v3_mirror_arches - Test falling back to default arch""" @@ -834,9 +838,9 @@ cfg, get_cloud(), "amd64" ) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir @mock.patch("cloudinit.config.cc_apt_configure.util.get_dpkg_architecture") def test_apt_v3_get_def_mir_non_intel_no_arch( @@ -848,16 +852,14 @@ "PRIMARY": "http://ports.ubuntu.com/ubuntu-ports", "SECURITY": "http://ports.ubuntu.com/ubuntu-ports", } - self.assertEqual(expected, cc_apt_configure.get_default_mirrors()) + assert expected == cc_apt_configure.get_default_mirrors() def test_apt_v3_get_default_mirrors_non_intel_with_arch(self): - found = cc_apt_configure.get_default_mirrors("ppc64el") - expected = { "PRIMARY": "http://ports.ubuntu.com/ubuntu-ports", "SECURITY": "http://ports.ubuntu.com/ubuntu-ports", } - self.assertEqual(expected, found) + assert expected == cc_apt_configure.get_default_mirrors("ppc64el") def test_apt_v3_mirror_arches_sysdefault(self): """test_apt_v3_mirror_arches - Test arches fallback to sys default""" @@ -879,9 +881,9 @@ mirrors = cc_apt_configure.find_apt_mirror_info(cfg, mycloud, arch) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir def test_apt_v3_mirror_search(self): """test_apt_v3_mirror_search - Test searching mirrors in a list @@ -905,9 +907,9 @@ calls = [call(["pfailme", pmir]), call(["sfailme", smir])] mocksearch.assert_has_calls(calls) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir def test_apt_v3_mirror_search_many2(self): """test_apt_v3_mirror_search_many3 - Test both mirrors specs at once""" @@ -953,9 +955,9 @@ ) mockse.assert_not_called() - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir @pytest.mark.allow_dns_lookup def test_apt_v3_url_resolvable(self): @@ -985,8 +987,8 @@ mocksock.assert_any_call("example.invalid.", None, 0, 0, 1, 2) mocksock.assert_any_call("us.archive.ubuntu.com", None) - self.assertTrue(ret) - self.assertTrue(ret2) + assert ret is True + assert ret2 is True # side effect need only bad ret after initial call with mock.patch.object( @@ -995,7 +997,7 @@ ret3 = util.is_resolvable_url("http://failme.com/ubuntu") calls = [call("failme.com", None)] mocksock.assert_has_calls(calls) - self.assertFalse(ret3) + assert ret3 is False def test_apt_v3_disable_suites(self): """test_disable_suites - disable_suites with many configurations""" @@ -1014,7 +1016,7 @@ deb-src http://ubuntu.com//ubuntu universe multiverse deb http://ubuntu.com/ubuntu/ xenial-proposed main""" result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable release suite disabled = ["$RELEASE"] @@ -1025,7 +1027,7 @@ deb-src http://ubuntu.com//ubuntu universe multiverse deb http://ubuntu.com/ubuntu/ xenial-proposed main""" result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable other suite disabled = ["$RELEASE-updates"] @@ -1038,7 +1040,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # multi disable disabled = ["$RELEASE-updates", "$RELEASE-security"] @@ -1052,7 +1054,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # multi line disable (same suite multiple times in input) disabled = ["$RELEASE-updates", "$RELEASE-security"] @@ -1077,7 +1079,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # comment in input disabled = ["$RELEASE-updates", "$RELEASE-security"] @@ -1103,7 +1105,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable custom suite disabled = ["foobar"] @@ -1116,7 +1118,7 @@ deb http://ubuntu.com//ubuntu xenial-security main # suite disabled by cloud-init: deb http://ubuntu.com/ubuntu/ foobar main""" result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable non existing suite disabled = ["foobar"] @@ -1129,7 +1131,7 @@ deb http://ubuntu.com//ubuntu xenial-security main deb http://ubuntu.com/ubuntu/ notfoobar main""" result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable suite with option disabled = ["$RELEASE-updates"] @@ -1147,7 +1149,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable suite with more options and auto $RELEASE expansion disabled = ["updates"] @@ -1163,7 +1165,7 @@ deb-src http://ubuntu.com//ubuntu universe multiverse deb http://ubuntu.com/ubuntu/ xenial-proposed main""" result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result # single disable suite while options at others disabled = ["$RELEASE-security"] @@ -1181,7 +1183,7 @@ deb http://ubuntu.com/ubuntu/ xenial-proposed main""" ) result = cc_apt_configure.disable_suites(disabled, orig, release) - self.assertEqual(expect, result) + assert expect == result def test_disable_suites_blank_lines(self): """test_disable_suites_blank_lines - ensure blank lines allowed""" @@ -1196,9 +1198,8 @@ rel = "trusty" repo = "http://example.com/mirrors/ubuntu" orig = "\n".join(lines) % {"repo": repo, "rel": rel} - self.assertEqual( - orig, cc_apt_configure.disable_suites(["proposed"], orig, rel) - ) + + assert orig == cc_apt_configure.disable_suites(["proposed"], orig, rel) @mock.patch("cloudinit.util.get_hostname", return_value="abc.localdomain") def test_apt_v3_mirror_search_dns(self, m_get_hostname): @@ -1260,11 +1261,11 @@ ] mockse.assert_has_calls(calls) - self.assertEqual(mirrors["MIRROR"], pmir) - self.assertEqual(mirrors["PRIMARY"], pmir) - self.assertEqual(mirrors["SECURITY"], smir) + assert mirrors["MIRROR"] == pmir + assert mirrors["PRIMARY"] == pmir + assert mirrors["SECURITY"] == smir - def test_apt_v3_add_mirror_keys(self): + def test_apt_v3_add_mirror_keys(self, tmpdir): """test_apt_v3_add_mirror_keys - Test adding key for mirrors""" arch = "amd64" cfg = { @@ -1287,7 +1288,7 @@ } with mock.patch.object(cc_apt_configure, "add_apt_key_raw") as mockadd: - cc_apt_configure.add_mirror_keys(cfg, TARGET) + cc_apt_configure.add_mirror_keys(cfg, None, tmpdir.strpath) calls = [ mock.call("fakekey_primary", "primary", hardened=False), mock.call("fakekey_security", "security", hardened=False), @@ -1295,7 +1296,7 @@ mockadd.assert_has_calls(calls, any_order=True) -class TestDebconfSelections(TestCase): +class TestDebconfSelections: @mock.patch("cloudinit.config.cc_apt_configure.subp.subp") def test_set_sel_appends_newline_if_absent(self, m_subp): """Automatically append a newline to debconf-set-selections config.""" @@ -1308,7 +1309,7 @@ capture=True, target=None, ) - self.assertEqual([m_call, m_call], m_subp.call_args_list) + assert [m_call, m_call] == m_subp.call_args_list @mock.patch("cloudinit.config.cc_apt_configure.debconf_set_selections") def test_no_set_sel_if_none_to_set(self, m_set_sel): @@ -1333,8 +1334,8 @@ m_set_sel.return_value = None cc_apt_configure.apply_debconf_selections({"debconf_selections": data}) - self.assertTrue(m_get_inst.called) - self.assertEqual(m_set_sel.call_count, 1) + assert m_get_inst.called is True + assert m_set_sel.call_count == 1 # assumes called with *args value. selections = m_set_sel.call_args_list[0][0][0].decode() @@ -1342,7 +1343,7 @@ missing = [ line for line in lines if line not in selections.splitlines() ] - self.assertEqual([], missing) + assert [] == missing @mock.patch("cloudinit.config.cc_apt_configure.dpkg_reconfigure") @mock.patch("cloudinit.config.cc_apt_configure.debconf_set_selections") @@ -1374,10 +1375,10 @@ # reconfigure should be called with the intersection # of (packages in config, packages installed) - self.assertEqual(m_dpkg_r.call_count, 1) + assert m_dpkg_r.call_count == 1 # assumes called with *args (dpkg_reconfigure([a,b,c], target=)) packages = m_dpkg_r.call_args_list[0][0][0] - self.assertEqual(set(["cloud-init", "pkgb"]), set(packages)) + assert set(["cloud-init", "pkgb"]) == set(packages) @mock.patch("cloudinit.config.cc_apt_configure.dpkg_reconfigure") @mock.patch("cloudinit.config.cc_apt_configure.debconf_set_selections") @@ -1400,12 +1401,12 @@ cc_apt_configure.apply_debconf_selections({"debconf_selections": data}) - self.assertTrue(m_get_inst.called) - self.assertEqual(m_dpkg_r.call_count, 0) + assert m_get_inst.called is True + assert m_dpkg_r.call_count == 0 @mock.patch("cloudinit.config.cc_apt_configure.subp.subp") - def test_dpkg_reconfigure_does_reconfigure(self, m_subp): - target = "/foo-target" + def test_dpkg_reconfigure_does_reconfigure(self, m_subp, tmpdir): + target = tmpdir.strpath # due to the way the cleaners are called (via dictionary reference) # mocking clean_cloud_init directly does not work. So we mock @@ -1421,16 +1422,15 @@ ) # cloud-init is actually the only package we have a cleaner for # so for now, its the only one that should reconfigured - self.assertTrue(m_subp.called) ci_cleaner.assert_called_with(target) - self.assertEqual(m_subp.call_count, 1) + assert m_subp.call_count == 1 found = m_subp.call_args_list[0][0][0] expected = [ "dpkg-reconfigure", "--frontend=noninteractive", "cloud-init", ] - self.assertEqual(expected, found) + assert expected == found @mock.patch("cloudinit.config.cc_apt_configure.subp.subp") def test_dpkg_reconfigure_not_done_on_no_data(self, m_subp): @@ -1443,5 +1443,226 @@ m_subp.assert_not_called() -# -# vi: ts=4 expandtab +DEB822_SINGLE_SUITE = """\ +Types: deb +URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +Suites: mantic # Some comment +Components: main +""" + +DEB822_DISABLED_SINGLE_SUITE = """\ +## Entry disabled by cloud-init, due to disable_suites +# disabled by cloud-init: Types: deb +# disabled by cloud-init: URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +# disabled by cloud-init: Suites: mantic # Some comment +# disabled by cloud-init: Components: main +""" + +DEB822_SINGLE_SECTION_TWO_SUITES = """\ +Types: deb +URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +Suites: mantic mantic-updates +Components: main +""" + +DEB822_SINGLE_SECTION_TWO_SUITES_DISABLE_ONE = """\ +Types: deb +URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +# cloud-init disable_suites redacted: Suites: mantic mantic-updates +Suites: mantic-updates +Components: main +""" + +DEB822_SUITE_2 = """ +# APT Suite 2 +Types: deb +URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +Suites: mantic-backports +Components: main +""" + + +DEB822_DISABLED_SINGLE_SUITE = """\ +## Entry disabled by cloud-init, due to disable_suites +# disabled by cloud-init: Types: deb +# disabled by cloud-init: URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +# disabled by cloud-init: Suites: mantic # Some comment +# disabled by cloud-init: Components: main +""" + +DEB822_DISABLED_MULTIPLE_SUITES = """\ +## Entry disabled by cloud-init, due to disable_suites +# disabled by cloud-init: Types: deb +# disabled by cloud-init: URIs: https://ppa.launchpadcontent.net/cloud-init-dev/daily/ubuntu/ +# disabled by cloud-init: Suites: mantic mantic-updates +# disabled by cloud-init: Components: main +""" + + +class TestDisableSuitesDeb822: + @pytest.mark.parametrize( + "disabled_suites,src,expected", + ( + pytest.param( + [], + DEB822_SINGLE_SUITE, + DEB822_SINGLE_SUITE, + id="empty_suites_nochange", + ), + pytest.param( + ["$RELEASE-updates"], + DEB822_SINGLE_SUITE, + DEB822_SINGLE_SUITE, + id="no_matching_suites_nochange", + ), + pytest.param( + ["$RELEASE"], + DEB822_SINGLE_SUITE, + DEB822_DISABLED_SINGLE_SUITE, + id="matching_all_suites_disables_whole_section", + ), + pytest.param( + ["$RELEASE"], + DEB822_SINGLE_SECTION_TWO_SUITES + DEB822_SUITE_2, + DEB822_SINGLE_SECTION_TWO_SUITES_DISABLE_ONE + + "\n" + + DEB822_SUITE_2, + id="matching_some_suites_redacts_matches_and_comments_orig", + ), + pytest.param( + ["$RELEASE", "$RELEASE-updates"], + DEB822_SINGLE_SECTION_TWO_SUITES + DEB822_SUITE_2, + DEB822_DISABLED_MULTIPLE_SUITES + "\n" + DEB822_SUITE_2, + id="matching_all_suites_disables_specific_section", + ), + ), + ) + def test_disable_deb822_suites_disables_proper_suites( + self, disabled_suites, src, expected + ): + assert expected == cc_apt_configure.disable_suites_deb822( + disabled_suites, src, "mantic" + ) + + +APT_CONFIG_DUMP = """ +APT ""; +Dir "/"; +Dir::Etc "etc/myapt"; +Dir::Etc::sourcelist "sources.my.list"; +Dir::Etc::sourceparts "sources.my.list.d"; +Dir::Etc::main "apt.conf"; +""" + + +class TestGetAptCfg: + @skipIfAptPkg() + @pytest.mark.parametrize( + "subp_side_effect,expected", + ( + pytest.param( + [(APT_CONFIG_DUMP, "")], + { + "sourcelist": "/etc/myapt/sources.my.list", + "sourceparts": "/etc/myapt/sources.my.list.d/", + }, + id="no_aptpkg_use_apt_config_cmd", + ), + pytest.param( + [("", "")], + { + "sourcelist": "/etc/apt/sources.list", + "sourceparts": "/etc/apt/sources.list.d/", + }, + id="no_aptpkg_unparsable_apt_config_cmd_defaults", + ), + pytest.param( + [ + subp.ProcessExecutionError( + "No such file or directory 'apt-config'" + ) + ], + { + "sourcelist": "/etc/apt/sources.list", + "sourceparts": "/etc/apt/sources.list.d/", + }, + id="no_aptpkg_no_apt_config_cmd_defaults", + ), + ), + ) + def test_use_defaults_or_apt_config_dump( + self, subp_side_effect, expected, mocker + ): + subp = mocker.patch("cloudinit.config.cc_apt_configure.subp.subp") + subp.side_effect = subp_side_effect + assert expected == cc_apt_configure.get_apt_cfg() + subp.assert_called_once_with(["apt-config", "dump"]) + + +class TestIsDeb822SourcesFormat: + @pytest.mark.parametrize( + "content,is_deb822,warnings", + ( + pytest.param( + "#Something\ndeb-src http://url lunar multiverse\n", + False, + [], + id="any_deb_src_is_not_deb822", + ), + pytest.param( + "#Something\ndeb http://url lunar multiverse\n", + False, + [], + id="any_deb_url_is_not_deb822", + ), + pytest.param( + "#Something\ndeb http://url lunar multiverse\nTypes: deb\n", + False, + [], + id="even_some_deb822_fields_not_deb822_if_any_deb_line", + ), + pytest.param( + "#Something\nTypes: deb\n", + True, + [], + id="types_deb822_keys_and_no_deb_or_deb_src_is_deb822", + ), + pytest.param( + "#Something\nURIs: http://url\n", + True, + [], + id="uris_deb822_keys_and_no_deb_or_deb_src_is_deb822", + ), + pytest.param( + "#Something\nSuites: http://url\n", + True, + [], + id="suites_deb822_keys_and_no_deb_deb_src_is_deb822", + ), + pytest.param( + "#Something\nComponents: http://url\n", + True, + [], + id="components_deb822_keys_and_no_deb_deb_src_is_deb822", + ), + pytest.param( + "#Something neither deb/deb-src nor deb822\n", + False, + [ + "apt.sources_list value does not match either deb822" + " source keys or deb/deb-src list keys. Assuming APT" + " deb/deb-src list format." + ], + id="neither_deb822_keys_nor_deb_deb_src_warn_and_not_deb822", + ), + ), + ) + def test_is_deb822_format_prefers_non_deb822( + self, content, is_deb822, warnings, caplog + ): + with caplog.at_level(logging.WARNING): + assert is_deb822 is cc_apt_configure.is_deb822_sources_format( + content + ) + for warning in warnings: + assert warning in caplog.text diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_ansible.py cloud-init-23.4.3/tests/unittests/config/test_cc_ansible.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_ansible.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_ansible.py 2024-02-02 21:21:52.000000000 +0000 @@ -310,14 +310,14 @@ if install == "distro": cloud.distro.install_packages.assert_called_once() cloud.distro.install_packages.assert_called_with( - "ansible-core" + ["ansible-core"] ) elif install == "pip": if HAS_PIP: assert 0 == cloud.distro.install_packages.call_count else: cloud.distro.install_packages.assert_called_with( - "python3-pip" + ["python3-pip"] ) @mock.patch(M_PATH + "which", return_value=False) diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_apk_configure.py cloud-init-23.4.3/tests/unittests/config/test_cc_apk_configure.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_apk_configure.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_apk_configure.py 2024-02-02 21:21:52.000000000 +0000 @@ -384,6 +384,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_apt_configure.py cloud-init-23.4.3/tests/unittests/config/test_cc_apt_configure.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_apt_configure.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_apt_configure.py 2024-02-02 21:21:52.000000000 +0000 @@ -6,12 +6,14 @@ import pytest +from cloudinit.config import cc_apt_configure from cloudinit.config.schema import ( SchemaValidationError, get_schema, validate_cloudconfig_schema, ) from tests.unittests.helpers import skipUnlessJsonSchema +from tests.unittests.util import get_cloud class TestAPTConfigureSchema: @@ -199,4 +201,68 @@ validate_cloudconfig_schema(config, schema, strict=True) -# vi: ts=4 expandtab +class TestEnsureDependencies: + @pytest.mark.parametrize( + "cfg, already_installed, expected_install", + ( + pytest.param({}, [], [], id="empty_cfg_no_pkg_installs"), + pytest.param( + {"sources": {"s1": {"keyid": "haveit"}}}, + ["gpg"], + [], + id="cfg_needs_gpg_no_installs_when_gpg_present", + ), + pytest.param( + {"sources": {"s1": {"keyid": "haveit"}}}, + [], + ["gnupg"], + id="cfg_needs_gpg_installs_gnupg_when_absent", + ), + pytest.param( + {"primary": [{"keyid": "haveit"}]}, + [], + ["gnupg"], + id="cfg_primary_needs_gpg_installs_gnupg_when_absent", + ), + pytest.param( + {"security": [{"keyid": "haveit"}]}, + [], + ["gnupg"], + id="cfg_security_needs_gpg_installs_gnupg_when_absent", + ), + pytest.param( + {"sources": {"s1": {"source": "ppa:yep"}}}, + ["add-apt-repository"], + [], + id="cfg_needs_sw_prop_common_when_present", + ), + pytest.param( + {"sources": {"s1": {"source": "ppa:yep"}}}, + [], + ["software-properties-common"], + id="cfg_needs_sw_prop_common_when_add_apt_repo_absent", + ), + ), + ) + def test_only_install_needed_packages( + self, cfg, already_installed, expected_install, mocker + ): + """Only invoke install_packages when package installs are necessary""" + mycloud = get_cloud("debian") + install_packages = mocker.patch.object( + mycloud.distro, "install_packages" + ) + matcher = re.compile(cc_apt_configure.ADD_APT_REPO_MATCH).search + + def fake_which(cmd): + if cmd in already_installed: + return "foundit" + return None + + which = mocker.patch.object(cc_apt_configure.shutil, "which") + which.side_effect = fake_which + cc_apt_configure._ensure_dependencies(cfg, matcher, mycloud) + if expected_install: + install_packages.assert_called_once_with(expected_install) + else: + install_packages.assert_not_called() diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_apt_pipelining.py cloud-init-23.4.3/tests/unittests/config/test_cc_apt_pipelining.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_apt_pipelining.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_apt_pipelining.py 2024-02-02 21:21:52.000000000 +0000 @@ -2,6 +2,8 @@ """Tests cc_apt_pipelining handler""" +import re + import pytest import cloudinit.config.cc_apt_pipelining as cc_apt_pipelining @@ -39,15 +41,15 @@ ({"apt_pipelining": 1}, None), ({"apt_pipelining": True}, None), ({"apt_pipelining": False}, None), - ({"apt_pipelining": "none"}, None), - ({"apt_pipelining": "unchanged"}, None), ({"apt_pipelining": "os"}, None), # Invalid schemas + ({"apt_pipelining": "none"}, "Deprecated in version"), + ({"apt_pipelining": "unchanged"}, "Deprecated in version"), ( {"apt_pipelining": "bogus"}, - "Cloud config schema errors: apt_pipelining: 'bogus' is not of" - " type 'integer', apt_pipelining: 'bogus' is not valid under" - " any of the given schemas", + re.escape( + "Cloud config schema errors: apt_pipelining: 'bogus' is" + ), ), ), ) @@ -61,6 +63,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_bootcmd.py cloud-init-23.4.3/tests/unittests/config/test_cc_bootcmd.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_bootcmd.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_bootcmd.py 2024-02-02 21:21:52.000000000 +0000 @@ -145,10 +145,8 @@ ] }, "Cloud config schema errors: bootcmd.1: 20 is not of type" - " 'array', bootcmd.1: 20 is not valid under any of the given" - " schemas, bootcmd.3: {'a': 'n'} is not of type 'array'," - " bootcmd.3: {'a': 'n'} is not valid under any of the given" - " schemas", + " 'array', bootcmd.1: 20 is not of type 'string', bootcmd.3:" + " {'a': 'n'} is not of type 'array'", ), ), ) @@ -159,6 +157,3 @@ schema = get_schema() with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_byobu.py cloud-init-23.4.3/tests/unittests/config/test_cc_byobu.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_byobu.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_byobu.py 2024-02-02 21:21:52.000000000 +0000 @@ -46,6 +46,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_ca_certs.py cloud-init-23.4.3/tests/unittests/config/test_cc_ca_certs.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_ca_certs.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_ca_certs.py 2024-02-02 21:21:52.000000000 +0000 @@ -356,8 +356,10 @@ if distro_name in ["debian", "ubuntu"]: mock_subp.assert_called_once_with( ("debconf-set-selections", "-"), - "ca-certificates ca-certificates/trust_new_crts" - " select no", + data=( + "ca-certificates ca-certificates/" + "trust_new_crts select no" + ), ) else: assert mock_subp.call_count == 0 @@ -433,7 +435,7 @@ @mock.patch.object(cc_ca_certs, "update_ca_certs") def test_deprecate_key_warnings(self, update_ca_certs, caplog): """Assert warnings are logged for deprecated keys.""" - logger.setupLogging() + logger.setup_logging() cloud = get_cloud("ubuntu") cc_ca_certs.handle( "IGNORE", {"ca-certs": {"remove-defaults": False}}, cloud, [] @@ -444,7 +446,7 @@ ] for warning in expected_warnings: assert warning in caplog.text - assert "DEPRECAT" in caplog.text + assert "deprecat" in caplog.text assert 1 == update_ca_certs.call_count @mock.patch.object(cc_ca_certs, "update_ca_certs") diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_disable_ec2_metadata.py cloud-init-23.4.3/tests/unittests/config/test_cc_disable_ec2_metadata.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_disable_ec2_metadata.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_disable_ec2_metadata.py 2024-02-02 21:21:52.000000000 +0000 @@ -73,6 +73,3 @@ schema = get_schema() with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_growpart.py cloud-init-23.4.3/tests/unittests/config/test_cc_growpart.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_growpart.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_growpart.py 2024-02-02 21:21:52.000000000 +0000 @@ -678,9 +678,8 @@ {"growpart": {"mode": "false"}}, pytest.raises( SchemaValidationError, - match=( - "growpart.mode: 'false' is not valid under any of the" - " given schemas" + match=re.escape( + "growpart.mode: 'false' is not one of [False" ), ), ), @@ -688,9 +687,8 @@ {"growpart": {"mode": "a"}}, pytest.raises( SchemaValidationError, - match=( - "growpart.mode: 'a' is not valid under any of the" - " given schemas" + match=re.escape( + "growpart.mode: 'a' is not one of ['auto'," ), ), ), diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_keyboard.py cloud-init-23.4.3/tests/unittests/config/test_cc_keyboard.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_keyboard.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_keyboard.py 2024-02-02 21:21:52.000000000 +0000 @@ -179,6 +179,3 @@ variant, ], ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_keys_to_console.py cloud-init-23.4.3/tests/unittests/config/test_cc_keys_to_console.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_keys_to_console.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_keys_to_console.py 2024-02-02 21:21:52.000000000 +0000 @@ -114,6 +114,3 @@ schema = get_schema() with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_landscape.py cloud-init-23.4.3/tests/unittests/config/test_cc_landscape.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_landscape.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_landscape.py 2024-02-02 21:21:52.000000000 +0000 @@ -56,7 +56,7 @@ None, ) mycloud.distro.install_packages.assert_called_once_with( - ("landscape-client",) + ["landscape-client"] ) assert [ mock.call( @@ -119,7 +119,7 @@ None, ) mycloud.distro.install_packages.assert_called_once_with( - ("landscape-client",) + ["landscape-client"] ) assert expected_calls == m_subp.call_args_list assert "RUN=1\n" == default_fn.read() diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_locale.py cloud-init-23.4.3/tests/unittests/config/test_cc_locale.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_locale.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_locale.py 2024-02-02 21:21:52.000000000 +0000 @@ -150,6 +150,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, schema, strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_lxd.py cloud-init-23.4.3/tests/unittests/config/test_cc_lxd.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_lxd.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_lxd.py 2024-02-02 21:21:52.000000000 +0000 @@ -471,6 +471,3 @@ cc_lxd.supplemental_schema_validation( init_cfg, bridge_cfg, preseed_str ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_mcollective.py cloud-init-23.4.3/tests/unittests/config/test_cc_mcollective.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_mcollective.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_mcollective.py 2024-02-02 21:21:52.000000000 +0000 @@ -152,7 +152,7 @@ cc_mcollective.handle("cc_mcollective", mycfg, cc, []) self.assertTrue(cc.distro.install_packages.called) install_pkg = cc.distro.install_packages.call_args_list[0][0][0] - self.assertEqual(install_pkg, ("mcollective",)) + self.assertEqual(install_pkg, ["mcollective"]) self.assertTrue(mock_subp.subp.called) self.assertEqual( @@ -190,6 +190,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_mounts.py cloud-init-23.4.3/tests/unittests/config/test_cc_mounts.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_mounts.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_mounts.py 2024-02-02 21:21:52.000000000 +0000 @@ -612,13 +612,16 @@ ({"swap": {"size": "1.5MB"}}, "swap.size:"), ( {"swap": {"maxsize": "1.5MT"}}, - "swap.maxsize: '1.5MT' is not valid", + re.escape("swap.maxsize: '1.5MT' does not match '^([0-9]+)?"), ), ( {"swap": {"maxsize": "..5T"}}, - "swap.maxsize: '..5T' is not valid", + re.escape("swap.maxsize: '..5T' does not match '^([0-9]+)?"), + ), + ( + {"swap": {"size": "K"}}, + "swap.size: 'K' is not of type 'integer'", ), - ({"swap": {"size": "K"}}, "swap.size: 'K' is not valid"), ], ) @test_helpers.skipUnlessJsonSchema() diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_ntp.py cloud-init-23.4.3/tests/unittests/config/test_cc_ntp.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_ntp.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_ntp.py 2024-02-02 21:21:52.000000000 +0000 @@ -540,9 +540,12 @@ # we are on ubuntu-core here self.m_snappy.return_value = True - # ubuntu core systems will have timesyncd installed + # ubuntu core systems will have timesyncd installed, so simulate that. + # First None is for the 'eatmydata' check when initializing apt + # when initializing the distro class. The rest represent possible + # finds the various npt services m_which.side_effect = iter( - [None, "/lib/systemd/systemd-timesyncd", None, None, None] + [None, None, "/lib/systemd/systemd-timesyncd", None, None, None] ) distro = "ubuntu" mycloud = self._get_cloud(distro) @@ -606,7 +609,7 @@ cfg = {"ntp": {"ntp_client": client}} for distro in cc_ntp.distros: # client is not installed - m_which.side_effect = iter([None]) + m_which.return_value = None mycloud = self._get_cloud(distro) with mock.patch.object( mycloud.distro, "install_packages" diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_package_update_upgrade_install.py cloud-init-23.4.3/tests/unittests/config/test_cc_package_update_upgrade_install.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_package_update_upgrade_install.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_package_update_upgrade_install.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,11 +1,182 @@ +# This file is part of cloud-init. See LICENSE file for license information. +import logging +from unittest import mock + import pytest +from cloudinit import subp +from cloudinit.config.cc_package_update_upgrade_install import handle from cloudinit.config.schema import ( SchemaValidationError, get_schema, validate_cloudconfig_schema, ) +from cloudinit.distros import PackageInstallerError +from cloudinit.subp import SubpResult from tests.unittests.helpers import skipUnlessJsonSchema +from tests.unittests.util import get_cloud + + +@pytest.fixture +def common_mocks(mocker): + mocker.patch("os.path.isfile", return_value=False) + mocker.patch("cloudinit.distros.Distro.update_package_sources") + mocker.patch( + "cloudinit.distros.package_management.apt.Apt.update_package_sources" + ) + mocker.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + return_value=True, + ) + + +class TestMultiplePackageManagers: + def test_explicit_apt(self, common_mocks): + def _new_subp(*args, **kwargs): + if args and "apt-cache" in args[0]: + return SubpResult("pkg1\npkg2\npkg3", None) + + cloud = get_cloud("ubuntu") + cfg = {"packages": [{"apt": ["pkg1", "pkg2"]}]} + with mock.patch( + "cloudinit.subp.subp", side_effect=_new_subp + ) as m_subp: + handle("", cfg, cloud, []) + + assert len(m_subp.call_args_list) == 2 + assert m_subp.call_args_list[0] == mock.call(["apt-cache", "pkgnames"]) + + for arg in ["apt-get", "install", "pkg1", "pkg2"]: + assert arg in m_subp.call_args_list[1][1]["args"] + + def test_explicit_apt_version(self, common_mocks): + def _new_subp(*args, **kwargs): + if args and "apt-cache" in args[0]: + return SubpResult("pkg1\npkg2\npkg3", None) + + cloud = get_cloud("ubuntu") + cfg = {"packages": [{"apt": ["pkg1", ["pkg2", "1.2.3"]]}]} + with mock.patch( + "cloudinit.subp.subp", side_effect=_new_subp + ) as m_subp: + handle("", cfg, cloud, []) + + assert len(m_subp.call_args_list) == 2 + assert m_subp.call_args_list[0] == mock.call(["apt-cache", "pkgnames"]) + + for arg in ["apt-get", "install", "pkg1", "pkg2=1.2.3"]: + assert arg in m_subp.call_args_list[1][1]["args"] + + @mock.patch("cloudinit.subp.subp") + def test_explicit_snap(self, m_subp, common_mocks): + cloud = get_cloud("ubuntu") + cfg = {"packages": [{"snap": ["pkg1", "pkg2"]}]} + handle("", cfg, cloud, []) + + assert len(m_subp.call_args_list) == 2 + assert mock.call(["snap", "install", "pkg1"]) in m_subp.call_args_list + assert mock.call(["snap", "install", "pkg2"]) in m_subp.call_args_list + + @mock.patch("cloudinit.subp.subp") + def test_explicit_snap_version(self, m_subp, common_mocks): + cloud = get_cloud("ubuntu") + cfg = {"packages": [{"snap": ["pkg1", ["pkg2", "--edge"]]}]} + handle("", cfg, cloud, []) + + assert len(m_subp.call_args_list) == 2 + assert mock.call(["snap", "install", "pkg1"]) in m_subp.call_args_list + assert ( + mock.call(["snap", "install", "pkg2", "--edge"]) + in m_subp.call_args_list + ) + + def test_combined(self, common_mocks): + """Ensure that pkg1 is installed by snap since it isn't available + in the apt cache, and ensure pkg5 is installed by snap even though it + is available under apt because it is explicitly listed under snap.""" + + def _new_subp(*args, **kwargs): + if args and "apt-cache" in args[0]: + return SubpResult("pkg2\npkg3\npkg5\npkg6", None) + + cloud = get_cloud("ubuntu") + cfg = { + "packages": [ + "pkg1", + {"apt": ["pkg2", ["pkg3", "1.2.3"]]}, + {"snap": ["pkg4", ["pkg5", "--edge"]]}, + "pkg6", + ], + } + with mock.patch( + "cloudinit.subp.subp", side_effect=_new_subp + ) as m_subp: + handle("", cfg, cloud, []) + + assert len(m_subp.call_args_list) == 5 + assert m_subp.call_args_list[0] == mock.call(["apt-cache", "pkgnames"]) + for arg in ["apt-get", "install", "pkg2", "pkg3=1.2.3", "pkg6"]: + assert arg in m_subp.call_args_list[1][1]["args"] + + assert mock.call(["snap", "install", "pkg1"]) in m_subp.call_args_list + assert ( + mock.call(["snap", "install", "pkg5", "--edge"]) + in m_subp.call_args_list + ) + assert mock.call(["snap", "install", "pkg4"]) in m_subp.call_args_list + + def test_error_apt(self, common_mocks): + """Since we have already checked that the package(s) exists in apt, + if we have an error in apt, we don't want to fall through to + additional package installs. + """ + + def _new_subp(*args, **kwargs): + if args and "apt-cache" in args[0]: + return SubpResult("pkg1", None) + if "args" in kwargs and "install" in kwargs["args"]: + raise subp.ProcessExecutionError( + cmd=kwargs["args"], + stdout="dontcare", + stderr="E: Unable to locate package pkg1", + exit_code=100, + ) + + cloud = get_cloud("ubuntu") + cfg = {"packages": ["pkg1"]} + with mock.patch("cloudinit.subp.subp", side_effect=_new_subp): + with pytest.raises(subp.ProcessExecutionError): + handle("", cfg, cloud, []) + + def test_error_snap(self, common_mocks, caplog): + """Since we haven't checked the package(s) existence, we should fall + through to additional package installs. + """ + caplog.set_level(logging.DEBUG) + + def _new_subp(*args, **kwargs): + if args: + if "apt-cache" in args[0]: + return SubpResult("", None) + if "install" in args[0]: + raise subp.ProcessExecutionError( + cmd=args[0], + stdout="dontcare", + stderr='error: snap "pkg1" not found', + exit_code=1, + ) + + cloud = get_cloud("ubuntu") + cfg = {"packages": ["pkg1"]} + with mock.patch("cloudinit.subp.subp", side_effect=_new_subp): + with pytest.raises(PackageInstallerError): + handle("", cfg, cloud, []) + + assert caplog.records[-3].levelname == "WARNING" + assert ( + caplog.records[-3].message + == "Failed to install packages: ['pkg1']" + ) class TestPackageUpdateUpgradeSchema: diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_power_state_change.py cloud-init-23.4.3/tests/unittests/config/test_cc_power_state_change.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_power_state_change.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_power_state_change.py 2024-02-02 21:21:52.000000000 +0000 @@ -215,6 +215,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_puppet.py cloud-init-23.4.3/tests/unittests/config/test_cc_puppet.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_puppet.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_puppet.py 2024-02-02 21:21:52.000000000 +0000 @@ -11,6 +11,7 @@ get_schema, validate_cloudconfig_schema, ) +from cloudinit.distros import PackageInstallerError from cloudinit.subp import ProcessExecutionError from tests.unittests.helpers import CiTestCase, mock, skipUnlessJsonSchema from tests.unittests.util import get_cloud @@ -115,7 +116,7 @@ cfg = {"puppet": {}} cc_puppet.handle("notimportant", cfg, self.cloud, None) self.assertEqual( - [mock.call(("puppet-agent", None))], + [mock.call(["puppet-agent"])], self.cloud.distro.install_packages.call_args_list, ) @@ -127,7 +128,7 @@ cfg = {"puppet": {"install": True}} cc_puppet.handle("notimportant", cfg, self.cloud, None) self.assertIn( - [mock.call(("puppet-agent", None))], + [mock.call(["puppet-agent"])], self.cloud.distro.install_packages.call_args_list, ) @@ -236,7 +237,7 @@ cfg = {"puppet": {"version": "3.8"}} cc_puppet.handle("notimportant", cfg, self.cloud, None) self.assertEqual( - [mock.call(("puppet-agent", "3.8"))], + [mock.call([["puppet-agent", "3.8"]])], self.cloud.distro.install_packages.call_args_list, ) @@ -407,7 +408,7 @@ "tests.unittests.util.MockDistro.install_packages" ) as install_pkg: # puppet-agent not installed, but puppet is - install_pkg.side_effect = (ProcessExecutionError, 0) + install_pkg.side_effect = (PackageInstallerError, 0) cc_puppet.handle("notimportant", cfg, self.cloud, None) expected_calls = [ diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_resizefs.py cloud-init-23.4.3/tests/unittests/config/test_cc_resizefs.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_resizefs.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_resizefs.py 2024-02-02 21:21:52.000000000 +0000 @@ -6,6 +6,7 @@ import pytest from cloudinit.config.cc_resizefs import ( + _resize_bcachefs, _resize_btrfs, _resize_ext, _resize_ufs, @@ -169,6 +170,14 @@ ("growfs", "-y", mount_point), _resize_ufs(mount_point, devpth) ) + def test_resize_bcachefs_cmd_return(self): + mount_point = "/" + devpth = "/dev/sdf3" + self.assertEqual( + ("bcachefs", "device", "resize", devpth), + _resize_bcachefs(mount_point, devpth), + ) + @mock.patch("cloudinit.util.is_container", return_value=False) @mock.patch("cloudinit.util.parse_mount") @mock.patch("cloudinit.util.get_device_info_from_zpool") @@ -508,6 +517,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_resolv_conf.py cloud-init-23.4.3/tests/unittests/config/test_cc_resolv_conf.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_resolv_conf.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_resolv_conf.py 2024-02-02 21:21:52.000000000 +0000 @@ -261,6 +261,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_rh_subscription.py cloud-init-23.4.3/tests/unittests/config/test_cc_rh_subscription.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_rh_subscription.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_rh_subscription.py 2024-02-02 21:21:52.000000000 +0000 @@ -339,6 +339,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_runcmd.py cloud-init-23.4.3/tests/unittests/config/test_cc_runcmd.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_runcmd.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_runcmd.py 2024-02-02 21:21:52.000000000 +0000 @@ -111,6 +111,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_seed_random.py cloud-init-23.4.3/tests/unittests/config/test_cc_seed_random.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_seed_random.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_seed_random.py 2024-02-02 21:21:52.000000000 +0000 @@ -254,6 +254,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_set_hostname.py cloud-init-23.4.3/tests/unittests/config/test_cc_set_hostname.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_set_hostname.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_set_hostname.py 2024-02-02 21:21:52.000000000 +0000 @@ -259,5 +259,92 @@ contents = util.load_file("/etc/hostname") self.assertEqual("blah", contents.strip()) + def test_create_hostname_file_false(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("debian") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") -# vi: ts=4 expandtab + def test_create_hostname_file_false_arch(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("arch") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") + + def test_create_hostname_file_false_alpine(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("alpine") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") + + def test_create_hostname_file_false_gentoo(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("gentoo") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") + + def test_create_hostname_file_false_photon(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("photon") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") + + def test_create_hostname_file_false_rhel(self): + cfg = { + "hostname": "foo", + "fqdn": "foo.blah.yahoo.com", + "create_hostname_file": False, + } + distro = self._fetch_distro("rhel") + paths = helpers.Paths({"cloud_dir": self.tmp}) + ds = None + cc = cloud.Cloud(ds, paths, {}, distro, None) + self.patchUtils(self.tmp) + cc_set_hostname.handle("cc_set_hostname", cfg, cc, []) + with self.assertRaises(FileNotFoundError): + util.load_file("/etc/hostname") diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_snap.py cloud-init-23.4.3/tests/unittests/config/test_cc_snap.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_snap.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_snap.py 2024-02-02 21:21:52.000000000 +0000 @@ -165,7 +165,6 @@ class TestRunCommands(CiTestCase): - with_logs = True allowed_subp = [CiTestCase.SUBP_SHELL_TRUE] @@ -266,13 +265,23 @@ ({"snap": {"commands": {"01": "also valid"}}}, None), ({"snap": {"assertions": ["valid"]}}, None), ({"snap": {"assertions": {"01": "also valid"}}}, None), - ({"commands": [["echo", "bye"], ["echo", "bye"]]}, None), - ({"commands": ["echo bye", "echo bye"]}, None), + ({"snap": {"commands": [["echo", "bye"], ["echo", "bye"]]}}, None), + ({"snap": {"commands": ["echo bye", "echo bye"]}}, None), + ( + { + "snap": { + "commands": { + "00": ["echo", "bye"], + "01": ["echo", "bye"], + } + } + }, + None, + ), ( - {"commands": {"00": ["echo", "bye"], "01": ["echo", "bye"]}}, + {"snap": {"commands": {"00": "echo bye", "01": "echo bye"}}}, None, ), - ({"commands": {"00": "echo bye", "01": "echo bye"}}, None), # Invalid ({"snap": "wrong type"}, "'wrong type' is not of type 'object'"), ( @@ -333,6 +342,3 @@ content = "\n".join(cfg["snap"]["assertions"]) util.write_file(compare_file, content.encode("utf-8")) assert util.load_file(compare_file) == util.load_file(assert_file) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_spacewalk.py cloud-init-23.4.3/tests/unittests/config/test_cc_spacewalk.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_spacewalk.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_spacewalk.py 2024-02-02 21:21:52.000000000 +0000 @@ -43,6 +43,3 @@ ], capture=False, ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_timezone.py cloud-init-23.4.3/tests/unittests/config/test_cc_timezone.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_timezone.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_timezone.py 2024-02-02 21:21:52.000000000 +0000 @@ -48,6 +48,3 @@ contents = util.load_file("/etc/localtime") self.assertEqual(dummy_contents, contents.strip()) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_ubuntu_advantage.py cloud-init-23.4.3/tests/unittests/config/test_cc_ubuntu_advantage.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_ubuntu_advantage.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_ubuntu_advantage.py 2024-02-02 21:21:52.000000000 +0000 @@ -65,17 +65,12 @@ ) sys.modules["uaclient.api.exceptions"] = _exceptions - # Messages - m_messages = mock.Mock() - m_messages.ALREADY_ENABLED.name = "service-already-enabled" - sys.modules["uaclient.messages"] = m_messages - @pytest.mark.usefixtures("fake_uaclient") @mock.patch(f"{MPATH}.subp.subp") class TestConfigureUA: def test_configure_ua_attach_error(self, m_subp): - """Errors from ua attach command are raised.""" + """Errors from pro attach command are raised.""" m_subp.side_effect = subp.ProcessExecutionError( "Invalid token SomeToken" ) @@ -90,13 +85,13 @@ @pytest.mark.parametrize( "kwargs, call_args_list, log_record_tuples", [ - # When token is provided, attach the machine to ua using the token. + # When token is provided, attach to pro using the token. pytest.param( {"token": "SomeToken"}, [ mock.call( - ["ua", "attach", "SomeToken"], - logstring=["ua", "attach", "REDACTED"], + ["pro", "attach", "SomeToken"], + logstring=["pro", "attach", "REDACTED"], rcs={0, 2}, ) ], @@ -104,7 +99,7 @@ ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach REDACTED", + "Attaching to Ubuntu Advantage. pro attach REDACTED", ) ], id="with_token", @@ -114,8 +109,8 @@ {"token": "SomeToken", "enable": []}, [ mock.call( - ["ua", "attach", "SomeToken"], - logstring=["ua", "attach", "REDACTED"], + ["pro", "attach", "SomeToken"], + logstring=["pro", "attach", "REDACTED"], rcs={0, 2}, ) ], @@ -123,7 +118,7 @@ ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach REDACTED", + "Attaching to Ubuntu Advantage. pro attach REDACTED", ) ], id="with_empty_services", @@ -133,9 +128,9 @@ {"token": "SomeToken", "enable": ["fips"]}, [ mock.call( - ["ua", "attach", "--no-auto-enable", "SomeToken"], + ["pro", "attach", "--no-auto-enable", "SomeToken"], logstring=[ - "ua", + "pro", "attach", "--no-auto-enable", "REDACTED", @@ -144,12 +139,11 @@ ), mock.call( [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", "fips", ], capture=True, @@ -160,7 +154,7 @@ ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach" + "Attaching to Ubuntu Advantage. pro attach" " --no-auto-enable REDACTED", ) ], @@ -171,9 +165,9 @@ {"token": "SomeToken", "enable": "fips"}, [ mock.call( - ["ua", "attach", "--no-auto-enable", "SomeToken"], + ["pro", "attach", "--no-auto-enable", "SomeToken"], logstring=[ - "ua", + "pro", "attach", "--no-auto-enable", "REDACTED", @@ -182,12 +176,11 @@ ), mock.call( [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", "fips", ], capture=True, @@ -198,7 +191,7 @@ ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach" + "Attaching to Ubuntu Advantage. pro attach" " --no-auto-enable REDACTED", ), ( @@ -215,8 +208,8 @@ {"token": "SomeToken", "enable": {"deffo": "wont work"}}, [ mock.call( - ["ua", "attach", "SomeToken"], - logstring=["ua", "attach", "REDACTED"], + ["pro", "attach", "SomeToken"], + logstring=["pro", "attach", "REDACTED"], rcs={0, 2}, ) ], @@ -224,7 +217,7 @@ ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach REDACTED", + "Attaching to Ubuntu Advantage. pro attach REDACTED", ), ( MPATH, @@ -248,20 +241,20 @@ assert record_tuple in caplog.record_tuples def test_configure_ua_already_attached(self, m_subp, caplog): - """ua is already attached to an subscription""" + """pro is already attached to an subscription""" m_subp.rcs = 2 configure_ua(token="SomeToken") assert m_subp.call_args_list == [ mock.call( - ["ua", "attach", "SomeToken"], - logstring=["ua", "attach", "REDACTED"], + ["pro", "attach", "SomeToken"], + logstring=["pro", "attach", "REDACTED"], rcs={0, 2}, ) ] assert ( MPATH, logging.DEBUG, - "Attaching to Ubuntu Advantage. ua attach REDACTED", + "Attaching to Ubuntu Advantage. pro attach REDACTED", ) in caplog.record_tuples def test_configure_ua_attach_on_service_enabled( @@ -271,12 +264,11 @@ def fake_subp(cmd, capture=None, rcs=None, logstring=None): fail_cmds = [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", "livepatch", ] if cmd == fail_cmds and capture: @@ -297,18 +289,17 @@ configure_ua(token="SomeToken", enable=["livepatch"]) assert m_subp.call_args_list == [ mock.call( - ["ua", "attach", "--no-auto-enable", "SomeToken"], - logstring=["ua", "attach", "--no-auto-enable", "REDACTED"], + ["pro", "attach", "--no-auto-enable", "SomeToken"], + logstring=["pro", "attach", "--no-auto-enable", "REDACTED"], rcs={0, 2}, ), mock.call( [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", "livepatch", ], capture=True, @@ -326,12 +317,11 @@ def fake_subp(cmd, capture=None, rcs=None, logstring=None): fail_cmd = [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", ] if cmd[: len(fail_cmd)] == fail_cmd and capture: response = { @@ -369,18 +359,17 @@ ) assert m_subp.call_args_list == [ mock.call( - ["ua", "attach", "--no-auto-enable", "SomeToken"], - logstring=["ua", "attach", "--no-auto-enable", "REDACTED"], + ["pro", "attach", "--no-auto-enable", "SomeToken"], + logstring=["pro", "attach", "--no-auto-enable", "REDACTED"], rcs={0, 2}, ), mock.call( [ - "ua", + "pro", "enable", "--assume-yes", "--format", "json", - "--", "esm", "cc", "fips", @@ -409,7 +398,7 @@ def test_ua_enable_unexpected_error_codes(self, m_subp): def fake_subp(cmd, capture=None, **kwargs): - if cmd[:2] == ["ua", "enable"] and capture: + if cmd[:2] == ["pro", "enable"] and capture: raise subp.ProcessExecutionError(exit_code=255) return subp.SubpResult(json.dumps({"errors": []}), "") @@ -423,7 +412,7 @@ def test_ua_enable_non_json_response(self, m_subp): def fake_subp(cmd, capture=None, **kwargs): - if cmd[:2] == ["ua", "enable"] and capture: + if cmd[:2] == ["pro", "enable"] and capture: return subp.SubpResult("I dream to be a Json", "") return subp.SubpResult(json.dumps({"errors": []}), "") @@ -978,7 +967,7 @@ is_pro, ): """Checks that attach is not called in the case where we want only to - enable or disable ua auto-attach. + enable or disable pro auto-attach. """ m_should_auto_attach.return_value = is_pro handle("nomatter", cfg=cfg, cloud=self.cloud, args=None) @@ -1041,7 +1030,7 @@ @mock.patch(f"{MPATH}.subp.subp") def test_ua_config_error_invalid_url(self, m_subp, caplog): - """Errors from ua config command are raised.""" + """Errors from pro config command are raised.""" cfg = { "ubuntu_advantage": { "token": "SomeToken", @@ -1300,8 +1289,8 @@ ]: assert ( mock.call( - ["ua", "config", "set", ua_arg], - logstring=["ua", "config", "set", redacted_arg], + ["pro", "config", "set", ua_arg], + logstring=["pro", "config", "set", redacted_arg], ) in m_subp.call_args_list ) @@ -1317,15 +1306,15 @@ } set_ua_config(ua_config) for call in [ - mock.call(["ua", "config", "unset", "http_proxy"]), + mock.call(["pro", "config", "unset", "http_proxy"]), mock.call( [ - "ua", + "pro", "config", "set", "https_proxy=https://user:pass@some-proxy:8088", ], - logstring=["ua", "config", "set", "https_proxy=REDACTED"], + logstring=["pro", "config", "set", "https_proxy=REDACTED"], ), ]: assert call in m_subp.call_args_list @@ -1359,8 +1348,8 @@ set_ua_config(ua_config) assert [ mock.call( - ["ua", "config", "set", "asdf=qwer"], - logstring=["ua", "config", "set", "asdf=REDACTED"], + ["pro", "config", "set", "asdf=qwer"], + logstring=["pro", "config", "set", "asdf=REDACTED"], ) ] == m_subp.call_args_list assert "qwer" not in caplog.text @@ -1421,6 +1410,3 @@ assert "https://user:pass@some-proxy:8088" not in caplog.text assert "Disabling UA config for https_proxy\n" in caplog.text assert 'Failure enabling/disabling "https_proxy":\n' in caplog.text - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_ubuntu_drivers.py cloud-init-23.4.3/tests/unittests/config/test_cc_ubuntu_drivers.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_ubuntu_drivers.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_ubuntu_drivers.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,12 +1,12 @@ # This file is part of cloud-init. See LICENSE file for license information. import copy +import logging import os import re import pytest -from cloudinit import log from cloudinit.config import cc_ubuntu_drivers as drivers from cloudinit.config.schema import ( SchemaValidationError, @@ -277,7 +277,7 @@ assert [mock.call(install_gpgpu)] == m_subp.call_args_list assert ( MPATH[:-1], - log.WARNING, + logging.WARNING, ( "the available version of ubuntu-drivers is" " too old to perform requested driver installation" @@ -428,6 +428,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_update_etc_hosts.py cloud-init-23.4.3/tests/unittests/config/test_cc_update_etc_hosts.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_update_etc_hosts.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_update_etc_hosts.py 2024-02-02 21:21:52.000000000 +0000 @@ -2,6 +2,7 @@ import logging import os +import re import shutil import pytest @@ -98,9 +99,9 @@ {"manage_etc_hosts": "templatey"}, pytest.raises( SchemaValidationError, - match=( - "manage_etc_hosts: 'templatey' is not valid under any" - " of the given schemas" + match=re.escape( + "manage_etc_hosts: 'templatey' is not one of" + " ['template']", ), ), ), diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_users_groups.py cloud-init-23.4.3/tests/unittests/config/test_cc_users_groups.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_users_groups.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_users_groups.py 2024-02-02 21:21:52.000000000 +0000 @@ -337,7 +337,10 @@ {"users": [{"name": "bbsw", "garbage-key": None}]}, pytest.raises( SchemaValidationError, - match="is not valid under any of the given schemas", + match=( + "users.0: {'name': 'bbsw', 'garbage-key': None} is" + " not of type 'string'" + ), ), True, ), @@ -407,7 +410,9 @@ {"user": ["no_list_allowed"]}, pytest.raises( SchemaValidationError, - match=re.escape("user: ['no_list_allowed'] is not valid "), + match=re.escape( + "user: ['no_list_allowed'] is not of type 'string'" + ), ), True, ), @@ -425,7 +430,10 @@ }, pytest.raises( SchemaValidationError, - match="errors: users.0: {'inactive': True", + match=( + "errors: users.0.inactive: True is not of type" + " 'string'" + ), ), True, ), @@ -491,10 +499,7 @@ {"users": [{"name": "a", "expiredate": "2038,1,19"}]}, pytest.raises( SchemaValidationError, - match=( - "users.0: {'name': 'a', 'expiredate': '2038,1,19'}" - " is not valid under any of the given schemas" - ), + match=("users.0.expiredate: '2038,1,19' is not a 'date'"), ), True, ), diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_write_files.py cloud-init-23.4.3/tests/unittests/config/test_cc_write_files.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_write_files.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_write_files.py 2024-02-02 21:21:52.000000000 +0000 @@ -3,13 +3,13 @@ import base64 import gzip import io +import logging import re import shutil import tempfile import pytest -from cloudinit import log as logging from cloudinit import util from cloudinit.config.cc_write_files import decode_perms, handle, write_files from cloudinit.config.schema import ( @@ -264,6 +264,3 @@ validate_cloudconfig_schema(config, get_schema(), strict=True) else: validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_write_files_deferred.py cloud-init-23.4.3/tests/unittests/config/test_cc_write_files_deferred.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_write_files_deferred.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_write_files_deferred.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,11 +1,11 @@ # This file is part of cloud-init. See LICENSE file for license information. +import logging import shutil import tempfile import pytest -from cloudinit import log as logging from cloudinit import util from cloudinit.config.cc_write_files_deferred import handle from cloudinit.config.schema import ( @@ -65,6 +65,3 @@ def test_schema_validation(self, config, error_msg): with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_cc_yum_add_repo.py cloud-init-23.4.3/tests/unittests/config/test_cc_yum_add_repo.py --- cloud-init-23.3.3/tests/unittests/config/test_cc_yum_add_repo.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_cc_yum_add_repo.py 2024-02-02 21:21:52.000000000 +0000 @@ -178,6 +178,3 @@ else: with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_schema(config, get_schema(), strict=True) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/config/test_schema.py cloud-init-23.4.3/tests/unittests/config/test_schema.py --- cloud-init-23.3.3/tests/unittests/config/test_schema.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/config/test_schema.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,6 +1,5 @@ # This file is part of cloud-init. See LICENSE file for license information. - import importlib import inspect import itertools @@ -12,6 +11,7 @@ import unittest from collections import namedtuple from copy import deepcopy +from errno import EACCES from pathlib import Path from textwrap import dedent from types import ModuleType @@ -19,7 +19,6 @@ import pytest -from cloudinit import log from cloudinit.config.schema import ( VERSIONED_USERDATA_SCHEMA_FILE, MetaSchema, @@ -40,6 +39,7 @@ from cloudinit.distros import OSFAMILIES from cloudinit.safeyaml import load, load_with_marks from cloudinit.settings import FREQUENCIES +from cloudinit.sources import DataSourceNotFoundException from cloudinit.util import load_file, write_file from tests.hypothesis import given from tests.hypothesis_jsonschema import from_schema @@ -113,8 +113,11 @@ ( ({}, None), ({"version": "v1"}, None), - ({"version": "v2"}, "is not valid"), - ({"version": "v1", "final_message": -1}, "is not valid"), + ({"version": "v2"}, "is not one of ['v1']"), + ( + {"version": "v1", "final_message": -1}, + "is not of type 'string'", + ), ({"version": "v1", "final_message": "some msg"}, None), ), ) @@ -210,7 +213,13 @@ [meta["id"] for meta in get_metas().values() if meta is not None] ) assert "http://json-schema.org/draft-04/schema#" == schema["$schema"] - assert ["$defs", "$schema", "allOf"] == sorted(list(schema.keys())) + assert [ + "$defs", + "$schema", + "additionalProperties", + "allOf", + "properties", + ] == sorted(list(schema.keys())) # New style schema should be defined in static schema file in $defs expected_subschema_defs = [ {"$ref": "#/$defs/base_config"}, @@ -269,6 +278,7 @@ {"$ref": "#/$defs/cc_yum_add_repo"}, {"$ref": "#/$defs/cc_zypper_add_repo"}, {"$ref": "#/$defs/reporting_config"}, + {"$ref": "#/$defs/output_config"}, ] found_subschema_defs = [] legacy_schema_keys = [] @@ -644,7 +654,6 @@ def test_validateconfig_logs_deprecations( self, schema, config, expected_msg, log_deprecations, caplog ): - log.setupLogging() validate_cloudconfig_schema( config, schema, @@ -740,7 +749,7 @@ config_file = tmpdir.join("my.yaml") config_file.write("#cloud-config\nasdf:\nasdf") error_msg = ( - f".*errors: format-l3.c1: File {config_file} is not valid yaml.*" + f".*errors: format-l3.c1: File {config_file} is not valid YAML.*" ) with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_file(config_file.strpath, {}, annotate) @@ -753,7 +762,7 @@ config_file = tmpdir.join("my.yaml") config_file.write("#cloud-config\n{}}") error_msg = ( - f"errors: format-l2.c3: File {config_file} is not valid yaml." + f"errors: format-l2.c3: File {config_file} is not valid YAML." ) with pytest.raises(SchemaValidationError, match=error_msg): validate_cloudconfig_file(config_file.strpath, {}, annotate) @@ -808,14 +817,14 @@ cloud_config_file = paths.get_ipath_cur("cloud_config") write_file(cloud_config_file, b"") - error_msg = ( - "Cloud config schema errors: format-l1.c1: " - f'Unrecognized user-data header in {cloud_config_file}: "".\n' - "Expected first line to be one of: #!, ## template: jinja," - " #cloud-boothook, #cloud-config," + validate_cloudconfig_file( + config_path=cloud_config_file, schema=schema, annotate=annotate + ) + out, _err = capsys.readouterr() + assert ( + f"Empty 'cloud-config' found at {cloud_config_file}." + " Nothing to validate" in out ) - with pytest.raises(SchemaValidationError, match=error_msg): - validate_cloudconfig_file(cloud_config_file, schema, annotate) class TestSchemaDocMarkdown: @@ -1625,6 +1634,29 @@ schema_errors=schema_errors, ) + @skipUnlessJsonSchema() + def test_annotated_invalid_top_level_key(self, tmp_path: Path, capsys): + expected_err = dedent( + """\ + #cloud-config + invalid_key: value # E1 + + # Errors: ------------- + # E1: Additional properties are not allowed ('invalid_key' was unexpected) + """ # noqa: E501 + ) + config_file = tmp_path / "my.yaml" + config_file.write_text("#cloud-config\ninvalid_key: value\n") + with pytest.raises( + SchemaValidationError, + match="errors: invalid_key: Additional properties are not allowed", + ): + validate_cloudconfig_file( + str(config_file), get_schema(), annotate=True + ) + out, _err = capsys.readouterr() + assert out.strip() == expected_err.strip() + @mock.patch(M_PATH + "read_cfg_paths") # called by parse_args help docs class TestMain: @@ -1648,6 +1680,31 @@ ) assert expected == err + @pytest.mark.parametrize( + "params,expectation", + [ + pytest.param(["--docs all"], does_not_raise()), + pytest.param(["--system"], pytest.raises(SystemExit)), + ], + ) + @mock.patch(M_PATH + "os.getuid", return_value=100) + def test_main_ignores_schema_type( + self, _read_cfg_paths, _os_getuid, params, expectation, capsys + ): + """Main ignores --schema-type param when --system or --docs present.""" + params = list(itertools.chain(*[a.split() for a in params])) + with mock.patch( + "sys.argv", ["mycmd", "--schema-type", "network-config"] + params + ): + with expectation: + main() + out, _err = capsys.readouterr() + expected = ( + "WARNING: The --schema-type parameter is inapplicable when" + " either --system or --docs present" + ) + assert expected in out + def test_main_missing_args(self, _read_cfg_paths, capsys): """Main exits non-zero and reports an error on missing parameters.""" with mock.patch("sys.argv", ["mycmd"]): @@ -1702,14 +1759,55 @@ with mock.patch("sys.argv", myargs): assert 0 == main(), "Expected 0 exit code" out, _err = capsys.readouterr() - assert f"Valid cloud-config: {myyaml}\n" == out + assert f"Valid schema {myyaml}\n" == out + @pytest.mark.parametrize( + "net_config,net_output,error_raised", + ( + pytest.param( + "network:\n version: 1\n config:\n - type: physical\n" + " name: eth0\n subnets:\n - type: dhcp\n", + " Valid schema network-config", + does_not_raise(), + id="netv1_schema_validated", + ), + pytest.param( + "network:\n version: 2\n ethernets:\n eth0:\n dhcp4:true\n", + "Skipping network-config schema validation." + " No network schema for version: 2", + does_not_raise(), + id="netv2_validation_is_skipped", + ), + pytest.param( + "network:\n", + "Skipping network-config schema validation on empty config.", + does_not_raise(), + id="empty_net_validation_is_skipped", + ), + pytest.param( + "network:\n version: 1\n config:\n - type: physical\n" + " name: eth0\n subnets:\n - type: dhcp\n", + " Invalid network-config {network_file}", + pytest.raises(SystemExit), + id="netv1_schema_errors_handled", + ), + ), + ) @mock.patch(M_PATH + "read_cfg_paths") @mock.patch(M_PATH + "os.getuid", return_value=0) - def test_main_validates_system_userdata_and_vendordata( - self, _read_cfg_paths, _getuid, read_cfg_paths, capsys, mocker, paths + def test_main_validates_system_userdata_vendordata_and_network_config( + self, + _read_cfg_paths, + _getuid, + read_cfg_paths, + net_config, + net_output, + error_raised, + capsys, + mocker, + paths, ): - """When --system is provided, main validates system userdata.""" + """When --system is provided, main validates all config userdata.""" paths.get_ipath = paths.get_ipath_cur read_cfg_paths.return_value = paths cloud_config_file = paths.get_ipath_cur("cloud_config") @@ -1717,32 +1815,35 @@ vd_file = paths.get_ipath_cur("vendor_cloud_config") write_file(vd_file, b"#cloud-config\nssh_import_id: [me]") vd2_file = paths.get_ipath_cur("vendor2_cloud_config") - write_file(vd2_file, b"#cloud-config\nssh_pw_auth: true") + write_file(vd2_file, b"#cloud-config\nssh_pwauth: true") + network_file = paths.get_ipath_cur("network_config") + write_file(network_file, net_config) myargs = ["mycmd", "--system"] - with mock.patch("sys.argv", myargs): - main() + with error_raised: + with mock.patch("sys.argv", myargs): + main() out, _err = capsys.readouterr() + net_output = net_output.format(network_file=network_file) + data_types = "user-data, vendor-data, vendor2-data, network-config" expected = dedent( - """\ - Found cloud-config data types: user-data, vendor-data, vendor2-data + f"""\ + Found cloud-config data types: {data_types} - 1. user-data at {ud_file}: - Valid cloud-config: user-data + 1. user-data at {cloud_config_file}: + Valid schema user-data 2. vendor-data at {vd_file}: - Valid cloud-config: vendor-data + Valid schema vendor-data 3. vendor2-data at {vd2_file}: - Valid cloud-config: vendor2-data + Valid schema vendor2-data + + 4. network-config at {network_file}: + {net_output} """ ) - assert ( - expected.format( - ud_file=cloud_config_file, vd_file=vd_file, vd2_file=vd2_file - ) - == out - ) + assert expected == out @mock.patch(M_PATH + "os.getuid", return_value=1000) def test_main_system_userdata_requires_root( @@ -1762,9 +1863,12 @@ assert expected == err -def _get_meta_doc_examples( - file_glob="cloud-config*.txt", exclusion_match=r"^cloud-config-archive.*" -): +def _get_meta_doc_examples(file_glob="cloud-config*.txt"): + exlusion_patterns = [ + "^cloud-config-archive.*", + "cloud-config-datasources.txt", + ] + exclusion_match = f"({'|'.join(exlusion_patterns)})" examples_dir = Path(cloud_init_project_dir("doc/examples")) assert examples_dir.is_dir() return ( @@ -1855,7 +1959,7 @@ "src_config, expectation", ( pytest.param( - {"network": {"version": 2}}, + {"network": {"config": [], "version": 2}}, pytest.raises( SchemaValidationError, match=re.escape("network.version: 2 is not one of [1]"), @@ -1876,12 +1980,17 @@ id="config_key_required", ), pytest.param( - {"network": {"version": 1, "config": [{"type": "typo"}]}}, + { + "network": { + "version": 1, + "config": [{"name": "me", "type": "typo"}], + } + }, pytest.raises( SchemaValidationError, match=( - r"network.config.0: {'type': 'typo'} is not valid " - "under any of the given schemas" + r"network.config.0: {'name': 'me', 'type': 'typo'} is" + " not valid under any of the given schemas" ), ), id="unknown_config_type_item", @@ -1944,7 +2053,10 @@ def test_network_schema(self, src_config, expectation): with expectation: validate_cloudconfig_schema( - config=src_config, schema=self.net_schema, strict=True + config=src_config, + schema=self.net_schema, + schema_type="netork-config", + strict=True, ) @@ -2028,6 +2140,8 @@ remove_modules(schema, set(modules)) if defs: remove_defs(schema, set(defs)) + del schema["properties"] + del schema["additionalProperties"] return schema @@ -2041,7 +2155,12 @@ @skipUnlessHypothesisJsonSchema() @given(from_schema(SCHEMA)) - def test_validate_full_schema(self, config): + def test_validate_full_schema(self, orig_config): + config = deepcopy(orig_config) + valid_props = get_schema()["properties"].keys() + for key in orig_config.keys(): + if key not in valid_props: + del config[key] try: validate_cloudconfig_schema(config, strict=True) except SchemaValidationError as ex: @@ -2050,7 +2169,59 @@ class TestHandleSchemaArgs: - Args = namedtuple("Args", "config_file docs system annotate instance_data") + Args = namedtuple( + "Args", "config_file schema_type docs system annotate instance_data" + ) + + @pytest.mark.parametrize( + "failure, expected_logs", + ( + ( + IOError("No permissions on /var/lib/cloud/instance"), + ["Using default instance-data/user-data paths for non-root"], + ), + ( + DataSourceNotFoundException("No cached datasource found yet"), + ["datasource not detected"], + ), + ), + ) + @mock.patch(M_PATH + "read_cfg_paths") + def test_handle_schema_unable_to_read_cfg_paths( + self, + read_cfg_paths, + failure, + expected_logs, + paths, + capsys, + caplog, + tmpdir, + ): + if isinstance(failure, IOError): + failure.errno = EACCES + read_cfg_paths.side_effect = [failure, paths] + user_data_fn = tmpdir.join("user-data") + with open(user_data_fn, "w") as f: + f.write( + dedent( + """\ + #cloud-config + packages: [sl] + """ + ) + ) + args = self.Args( + config_file=str(user_data_fn), + schema_type="cloud-config", + annotate=False, + docs=None, + system=None, + instance_data=None, + ) + handle_schema_args("unused", args) + assert "Valid schema" in capsys.readouterr().out + for expected_log in expected_logs: + assert expected_log in caplog.text @pytest.mark.parametrize( "annotate, expected_output", @@ -2071,7 +2242,7 @@ # D2: Default: ``false``. Deprecated in version 22.2. Use ``package_upgrade`` instead. # D3: Default: ``false``. Deprecated in version 22.2. Use ``package_reboot_if_required`` instead. - Valid cloud-config: {cfg_file} + Valid schema {cfg_file} """ # noqa: E501 ), ), @@ -2085,7 +2256,7 @@ ``false``. Deprecated in version 22.2. Use ``package_update`` instead.,\ apt_upgrade: Default: ``false``. Deprecated in version 22.2. Use \ ``package_upgrade`` instead.\ - Valid cloud-config: {cfg_file} + Valid schema {cfg_file} """ # noqa: E501 ), ), @@ -2120,6 +2291,7 @@ ) args = self.Args( config_file=str(user_data_fn), + schema_type="cloud-config", annotate=annotate, docs=None, system=None, @@ -2160,14 +2332,14 @@ False, dedent( """\ - Invalid cloud-config {cfg_file} + Invalid user-data {cfg_file} """ # noqa: E501 ), dedent( """\ Error: Cloud config schema errors: hostname: 123 is not of type 'string' - Error: Invalid cloud-config schema: user-data + Error: Invalid schema: user-data """ # noqa: E501 ), @@ -2214,6 +2386,87 @@ ) args = self.Args( config_file=str(user_data_fn), + schema_type="cloud-config", + annotate=annotate, + docs=None, + system=None, + instance_data=None, + ) + with expectation: + handle_schema_args("unused", args) + out, err = capsys.readouterr() + assert ( + expected_out.format(cfg_file=user_data_fn, id_path=id_path) == out + ) + assert ( + expected_err.format(cfg_file=user_data_fn, id_path=id_path) == err + ) + assert "deprec" not in caplog.text + assert read_cfg_paths.call_args_list == [ + mock.call(fetch_existing_datasource="trust") + ] + + @pytest.mark.parametrize( + "uid, annotate, expected_out, expected_err, expectation", + [ + pytest.param( + 0, + False, + dedent( + """\ + Invalid UNKNOWN_CONFIG_HEADER {cfg_file} + """ # noqa: E501 + ), + dedent( + """\ + Error: Cloud config schema errors: format-l1.c1: Unrecognized user-data header in {cfg_file}: "#bogus-config". + Expected first line to be one of: #!, ## template: jinja, #cloud-boothook, #cloud-config, #cloud-config-archive, #cloud-config-jsonp, #include, #include-once, #part-handler + + Error: Invalid schema: UNKNOWN_CONFIG_HEADER + + """ # noqa: E501 + ), + pytest.raises(SystemExit), + id="root_no_annotate_exception_with_unique_errors", + ), + ], + ) + @mock.patch(M_PATH + "os.getuid") + @mock.patch(M_PATH + "read_cfg_paths") + def test_handle_schema_args_unknown_header( + self, + read_cfg_paths, + getuid, + uid, + annotate, + expected_out, + expected_err, + expectation, + paths, + caplog, + capsys, + tmpdir, + ): + getuid.return_value = uid + paths.get_ipath = paths.get_ipath_cur + read_cfg_paths.return_value = paths + user_data_fn = tmpdir.join("user-data") + if uid == 0: + id_path = paths.get_runpath("instance_data_sensitive") + else: + id_path = paths.get_runpath("instance_data") + with open(user_data_fn, "w") as f: + f.write( + dedent( + """\ + #bogus-config + hostname: notgonnamakeit + """ + ) + ) + args = self.Args( + config_file=str(user_data_fn), + schema_type=None, annotate=annotate, docs=None, system=None, diff -Nru cloud-init-23.3.3/tests/unittests/conftest.py cloud-init-23.4.3/tests/unittests/conftest.py --- cloud-init-23.3.3/tests/unittests/conftest.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/conftest.py 2024-02-02 21:21:52.000000000 +0000 @@ -6,7 +6,7 @@ import pytest -from cloudinit import atomic_helper, util +from cloudinit import atomic_helper, log, util from tests.hypothesis import HAS_HYPOTHESIS from tests.unittests.helpers import retarget_many_wrapper @@ -76,6 +76,15 @@ yield +log.configure_root_logger() + + +@pytest.fixture(autouse=True) +def disable_root_logger_setup(request): + with mock.patch("cloudinit.cmd.main.configure_root_logger", autospec=True): + yield + + PYTEST_VERSION_TUPLE = tuple(map(int, pytest.__version__.split("."))) if PYTEST_VERSION_TUPLE < (3, 9, 0): diff -Nru cloud-init-23.3.3/tests/unittests/distros/package_management/test_apt.py cloud-init-23.4.3/tests/unittests/distros/package_management/test_apt.py --- cloud-init-23.3.3/tests/unittests/distros/package_management/test_apt.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/package_management/test_apt.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,88 @@ +# This file is part of cloud-init. See LICENSE file for license information. +from itertools import count, cycle +from unittest import mock + +import pytest + +from cloudinit import subp +from cloudinit.distros.package_management.apt import APT_GET_COMMAND, Apt + + +@mock.patch.dict("os.environ", {}, clear=True) +@mock.patch("cloudinit.distros.debian.subp.which", return_value=True) +@mock.patch("cloudinit.distros.debian.subp.subp") +class TestPackageCommand: + @mock.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + return_value=True, + ) + def test_simple_command(self, m_apt_avail, m_subp, m_which): + apt = Apt(runner=mock.Mock(), apt_get_wrapper_command=["eatmydata"]) + apt.run_package_command("update") + expected_call = { + "args": ["eatmydata"] + list(APT_GET_COMMAND) + ["update"], + "capture": False, + "env": {"DEBIAN_FRONTEND": "noninteractive"}, + } + assert m_subp.call_args == mock.call(**expected_call) + + @mock.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + side_effect=[False, False, True], + ) + @mock.patch("cloudinit.distros.package_management.apt.time.sleep") + def test_wait_for_lock(self, m_sleep, m_apt_avail, m_subp, m_which): + apt = Apt(runner=mock.Mock(), apt_get_wrapper_command=("dontcare",)) + apt._wait_for_apt_command("stub", {"args": "stub2"}) + assert m_sleep.call_args_list == [mock.call(1), mock.call(1)] + assert m_subp.call_args_list == [mock.call(args="stub2")] + + @mock.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + return_value=False, + ) + @mock.patch("cloudinit.distros.package_management.apt.time.sleep") + @mock.patch( + "cloudinit.distros.package_management.apt.time.time", + side_effect=count(), + ) + def test_lock_wait_timeout( + self, m_time, m_sleep, m_apt_avail, m_subp, m_which + ): + apt = Apt(runner=mock.Mock(), apt_get_wrapper_command=("dontcare",)) + with pytest.raises(TimeoutError): + apt._wait_for_apt_command("stub", "stub2", timeout=5) + assert m_subp.call_args_list == [] + + @mock.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + side_effect=cycle([True, False]), + ) + @mock.patch("cloudinit.distros.package_management.apt.time.sleep") + def test_lock_exception_wait(self, m_sleep, m_apt_avail, m_subp, m_which): + apt = Apt(runner=mock.Mock(), apt_get_wrapper_command=("dontcare",)) + exception = subp.ProcessExecutionError( + exit_code=100, stderr="Could not get apt lock" + ) + m_subp.side_effect = [exception, exception, "return_thing"] + ret = apt._wait_for_apt_command("stub", {"args": "stub2"}) + assert ret == "return_thing" + + @mock.patch( + "cloudinit.distros.package_management.apt.Apt._apt_lock_available", + side_effect=cycle([True, False]), + ) + @mock.patch("cloudinit.distros.package_management.apt.time.sleep") + @mock.patch( + "cloudinit.distros.package_management.apt.time.time", + side_effect=count(), + ) + def test_lock_exception_timeout( + self, m_time, m_sleep, m_apt_avail, m_subp, m_which + ): + apt = Apt(runner=mock.Mock(), apt_get_wrapper_command=("dontcare",)) + m_subp.side_effect = subp.ProcessExecutionError( + exit_code=100, stderr="Could not get apt lock" + ) + with pytest.raises(TimeoutError): + apt._wait_for_apt_command("stub", {"args": "stub2"}, timeout=5) diff -Nru cloud-init-23.3.3/tests/unittests/distros/test__init__.py cloud-init-23.4.3/tests/unittests/distros/test__init__.py --- cloud-init-23.3.3/tests/unittests/distros/test__init__.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test__init__.py 2024-02-02 21:21:52.000000000 +0000 @@ -501,6 +501,3 @@ assert "/tmp" == tmp_path else: assert "/usr_lib_exec/cloud-init/clouddir" == tmp_path - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_debian.py cloud-init-23.4.3/tests/unittests/distros/test_debian.py --- cloud-init-23.3.3/tests/unittests/distros/test_debian.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_debian.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,211 +1,125 @@ # This file is part of cloud-init. See LICENSE file for license information. -from itertools import count, cycle -from unittest import mock - import pytest -from cloudinit import distros, subp, util -from cloudinit.distros.debian import APT_GET_COMMAND, APT_GET_WRAPPER -from tests.unittests.helpers import FilesystemMockingTestCase - - -@mock.patch("cloudinit.distros.debian.subp.subp") -class TestDebianApplyLocale(FilesystemMockingTestCase): - def setUp(self): - super(TestDebianApplyLocale, self).setUp() - self.new_root = self.tmp_dir() - self.patchOS(self.new_root) - self.patchUtils(self.new_root) - self.spath = self.tmp_path("etc/default/locale", self.new_root) - cls = distros.fetch("debian") - self.distro = cls("debian", {}, None) +from cloudinit import util +from tests.unittests.util import get_cloud + +LOCALE_PATH = "/etc/default/locale" + + +@pytest.mark.usefixtures("fake_filesystem") +class TestDebianApplyLocale: + @pytest.fixture + def m_subp(self, mocker): + yield mocker.patch( + "cloudinit.distros.debian.subp.subp", return_value=(None, None) + ) + + @pytest.fixture + def distro(self): + yield get_cloud(distro="debian").distro - def test_no_rerun(self, m_subp): + def test_no_rerun(self, distro, m_subp): """If system has defined locale, no re-run is expected.""" - m_subp.return_value = (None, None) locale = "en_US.UTF-8" - util.write_file(self.spath, "LANG=%s\n" % locale, omode="w") - self.distro.apply_locale(locale, out_fn=self.spath) + util.write_file(LOCALE_PATH, "LANG=%s\n" % locale, omode="w") + distro.apply_locale(locale, out_fn=LOCALE_PATH) m_subp.assert_not_called() - def test_no_regen_on_c_utf8(self, m_subp): + def test_no_regen_on_c_utf8(self, distro, m_subp): """If locale is set to C.UTF8, do not attempt to call locale-gen""" m_subp.return_value = (None, None) locale = "C.UTF-8" - util.write_file(self.spath, "LANG=%s\n" % "en_US.UTF-8", omode="w") - self.distro.apply_locale(locale, out_fn=self.spath) - self.assertEqual( - [ - [ - "update-locale", - "--locale-file=" + self.spath, - "LANG=%s" % locale, - ] - ], - [p[0][0] for p in m_subp.call_args_list], - ) + util.write_file(LOCALE_PATH, "LANG=%s\n" % "en_US.UTF-8", omode="w") + distro.apply_locale(locale, out_fn=LOCALE_PATH) + assert [ + ["update-locale", f"--locale-file={LOCALE_PATH}", f"LANG={locale}"] + ] == [p[0][0] for p in m_subp.call_args_list] - def test_rerun_if_different(self, m_subp): + def test_rerun_if_different(self, distro, m_subp, caplog): """If system has different locale, locale-gen should be called.""" - m_subp.return_value = (None, None) locale = "en_US.UTF-8" - util.write_file(self.spath, "LANG=fr_FR.UTF-8", omode="w") - self.distro.apply_locale(locale, out_fn=self.spath) - self.assertEqual( + util.write_file(LOCALE_PATH, "LANG=fr_FR.UTF-8", omode="w") + distro.apply_locale(locale, out_fn=LOCALE_PATH) + assert [ + ["locale-gen", locale], [ - ["locale-gen", locale], - [ - "update-locale", - "--locale-file=" + self.spath, - "LANG=%s" % locale, - ], + "update-locale", + f"--locale-file={LOCALE_PATH}", + f"LANG={locale}", ], - [p[0][0] for p in m_subp.call_args_list], + ] == [p[0][0] for p in m_subp.call_args_list] + assert ( + "System locale set to fr_FR.UTF-8 via /etc/default/locale" + in caplog.text + ) + assert "Generating locales for en_US.UTF-8" in caplog.text + assert ( + "Updating /etc/default/locale with locale setting LANG=en_US.UTF-8" + in caplog.text ) - def test_rerun_if_no_file(self, m_subp): + def test_rerun_if_no_file(self, distro, m_subp): """If system has no locale file, locale-gen should be called.""" - m_subp.return_value = (None, None) locale = "en_US.UTF-8" - self.distro.apply_locale(locale, out_fn=self.spath) - self.assertEqual( + distro.apply_locale(locale, out_fn=LOCALE_PATH) + assert [ + ["locale-gen", locale], [ - ["locale-gen", locale], - [ - "update-locale", - "--locale-file=" + self.spath, - "LANG=%s" % locale, - ], + "update-locale", + f"--locale-file={LOCALE_PATH}", + f"LANG={locale}", ], - [p[0][0] for p in m_subp.call_args_list], - ) + ] == [p[0][0] for p in m_subp.call_args_list] - def test_rerun_on_unset_system_locale(self, m_subp): + def test_rerun_on_unset_system_locale(self, distro, m_subp, caplog): """If system has unset locale, locale-gen should be called.""" - m_subp.return_value = (None, None) locale = "en_US.UTF-8" - util.write_file(self.spath, "LANG=", omode="w") - self.distro.apply_locale(locale, out_fn=self.spath) - self.assertEqual( + util.write_file(LOCALE_PATH, "LANG=", omode="w") + distro.apply_locale(locale, out_fn=LOCALE_PATH) + assert [ + ["locale-gen", locale], [ - ["locale-gen", locale], - [ - "update-locale", - "--locale-file=" + self.spath, - "LANG=%s" % locale, - ], + "update-locale", + f"--locale-file={LOCALE_PATH}", + f"LANG={locale}", ], - [p[0][0] for p in m_subp.call_args_list], + ] == [p[0][0] for p in m_subp.call_args_list] + assert ( + "System locale not found in /etc/default/locale. Assuming system " + "locale is C.UTF-8 based on hardcoded default" in caplog.text + ) + assert "Generating locales for en_US.UTF-8" in caplog.text + assert ( + "Updating /etc/default/locale with locale setting LANG=en_US.UTF-8" + in caplog.text ) - def test_rerun_on_mismatched_keys(self, m_subp): + def test_rerun_on_mismatched_keys(self, distro, m_subp): """If key is LC_ALL and system has only LANG, rerun is expected.""" - m_subp.return_value = (None, None) locale = "en_US.UTF-8" - util.write_file(self.spath, "LANG=", omode="w") - self.distro.apply_locale(locale, out_fn=self.spath, keyname="LC_ALL") - self.assertEqual( + util.write_file(LOCALE_PATH, "LANG=", omode="w") + distro.apply_locale(locale, out_fn=LOCALE_PATH, keyname="LC_ALL") + assert [ + ["locale-gen", locale], [ - ["locale-gen", locale], - [ - "update-locale", - "--locale-file=" + self.spath, - "LC_ALL=%s" % locale, - ], + "update-locale", + f"--locale-file={LOCALE_PATH}", + f"LC_ALL={locale}", ], - [p[0][0] for p in m_subp.call_args_list], - ) + ] == [p[0][0] for p in m_subp.call_args_list] - def test_falseish_locale_raises_valueerror(self, m_subp): + def test_falseish_locale_raises_valueerror(self, distro, m_subp): """locale as None or "" is invalid and should raise ValueError.""" - with self.assertRaises(ValueError) as ctext_m: - self.distro.apply_locale(None) + with pytest.raises( + ValueError, match="Failed to provide locale value." + ): + distro.apply_locale(None) m_subp.assert_not_called() - self.assertEqual( - "Failed to provide locale value.", str(ctext_m.exception) - ) - - with self.assertRaises(ValueError) as ctext_m: - self.distro.apply_locale("") + with pytest.raises( + ValueError, match="Failed to provide locale value." + ): + distro.apply_locale("") m_subp.assert_not_called() - self.assertEqual( - "Failed to provide locale value.", str(ctext_m.exception) - ) - - -@mock.patch.dict("os.environ", {}, clear=True) -@mock.patch("cloudinit.distros.debian.subp.which", return_value=True) -@mock.patch("cloudinit.distros.debian.subp.subp") -class TestPackageCommand: - distro = distros.fetch("debian")("debian", {}, None) - - @mock.patch( - "cloudinit.distros.debian.Distro._apt_lock_available", - return_value=True, - ) - def test_simple_command(self, m_apt_avail, m_subp, m_which): - self.distro.package_command("update") - apt_args = [APT_GET_WRAPPER["command"]] - apt_args.extend(APT_GET_COMMAND) - apt_args.append("update") - expected_call = { - "args": apt_args, - "capture": False, - "env": {"DEBIAN_FRONTEND": "noninteractive"}, - } - assert m_subp.call_args == mock.call(**expected_call) - - @mock.patch( - "cloudinit.distros.debian.Distro._apt_lock_available", - side_effect=[False, False, True], - ) - @mock.patch("cloudinit.distros.debian.time.sleep") - def test_wait_for_lock(self, m_sleep, m_apt_avail, m_subp, m_which): - self.distro._wait_for_apt_command("stub", {"args": "stub2"}) - assert m_sleep.call_args_list == [mock.call(1), mock.call(1)] - assert m_subp.call_args_list == [mock.call(args="stub2")] - - @mock.patch( - "cloudinit.distros.debian.Distro._apt_lock_available", - return_value=False, - ) - @mock.patch("cloudinit.distros.debian.time.sleep") - @mock.patch("cloudinit.distros.debian.time.time", side_effect=count()) - def test_lock_wait_timeout( - self, m_time, m_sleep, m_apt_avail, m_subp, m_which - ): - with pytest.raises(TimeoutError): - self.distro._wait_for_apt_command("stub", "stub2", timeout=5) - assert m_subp.call_args_list == [] - - @mock.patch( - "cloudinit.distros.debian.Distro._apt_lock_available", - side_effect=cycle([True, False]), - ) - @mock.patch("cloudinit.distros.debian.time.sleep") - def test_lock_exception_wait(self, m_sleep, m_apt_avail, m_subp, m_which): - exception = subp.ProcessExecutionError( - exit_code=100, stderr="Could not get apt lock" - ) - m_subp.side_effect = [exception, exception, "return_thing"] - ret = self.distro._wait_for_apt_command("stub", {"args": "stub2"}) - assert ret == "return_thing" - - @mock.patch( - "cloudinit.distros.debian.Distro._apt_lock_available", - side_effect=cycle([True, False]), - ) - @mock.patch("cloudinit.distros.debian.time.sleep") - @mock.patch("cloudinit.distros.debian.time.time", side_effect=count()) - def test_lock_exception_timeout( - self, m_time, m_sleep, m_apt_avail, m_subp, m_which - ): - m_subp.side_effect = subp.ProcessExecutionError( - exit_code=100, stderr="Could not get apt lock" - ) - with pytest.raises(TimeoutError): - self.distro._wait_for_apt_command( - "stub", {"args": "stub2"}, timeout=5 - ) diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_hostname.py cloud-init-23.4.3/tests/unittests/distros/test_hostname.py --- cloud-init-23.3.3/tests/unittests/distros/test_hostname.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_hostname.py 2024-02-02 21:21:52.000000000 +0000 @@ -37,6 +37,3 @@ bbbbd """ self.assertEqual(str(hn).strip(), expected_out.strip()) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_hosts.py cloud-init-23.4.3/tests/unittests/distros/test_hosts.py --- cloud-init-23.3.3/tests/unittests/distros/test_hosts.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_hosts.py 2024-02-02 21:21:52.000000000 +0000 @@ -42,6 +42,3 @@ eh.del_entries("127.0.0.0") self.assertEqual(eh.get_entry("127.0.0.0"), []) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_init.py cloud-init-23.4.3/tests/unittests/distros/test_init.py --- cloud-init-23.3.3/tests/unittests/distros/test_init.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_init.py 2024-02-02 21:21:52.000000000 +0000 @@ -9,7 +9,12 @@ import pytest -from cloudinit.distros import LDH_ASCII_CHARS, _get_package_mirror_info +from cloudinit.distros import ( + LDH_ASCII_CHARS, + PackageInstallerError, + _get_package_mirror_info, +) +from tests.unittests.distros import _get_distro # In newer versions of Python, these characters will be omitted instead # of substituted because of security concerns. @@ -246,3 +251,91 @@ print(patterns) print(expected) assert {"primary": expected} == ret + + +class TestInstall: + """Tests for cloudinit.distros.Distro.install_packages.""" + + @pytest.fixture + def m_apt_install(self, mocker): + return mocker.patch( + "cloudinit.distros.package_management.apt.Apt.install_packages", + return_value=[], + ) + + @pytest.fixture + def m_snap_install(self, mocker): + return mocker.patch( + "cloudinit.distros.package_management.snap.Snap.install_packages", + return_value=[], + ) + + @pytest.fixture + def m_subp(self, mocker): + return mocker.patch( + "cloudinit.distros.bsd.subp.subp", return_value=("", "") + ) + + def test_invalid_yaml(self, m_apt_install): + """Test that an invalid YAML raises an exception.""" + with pytest.raises(ValueError): + _get_distro("debian").install_packages([["invalid"]]) + m_apt_install.assert_not_called() + + def test_unknown_package_manager(self, m_apt_install, caplog): + """Test that an unknown package manager raises an exception.""" + _get_distro("debian").install_packages( + [{"apt": ["pkg1"]}, "pkg2", {"invalid": ["pkg3"]}] + ) + assert ( + "Cannot install packages under 'invalid' as it is not a supported " + "package manager!" in caplog.text + ) + install_args = m_apt_install.call_args_list[0][0][0] + assert "pkg1" in install_args + assert "pkg2" in install_args + assert "pkg3" not in install_args + + def test_non_default_package_manager(self, m_apt_install, m_snap_install): + """Test success from package manager not supported by distro.""" + _get_distro("debian").install_packages( + [{"apt": ["pkg1"]}, "pkg2", {"snap": ["pkg3"]}] + ) + apt_install_args = m_apt_install.call_args_list[0][0][0] + assert "pkg1" in apt_install_args + assert "pkg2" in apt_install_args + assert "pkg3" not in apt_install_args + + assert "pkg3" in m_snap_install.call_args_list[0][1]["pkglist"] + + def test_non_default_package_manager_fail( + self, m_apt_install, mocker, caplog + ): + """Test fail from package manager not supported by distro.""" + m_snap_install = mocker.patch( + "cloudinit.distros.package_management.snap.Snap.install_packages", + return_value=["pkg3"], + ) + with pytest.raises( + PackageInstallerError, + match="Failed to install the following packages: \\['pkg3'\\]", + ): + _get_distro("debian").install_packages( + [{"apt": ["pkg1"]}, "pkg2", {"snap": ["pkg3"]}] + ) + + assert "pkg3" in m_snap_install.call_args_list[0][1]["pkglist"] + + def test_default_and_specific_package_manager( + self, m_apt_install, m_snap_install + ): + """Test success from package manager not supported by distro.""" + _get_distro("ubuntu").install_packages( + ["pkg1", ["pkg3", "ver3"], {"apt": [["pkg2", "ver2"]]}] + ) + apt_install_args = m_apt_install.call_args_list[0][0][0] + assert "pkg1" in apt_install_args + assert ("pkg2", "ver2") in apt_install_args + assert "pkg3" not in apt_install_args + + m_snap_install.assert_not_called() diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_netconfig.py cloud-init-23.4.3/tests/unittests/distros/test_netconfig.py --- cloud-init-23.3.3/tests/unittests/distros/test_netconfig.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_netconfig.py 2024-02-02 21:21:52.000000000 +0000 @@ -1333,6 +1333,3 @@ def get_mode(path, target=None): # Mask upper st_mode bits like S_IFREG bit preserve sticky and isuid/osgid return os.stat(subp.target_path(target, path)).st_mode & 0o777 - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_resolv.py cloud-init-23.4.3/tests/unittests/distros/test_resolv.py --- cloud-init-23.3.3/tests/unittests/distros/test_resolv.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_resolv.py 2024-02-02 21:21:52.000000000 +0000 @@ -65,6 +65,3 @@ self.assertEqual(len(rp.search_domains), 6) self.assertRaises(ValueError, rp.add_search_domain, "bbb5.y.com") self.assertEqual(len(rp.search_domains), 6) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_sysconfig.py cloud-init-23.4.3/tests/unittests/distros/test_sysconfig.py --- cloud-init-23.3.3/tests/unittests/distros/test_sysconfig.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_sysconfig.py 2024-02-02 21:21:52.000000000 +0000 @@ -65,7 +65,7 @@ conf["IPV6TO4_ROUTING"] = "blah \tblah" contents2 = str(conf).strip() # Should be requoted due to whitespace - self.assertRegex(contents2, r"IPV6TO4_ROUTING=[\']blah\s+blah[\']") + self.assertRegex(contents2, r"IPV6TO4_ROUTING='blah\s+blah'") def test_parse_no_adjust_shell(self): conf = SysConf("".splitlines()) @@ -85,6 +85,3 @@ contents = str(conf) self.assertIn("Z=d", contents) self.assertIn("BLAH=b", contents) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/distros/test_user_data_normalize.py cloud-init-23.4.3/tests/unittests/distros/test_user_data_normalize.py --- cloud-init-23.3.3/tests/unittests/distros/test_user_data_normalize.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/distros/test_user_data_normalize.py 2024-02-02 21:21:52.000000000 +0000 @@ -362,6 +362,3 @@ mock_subp.assert_any_call(groupcmd) mock_subp.assert_any_call(addcmd, logstring=addcmd) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/early_patches.py cloud-init-23.4.3/tests/unittests/early_patches.py --- cloud-init-23.3.3/tests/unittests/early_patches.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/early_patches.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,12 +1,13 @@ import functools +from typing import Callable old_lru_cache = functools.lru_cache cached_functions = [] def wrapped_lru_cache(*args, **kwargs): - def wrapper(func): - new_func = old_lru_cache(*args, **kwargs)(func) + def wrapper(func: Callable, *a, **k): + new_func = old_lru_cache(*args, **kwargs)(func, *a, **k) # Without this check, we'll also store stdlib functions with @lru_cache if "cloudinit" in func.__module__: diff -Nru cloud-init-23.3.3/tests/unittests/filters/test_launch_index.py cloud-init-23.4.3/tests/unittests/filters/test_launch_index.py --- cloud-init-23.3.3/tests/unittests/filters/test_launch_index.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/filters/test_launch_index.py 2024-02-02 21:21:52.000000000 +0000 @@ -129,6 +129,3 @@ "1": 2, } self.assertCounts(message, expected_counts) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/helpers.py cloud-init-23.4.3/tests/unittests/helpers.py --- cloud-init-23.3.3/tests/unittests/helpers.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/helpers.py 2024-02-02 21:21:52.000000000 +0000 @@ -21,7 +21,7 @@ import responses import cloudinit -from cloudinit import cloud, distros +from cloudinit import atomic_helper, cloud, distros from cloudinit import helpers as ch from cloudinit import subp, util from cloudinit.config.schema import ( @@ -39,6 +39,14 @@ skipIf = unittest.skipIf +try: + import apt_pkg # type: ignore # noqa: F401 + + HAS_APT_PKG = True +except ImportError: + HAS_APT_PKG = False + + # Makes the old path start # with new base instead of whatever # it previously had @@ -268,8 +276,8 @@ make_path = rebase_path(real_path[len(real_root) :], target_root) util.ensure_dir(make_path) for f in filenames: - real_path = util.abs_join(real_path, f) - make_path = util.abs_join(make_path, f) + real_path = os.path.abspath(os.path.join(real_path, f)) + make_path = os.path.abspath(os.path.join(make_path, f)) shutil.copy(real_path, make_path) def patchUtils(self, new_root): @@ -285,6 +293,9 @@ ("sym_link", -1), ("copy", -1), ], + atomic_helper: [ + ("write_json", 1), + ], } for (mod, funcs) in patch_funcs.items(): for (f, am) in funcs: @@ -519,6 +530,13 @@ return fh.read() +def skipIfAptPkg(): + return skipIf( + HAS_APT_PKG, + "No python-apt dependency present.", + ) + + try: import jsonschema @@ -619,6 +637,3 @@ """ yield - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/net/test_ephemeral.py cloud-init-23.4.3/tests/unittests/net/test_ephemeral.py --- cloud-init-23.3.3/tests/unittests/net/test_ephemeral.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/net/test_ephemeral.py 2024-02-02 21:21:52.000000000 +0000 @@ -5,6 +5,8 @@ import pytest from cloudinit.net.ephemeral import EphemeralIPNetwork +from cloudinit.subp import ProcessExecutionError +from tests.unittests.helpers import does_not_raise from tests.unittests.util import MockDistro M_PATH = "cloudinit.net.ephemeral." @@ -51,3 +53,52 @@ expected_call_args_list == m_exit_stack.return_value.enter_context.call_args_list ) + + @pytest.mark.parametrize( + "m_v4, m_v6, m_context, m_side_effects", + [ + pytest.param( + False, True, does_not_raise(), [None, None], id="v6_only" + ), + pytest.param( + True, False, does_not_raise(), [None, None], id="v4_only" + ), + pytest.param( + True, + True, + does_not_raise(), + [ProcessExecutionError, None], + id="v4_error", + ), + pytest.param( + True, + True, + does_not_raise(), + [None, ProcessExecutionError], + id="v6_error", + ), + pytest.param( + True, + True, + pytest.raises(ProcessExecutionError), + [ + ProcessExecutionError, + ProcessExecutionError, + ], + id="v4_v6_error", + ), + ], + ) + def test_interface_init_failures( + self, m_v4, m_v6, m_context, m_side_effects, mocker + ): + mocker.patch( + "cloudinit.net.ephemeral.EphemeralDHCPv4" + ).return_value.__enter__.side_effect = m_side_effects[0] + mocker.patch( + "cloudinit.net.ephemeral.EphemeralIPv6Network" + ).return_value.__enter__.side_effect = m_side_effects[1] + distro = MockDistro() + with m_context: + with EphemeralIPNetwork(distro, "eth0", ipv4=m_v4, ipv6=m_v6): + pass diff -Nru cloud-init-23.3.3/tests/unittests/net/test_network_state.py cloud-init-23.4.3/tests/unittests/net/test_network_state.py --- cloud-init-23.3.3/tests/unittests/net/test_network_state.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/net/test_network_state.py 2024-02-02 21:21:52.000000000 +0000 @@ -214,7 +214,7 @@ In netplan targets we perform a passthrough and the warning is not needed. """ - log.setupLogging() + log.setup_logging() util.deprecate._log = set() # type: ignore ncfg = safeyaml.load( diff -Nru cloud-init-23.3.3/tests/unittests/reporting/test_reporting.py cloud-init-23.4.3/tests/unittests/reporting/test_reporting.py --- cloud-init-23.3.3/tests/unittests/reporting/test_reporting.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/reporting/test_reporting.py 2024-02-02 21:21:52.000000000 +0000 @@ -2,6 +2,7 @@ # # This file is part of cloud-init. See LICENSE file for license information. +import re from unittest import mock import pytest @@ -537,10 +538,19 @@ ({"reporting": "a"}, "'a' is not of type 'object'"), ({"reporting": {"a": "b"}}, "'b' is not of type 'object'"), # BAD: invalid type - ({"reporting": {"a": {"type": "b"}}}, "not valid"), + ( + {"reporting": {"a": {"type": "b"}}}, + re.escape("'b' is not one of ['log']"), + ), # BAD: invalid additional properties - ({"reporting": {"a": {"type": "print", "a": "b"}}}, "not valid"), - ({"reporting": {"a": {"type": "log", "a": "b"}}}, "not valid"), + ( + {"reporting": {"a": {"type": "print", "a": "b"}}}, + "'a' was unexpected", + ), + ( + {"reporting": {"a": {"type": "log", "a": "b"}}}, + "'a' was unexpected", + ), ( { "reporting": { @@ -551,14 +561,26 @@ } } }, - "not valid", + "'a' was unexpected", + ), + ( + {"reporting": {"a": {"type": "hyperv", "a": "b"}}}, + "'a' was unexpected", ), - ({"reporting": {"a": {"type": "hyperv", "a": "b"}}}, "not valid"), # BAD: missing required properties - ({"reporting": {"a": {"level": "FATAL"}}}, "not valid"), - ({"reporting": {"a": {"endpoint": "http://a"}}}, "not valid"), - ({"reporting": {"a": {"kvp_file_path": "/a/b"}}}, "not valid"), - ({"reporting": {"a": {"type": "webhook"}}}, "not valid"), + ({"reporting": {"a": {"level": "FATAL"}}}, "'type' is a required"), + ( + {"reporting": {"a": {"endpoint": "http://a"}}}, + "'type' is a required", + ), + ( + {"reporting": {"a": {"kvp_file_path": "/a/b"}}}, + "'endpoint' is a required", + ), + ( + {"reporting": {"a": {"type": "webhook"}}}, + "'endpoint' is a required", + ), ], ) def test_schema_validation(self, config, error_msg): diff -Nru cloud-init-23.3.3/tests/unittests/runs/test_merge_run.py cloud-init-23.4.3/tests/unittests/runs/test_merge_run.py --- cloud-init-23.3.3/tests/unittests/runs/test_merge_run.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/runs/test_merge_run.py 2024-02-02 21:21:52.000000000 +0000 @@ -57,6 +57,3 @@ self.assertIn("write_files", which_ran) contents = util.load_file("/etc/blah.ini") self.assertEqual(contents, "blah") - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/runs/test_simple_run.py cloud-init-23.4.3/tests/unittests/runs/test_simple_run.py --- cloud-init-23.3.3/tests/unittests/runs/test_simple_run.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/runs/test_simple_run.py 2024-02-02 21:21:52.000000000 +0000 @@ -3,9 +3,10 @@ import copy import os -from cloudinit import safeyaml, stages, util +from cloudinit import atomic_helper, safeyaml, stages, util from cloudinit.config.modules import Modules from cloudinit.settings import PER_INSTANCE +from cloudinit.sources import NetworkConfigSource from tests.unittests import helpers @@ -47,6 +48,15 @@ def test_none_ds_populates_var_lib_cloud(self): """Init and run_section default behavior creates appropriate dirs.""" # Now start verifying whats created + netcfg = { + "version": 1, + "config": [{"type": "physical", "name": "eth9"}], + } + + def fake_network_config(): + return netcfg, NetworkConfigSource.FALLBACK + + self.assertFalse(os.path.exists("/var/lib/cloud")) initer = stages.Init() initer.read_cfg() initer.initialize() @@ -55,10 +65,20 @@ self.assertTrue(os.path.isdir(os.path.join("/var/lib/cloud", d))) initer.fetch() + self.assertFalse(os.path.islink("var/lib/cloud/instance")) iid = initer.instancify() self.assertEqual(iid, "iid-datasource-none") initer.update() self.assertTrue(os.path.islink("var/lib/cloud/instance")) + initer._find_networking_config = fake_network_config + self.assertFalse( + os.path.exists("/var/lib/cloud/instance/network-config.json") + ) + initer.apply_network_config(False) + self.assertEqual( + f"{atomic_helper.json_dumps(netcfg)}\n", + util.load_file("/var/lib/cloud/instance/network-config.json"), + ) def test_none_ds_runs_modules_which_do_not_define_distros(self): """Any modules which do not define a distros attribute are run.""" @@ -205,6 +225,3 @@ (which_ran, failures) = mods.run_section("cloud_init_modules") self.assertTrue(len(failures) == 0) self.assertEqual([], which_ran) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/azure/test_errors.py cloud-init-23.4.3/tests/unittests/sources/azure/test_errors.py --- cloud-init-23.3.3/tests/unittests/sources/azure/test_errors.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/azure/test_errors.py 2024-02-02 21:21:52.000000000 +0000 @@ -160,9 +160,16 @@ ( UrlError( Exception(), + code=404, + ), + "http error 404 querying IMDS", + ), + ( + UrlError( + Exception(), code=500, ), - "http error querying IMDS", + "http error 500 querying IMDS", ), ( UrlError( @@ -218,3 +225,13 @@ quoted_value = quote_csv_value(f"exception={source_error!r}") assert f"|{quoted_value}|" in error.as_encoded_report() + + +def test_imds_invalid_metadata(): + key = "compute" + value = "Running" + error = errors.ReportableErrorImdsInvalidMetadata(key=key, value=value) + + assert error.reason == "invalid IMDS metadata for key=compute" + assert error.supporting_data["key"] == key + assert error.supporting_data["value"] == repr(value) diff -Nru cloud-init-23.3.3/tests/unittests/sources/azure/test_imds.py cloud-init-23.4.3/tests/unittests/sources/azure/test_imds.py --- cloud-init-23.3.3/tests/unittests/sources/azure/test_imds.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/azure/test_imds.py 2024-02-02 21:21:52.000000000 +0000 @@ -8,6 +8,7 @@ import pytest import requests +import responses from cloudinit.sources.azure import imds from cloudinit.url_helper import UrlError, readurl @@ -23,6 +24,24 @@ def __eq__(self, other) -> bool: return bool(re.match("^" + self.regex + "$", other)) + def __repr__(self) -> str: + return repr(self.regex) + + +@pytest.fixture(autouse=True) +def caplog(caplog): + # Ensure caplog is set to debug. + caplog.set_level(logging.DEBUG) + yield caplog + + +@pytest.fixture +def mock_requests(): + with responses.RequestsMock( + assert_all_requests_are_fired=True + ) as response: + yield response + @pytest.fixture def wrapped_readurl(): @@ -67,6 +86,49 @@ ) +REQUESTS_CONNECTION_ERROR = requests.ConnectionError("Fake connection error") + +REQUESTS_TIMEOUT_ERROR = requests.Timeout("Fake connection timeout") + + +def add_errors_to_mock_requests(mock_requests, errors, url): + def callback_connection_error(request): + raise REQUESTS_CONNECTION_ERROR + + def callback_timeout(request): + raise REQUESTS_TIMEOUT_ERROR + + for error in errors: + if isinstance(error, int): + mock_requests.add( + method=responses.GET, + url=url, + status=error, + ) + elif error == REQUESTS_CONNECTION_ERROR: + mock_requests.add_callback( + method=responses.GET, + url=url, + callback=callback_connection_error, + ) + elif error == REQUESTS_TIMEOUT_ERROR: + mock_requests.add_callback( + method=responses.GET, + url=url, + callback=callback_timeout, + ) + else: + assert False + + +def regex_for_http_error(error): + if isinstance(error, int): + return f".*{error!s}.*" + + # Returns 'Fake connection error' or 'Fake connection timeout' + return f".*{error!s}.*" + + class TestFetchMetadataWithApiFallback: default_url = ( "http://169.254.169.254/metadata/instance?" @@ -75,6 +137,9 @@ fallback_url = ( "http://169.254.169.254/metadata/instance?api-version=2019-06-01" ) + + # Early versions of responses do not appreciate the parameters... + base_url = "http://169.254.169.254/metadata/instance" headers = {"Metadata": "true"} timeout = 30 @@ -82,14 +147,17 @@ def test_basic( self, caplog, - mock_requests_session_request, retry_deadline, wrapped_readurl, + mock_requests, ): fake_md = {"foo": {"bar": []}} - mock_requests_session_request.side_effect = [ - mock.Mock(content=json.dumps(fake_md)), - ] + mock_requests.add( + method=responses.GET, + url=self.base_url, + json=fake_md, + status=200, + ) md = imds.fetch_metadata_with_api_fallback( retry_deadline=retry_deadline @@ -121,17 +189,20 @@ @pytest.mark.parametrize("retry_deadline", [0.0, 1.0, 60.0]) def test_basic_fallback( - self, - caplog, - mock_requests_session_request, - retry_deadline, - wrapped_readurl, + self, caplog, retry_deadline, wrapped_readurl, mock_requests ): fake_md = {"foo": {"bar": []}} - mock_requests_session_request.side_effect = [ - UrlError("No IMDS version", code=400), - mock.Mock(content=json.dumps(fake_md)), - ] + mock_requests.add( + method=responses.GET, + url=self.base_url, + status=400, + ) + mock_requests.add( + method=responses.GET, + url=self.base_url, + json=fake_md, + status=200, + ) md = imds.fetch_metadata_with_api_fallback( retry_deadline=retry_deadline @@ -165,8 +236,15 @@ ), ( LOG_PATH, + logging.INFO, + StringMatch("Polling IMDS failed attempt 1 with.*400.*"), + ), + ( + LOG_PATH, logging.WARNING, - "Failed to fetch metadata from IMDS: No IMDS version", + StringMatch( + "Failed to fetch metadata from IMDS:.*400 Client Error.*" + ), ), ( LOG_PATH, @@ -181,44 +259,53 @@ ( "cloudinit.url_helper", logging.DEBUG, - StringMatch("Read from.*"), + f"Read from {self.fallback_url} (200, 20b) after 1 attempts", ), ] @pytest.mark.parametrize( "error", [ - fake_http_error_for_code(404), - fake_http_error_for_code(410), - fake_http_error_for_code(429), - fake_http_error_for_code(500), - requests.ConnectionError("Fake connection error"), - requests.Timeout("Fake connection timeout"), + 404, + 410, + 429, + 500, + REQUESTS_CONNECTION_ERROR, + REQUESTS_TIMEOUT_ERROR, ], ) - @pytest.mark.parametrize("max_attempts,retry_deadline", [(2, 1.0)]) + @pytest.mark.parametrize("error_count,retry_deadline", [(1, 2.0)]) def test_will_retry_errors( self, caplog, - max_attempts, retry_deadline, - mock_requests_session_request, + mock_requests, mock_url_helper_time_sleep, error, + error_count, ): fake_md = {"foo": {"bar": []}} - mock_requests_session_request.side_effect = [ - error, - mock.Mock(content=json.dumps(fake_md)), - ] + add_errors_to_mock_requests( + mock_requests, [error] * error_count, self.base_url + ) + mock_requests.add( + method=responses.GET, + url=self.base_url, + json=fake_md, + status=200, + ) md = imds.fetch_metadata_with_api_fallback( retry_deadline=retry_deadline ) assert md == fake_md - assert len(mock_requests_session_request.mock_calls) == max_attempts - assert mock_url_helper_time_sleep.mock_calls == [mock.call(1)] + assert ( + mock_url_helper_time_sleep.mock_calls + == [mock.call(1)] * error_count + ) + + error_regex = regex_for_http_error(error) assert caplog.record_tuples == [ ( "cloudinit.url_helper", @@ -229,8 +316,8 @@ LOG_PATH, logging.INFO, StringMatch( - "Polling IMDS failed attempt 1 with exception:" - f".*{error!s}.*" + "Polling IMDS failed attempt 1 with exception: " + f"{error_regex}" ), ), ( @@ -255,24 +342,32 @@ self, caplog, retry_deadline, - mock_requests_session_request, + mock_requests, mock_url_helper_time_sleep, ): - error = fake_http_error_for_code(400) fake_md = {"foo": {"bar": []}} - mock_requests_session_request.side_effect = [ - error, - fake_http_error_for_code(429), - mock.Mock(content=json.dumps(fake_md)), - ] - max_attempts = len(mock_requests_session_request.side_effect) + mock_requests.add( + method=responses.GET, + url=self.base_url, + status=400, + ) + mock_requests.add( + method=responses.GET, + url=self.base_url, + status=429, + ) + mock_requests.add( + method=responses.GET, + url=self.base_url, + json=fake_md, + status=200, + ) md = imds.fetch_metadata_with_api_fallback( retry_deadline=retry_deadline ) assert md == fake_md - assert len(mock_requests_session_request.mock_calls) == max_attempts assert mock_url_helper_time_sleep.mock_calls == [mock.call(1)] assert caplog.record_tuples == [ ( @@ -284,14 +379,15 @@ LOG_PATH, logging.INFO, StringMatch( - "Polling IMDS failed attempt 1 with exception:" - f".*{error!s}.*" + "Polling IMDS failed attempt 1 with exception:.*400.*" ), ), ( LOG_PATH, logging.WARNING, - "Failed to fetch metadata from IMDS: fake error", + StringMatch( + "Failed to fetch metadata from IMDS:.*400 Client Error.*" + ), ), ( LOG_PATH, @@ -307,8 +403,7 @@ LOG_PATH, logging.INFO, StringMatch( - "Polling IMDS failed attempt 1 with exception:" - f".*{error!s}.*" + "Polling IMDS failed attempt 1 with exception:.*429.*" ), ), ( @@ -331,43 +426,51 @@ @pytest.mark.parametrize( "error", [ - fake_http_error_for_code(404), - fake_http_error_for_code(410), - fake_http_error_for_code(429), - fake_http_error_for_code(500), - requests.ConnectionError("Fake connection error"), - requests.Timeout("Fake connection timeout"), + 404, + 410, + 429, + 500, + REQUESTS_CONNECTION_ERROR, + REQUESTS_TIMEOUT_ERROR, ], ) @pytest.mark.parametrize( - "max_attempts,retry_deadline", [(1, 0.0), (2, 1.0), (301, 300.0)] + "error_count,retry_deadline", [(1, 0.0), (2, 1.0), (301, 300.0)] ) + @pytest.mark.parametrize("max_connection_errors", [None, 1, 11]) def test_retry_until_failure( self, error, - max_attempts, + error_count, + max_connection_errors, retry_deadline, caplog, - mock_requests_session_request, + mock_requests, mock_url_helper_time_sleep, ): - mock_requests_session_request.side_effect = error + add_errors_to_mock_requests( + mock_requests, [error] * error_count, self.base_url + ) with pytest.raises(UrlError) as exc_info: imds.fetch_metadata_with_api_fallback( - retry_deadline=retry_deadline + max_connection_errors=max_connection_errors, + retry_deadline=retry_deadline, ) - assert exc_info.value.cause == error + error_regex = regex_for_http_error(error) + assert re.search(error_regex, str(exc_info.value.cause)) - # Connection errors max out at 11 attempts. max_attempts = ( - 11 + min(max_connection_errors, int(retry_deadline) + 1) if isinstance(error, requests.ConnectionError) - and max_attempts > 11 - else max_attempts + and isinstance(max_connection_errors, int) + else error_count ) - assert len(mock_requests_session_request.mock_calls) == (max_attempts) + + if max_attempts < error_count: + mock_requests.assert_all_requests_are_fired = False + assert mock_url_helper_time_sleep.mock_calls == [mock.call(1)] * ( max_attempts - 1 ) @@ -379,7 +482,7 @@ logging.INFO, StringMatch( f"Polling IMDS failed attempt {i} with exception:" - f".*{error!s}.*" + f"{error_regex}" ), ) for i in range(1, max_attempts + 1) @@ -387,15 +490,17 @@ ( LOG_PATH, logging.WARNING, - f"Failed to fetch metadata from IMDS: {error!s}", + StringMatch( + f"Failed to fetch metadata from IMDS: {error_regex}", + ), ) ] @pytest.mark.parametrize( "error", [ - fake_http_error_for_code(403), - fake_http_error_for_code(501), + 403, + 501, ], ) @pytest.mark.parametrize("retry_deadline", [0.0, 1.0, 60.0]) @@ -404,22 +509,22 @@ error, retry_deadline, caplog, - mock_requests_session_request, + mock_requests, mock_url_helper_time_sleep, ): - fake_md = {"foo": {"bar": []}} - mock_requests_session_request.side_effect = [ - error, - mock.Mock(content=json.dumps(fake_md)), - ] + mock_requests.add( + method=responses.GET, + url=self.base_url, + status=error, + ) with pytest.raises(UrlError) as exc_info: imds.fetch_metadata_with_api_fallback( retry_deadline=retry_deadline ) - assert exc_info.value.cause == error - assert len(mock_requests_session_request.mock_calls) == 1 + error_regex = regex_for_http_error(error) + assert re.search(error_regex, str(exc_info.value.cause)) assert mock_url_helper_time_sleep.mock_calls == [] assert caplog.record_tuples == [ @@ -433,27 +538,33 @@ logging.INFO, StringMatch( "Polling IMDS failed attempt 1 with exception:" - f".*{error!s}.*" + f".*{error_regex}" ), ), ( LOG_PATH, logging.WARNING, - f"Failed to fetch metadata from IMDS: {error!s}", + StringMatch( + f"Failed to fetch metadata from IMDS: {error_regex!s}" + ), ), ] + @pytest.mark.parametrize("body", ["", "invalid", ""]) @pytest.mark.parametrize("retry_deadline", [0.0, 1.0, 60.0]) def test_non_json_repsonse( self, + body, retry_deadline, caplog, - mock_requests_session_request, + mock_requests, wrapped_readurl, ): - mock_requests_session_request.side_effect = [ - mock.Mock(content=b"bad data") - ] + mock_requests.add( + method=responses.GET, + url=self.base_url, + body=body, + ) with pytest.raises(ValueError): imds.fetch_metadata_with_api_fallback( @@ -481,6 +592,47 @@ ) ] + def test_logs_all_errors( + self, + caplog, + mock_requests, + mock_url_helper_time_sleep, + ): + fake_md = {"foo": {"bar": []}} + + errors = ( + [404] * 10 + + [410] * 10 + + [429] * 10 + + [500] * 10 + + [REQUESTS_CONNECTION_ERROR] * 10 + + [REQUESTS_TIMEOUT_ERROR] * 10 + ) + + add_errors_to_mock_requests(mock_requests, errors, self.base_url) + mock_requests.add( + method=responses.GET, + url=self.base_url, + json=fake_md, + ) + + md = imds.fetch_metadata_with_api_fallback(retry_deadline=300) + + assert md == fake_md + assert mock_url_helper_time_sleep.mock_calls == [mock.call(1)] * ( + len(errors) + ) + + expected_logs = [ + StringMatch( + f"Polling IMDS failed attempt {i} with exception:" + f"{regex_for_http_error(error)}" + ) + for i, error in enumerate(errors, start=1) + ] + logs = [t[2] for t in caplog.record_tuples if "Polling IMDS" in t[2]] + assert logs == expected_logs + class TestFetchReprovisionData: url = ( @@ -490,6 +642,9 @@ headers = {"Metadata": "true"} timeout = 30 + # Early versions of responses do not appreciate the parameters... + base_url = "http://169.254.169.254/metadata/reprovisiondata" + def test_basic( self, caplog, @@ -533,6 +688,7 @@ [ fake_http_error_for_code(404), fake_http_error_for_code(410), + fake_http_error_for_code(429), ], ) @pytest.mark.parametrize("failures", [1, 5, 100, 1000]) @@ -591,6 +747,7 @@ [ fake_http_error_for_code(404), fake_http_error_for_code(410), + fake_http_error_for_code(429), ], ) @pytest.mark.parametrize("failures", [1, 5, 100, 1000]) @@ -683,3 +840,45 @@ f"{exc_info.value!r}", ), ] + + def test_logs_unique_errors( + self, + caplog, + mock_requests, + mock_url_helper_time_sleep, + ): + content = b"ovf content" + + errors = ( + [404] * 10 + + [410] * 10 + + [429] * 10 + + [404] * 10 + + [410] * 10 + + [429] * 10 + ) + + add_errors_to_mock_requests(mock_requests, errors, self.base_url) + mock_requests.add( + method=responses.GET, + url=self.base_url, + body=content, + ) + + ovf = imds.fetch_reprovision_data() + + assert ovf == content + assert mock_url_helper_time_sleep.mock_calls == [mock.call(1)] * ( + len(errors) + ) + + backoff_logs = [ + StringMatch( + f"Polling IMDS failed attempt {i} with exception:" + f"{regex_for_http_error(error)}" + ) + for i, error in enumerate(errors, start=1) + if i == 1 or math.log2(i).is_integer() or (i - 1) % 10 == 0 + ] + logs = [t[2] for t in caplog.record_tuples if "Polling IMDS" in t[2]] + assert logs == backoff_logs diff -Nru cloud-init-23.3.3/tests/unittests/sources/helpers/test_ec2.py cloud-init-23.4.3/tests/unittests/sources/helpers/test_ec2.py --- cloud-init-23.3.3/tests/unittests/sources/helpers/test_ec2.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/helpers/test_ec2.py 2024-02-02 21:21:52.000000000 +0000 @@ -322,6 +322,3 @@ self.assertEqual(md["ami-launch-index"], "1") md = ec2.get_instance_metadata(self.VERSION, retries=0, timeout=0.1) self.assertEqual(len(md), 0) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_aliyun.py cloud-init-23.4.3/tests/unittests/sources/test_aliyun.py --- cloud-init-23.3.3/tests/unittests/sources/test_aliyun.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_aliyun.py 2024-02-02 21:21:52.000000000 +0000 @@ -179,6 +179,53 @@ "metadata (http://100.100.100.200)", self.ds.subplatform ) + @mock.patch("cloudinit.net.ephemeral.EphemeralIPv6Network") + @mock.patch("cloudinit.net.ephemeral.EphemeralIPv4Network") + @mock.patch("cloudinit.sources.DataSourceEc2.util.is_resolvable") + @mock.patch("cloudinit.sources.DataSourceAliYun._is_aliyun") + @mock.patch("cloudinit.net.find_fallback_nic") + @mock.patch("cloudinit.net.ephemeral.maybe_perform_dhcp_discovery") + @mock.patch("cloudinit.sources.DataSourceEc2.util.is_FreeBSD") + def test_aliyun_local_with_mock_server( + self, + m_is_bsd, + m_dhcp, + m_fallback_nic, + m_is_aliyun, + m_resolva, + m_net4, + m_net6, + ): + m_is_aliyun.return_value = True + m_fallback_nic.return_value = "eth9" + m_dhcp.return_value = [ + { + "interface": "eth9", + "fixed-address": "192.168.2.9", + "routers": "192.168.2.1", + "subnet-mask": "255.255.255.0", + "broadcast-address": "192.168.2.255", + } + ] + m_is_bsd.return_value = False + cfg = {"datasource": {"AliYun": {"timeout": "1", "max_wait": "1"}}} + distro = mock.MagicMock() + paths = helpers.Paths({"run_dir": self.tmp_dir()}) + self.ds = ay.DataSourceAliYunLocal(cfg, distro, paths) + self.regist_default_server() + ret = self.ds.get_data() + self.assertEqual(True, ret) + self.assertEqual(1, m_is_aliyun.call_count) + self._test_get_data() + self._test_get_sshkey() + self._test_get_iid() + self._test_host_name() + self.assertEqual("aliyun", self.ds.cloud_name) + self.assertEqual("ec2", self.ds.platform) + self.assertEqual( + "metadata (http://100.100.100.200)", self.ds.subplatform + ) + @mock.patch("cloudinit.sources.DataSourceAliYun._is_aliyun") def test_returns_false_when_not_on_aliyun(self, m_is_aliyun): """If is_aliyun returns false, then get_data should return False.""" @@ -292,6 +339,3 @@ self.read_dmi_data_expected, m_read_dmi_data.call_args_list ) self.assertEqual(False, ret) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_altcloud.py cloud-init-23.4.3/tests/unittests/sources/test_altcloud.py --- cloud-init-23.3.3/tests/unittests/sources/test_altcloud.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_altcloud.py 2024-02-02 21:21:52.000000000 +0000 @@ -454,6 +454,3 @@ setattr(os, "uname", _os_uname) elif arch is None: setattr(os, "uname", OS_UNAME_ORIG) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_azure.py cloud-init-23.4.3/tests/unittests/sources/test_azure.py --- cloud-init-23.3.3/tests/unittests/sources/test_azure.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_azure.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,6 +1,7 @@ # This file is part of cloud-init. See LICENSE file for license information. import copy +import datetime import json import os import stat @@ -13,7 +14,7 @@ from cloudinit import distros, dmi, helpers, subp, url_helper from cloudinit.atomic_helper import b64e, json_dumps -from cloudinit.net import dhcp +from cloudinit.net import dhcp, ephemeral from cloudinit.reporting.handlers import HyperVKvpReportingHandler from cloudinit.sources import UNSET from cloudinit.sources import DataSourceAzure as dsaz @@ -84,6 +85,15 @@ yield m +@pytest.fixture(autouse=True) +def mock_device_driver(): + with mock.patch( + MOCKPATH + "device_driver", + return_value="fake_driver", + ) as m: + yield m + + @pytest.fixture def mock_generate_fallback_config(): with mock.patch( @@ -121,11 +131,31 @@ @pytest.fixture -def mock_ephemeral_dhcp_v4(): +def mock_ephemeral_dhcp_v4(mock_ephemeral_ipv4_network): with mock.patch( MOCKPATH + "EphemeralDHCPv4", + ) as m: + m.return_value._ephipv4 = mock_ephemeral_ipv4_network.return_value + yield m + + +@pytest.fixture +def mock_ephemeral_ipv4_network(): + with mock.patch( + "cloudinit.net.ephemeral.EphemeralIPv4Network", autospec=True, ) as m: + m.return_value.distro = "ubuntu" + m.return_value.interface = "eth0" + m.return_value.ip = "10.0.0.4" + m.return_value.prefix_or_mask = "32" + m.return_value.broadcast = "255.255.255.255" + m.return_value.router = "10.0.0.1" + m.return_value.static_routes = [ + ("0.0.0.0/0", "10.0.0.1"), + ("168.63.129.16/32", "10.0.0.1"), + ("169.254.169.254/32", "10.0.0.1"), + ] yield m @@ -177,15 +207,6 @@ yield m -@pytest.fixture -def mock_net_dhcp_EphemeralIPv4Network(): - with mock.patch( - "cloudinit.net.ephemeral.EphemeralIPv4Network", - autospec=True, - ) as m: - yield m - - @pytest.fixture(autouse=True) def mock_get_interfaces(): with mock.patch( @@ -254,6 +275,14 @@ @pytest.fixture +def mock_timestamp(): + timestamp = datetime.datetime.utcnow() + with mock.patch.object(errors, "datetime", autospec=True) as m: + m.utcnow.return_value = timestamp + yield timestamp + + +@pytest.fixture def mock_util_ensure_dir(): with mock.patch( MOCKPATH + "util.ensure_dir", @@ -520,11 +549,12 @@ class TestGenerateNetworkConfig: @pytest.mark.parametrize( - "label,metadata,expected", + "label,metadata,ip_config,expected", [ ( "simple interface", NETWORK_METADATA["network"], + True, { "ethernets": { "eth0": { @@ -559,6 +589,7 @@ } ] }, + True, { "ethernets": { "eth0": { @@ -596,6 +627,7 @@ } ] }, + True, { "ethernets": { "eth0": { @@ -633,6 +665,7 @@ } ] }, + True, { "ethernets": { "eth0": { @@ -670,6 +703,7 @@ ] * 3 }, + True, { "ethernets": { "eth0": { @@ -736,6 +770,7 @@ } ] }, + True, { "ethernets": { "eth0": { @@ -752,6 +787,60 @@ }, ), ( + "secondary IPv4s are not configured", + { + "interface": [ + { + "macAddress": "000D3A047598", + "ipv6": { + "subnet": [ + { + "prefix": "10", + "address": "2001:dead:beef::16", + } + ], + "ipAddress": [ + {"privateIpAddress": "2001:dead:beef::1"} + ], + }, + "ipv4": { + "subnet": [ + {"prefix": "24", "address": "10.0.0.0"}, + ], + "ipAddress": [ + { + "privateIpAddress": "10.0.0.4", + "publicIpAddress": "104.46.124.81", + }, + { + "privateIpAddress": "11.0.0.5", + "publicIpAddress": "104.46.124.82", + }, + { + "privateIpAddress": "12.0.0.6", + "publicIpAddress": "104.46.124.83", + }, + ], + }, + } + ] + }, + False, + { + "ethernets": { + "eth0": { + "dhcp4": True, + "dhcp4-overrides": {"route-metric": 100}, + "dhcp6": True, + "dhcp6-overrides": {"route-metric": 100}, + "match": {"macaddress": "00:0d:3a:04:75:98"}, + "set-name": "eth0", + } + }, + "version": 2, + }, + ), + ( "ipv6 secondaries", { "interface": [ @@ -772,6 +861,7 @@ } ] }, + True, { "ethernets": { "eth0": { @@ -787,14 +877,50 @@ "version": 2, }, ), + ( + "ipv6 secondaries not configured", + { + "interface": [ + { + "macAddress": "000D3A047598", + "ipv6": { + "subnet": [ + { + "prefix": "10", + "address": "2001:dead:beef::16", + } + ], + "ipAddress": [ + {"privateIpAddress": "2001:dead:beef::1"}, + {"privateIpAddress": "2001:dead:beef::2"}, + ], + }, + } + ] + }, + False, + { + "ethernets": { + "eth0": { + "dhcp4": True, + "dhcp4-overrides": {"route-metric": 100}, + "dhcp6": True, + "dhcp6-overrides": {"route-metric": 100}, + "match": {"macaddress": "00:0d:3a:04:75:98"}, + "set-name": "eth0", + } + }, + "version": 2, + }, + ), ], ) def test_parsing_scenarios( - self, label, mock_get_interfaces, metadata, expected + self, label, mock_get_interfaces, metadata, ip_config, expected ): assert ( dsaz.generate_network_config_from_instance_network_metadata( - metadata + metadata, apply_network_config_for_secondary_ips=ip_config ) == expected ) @@ -908,7 +1034,6 @@ mock.patch.object( dsaz, "EphemeralDHCPv4", - autospec=True, ) ) self.m_dhcp.return_value.lease = {} @@ -2479,6 +2604,28 @@ self.assertTrue(value) self.assertIn("safe for", msg.lower()) + def test_one_partition_ntfs_empty_with_svi_file_is_true(self): + """1 mountable ntfs partition and only warn file can be formatted.""" + bypath = self.patchup( + { + "/dev/sda": { + "partitions": { + "/dev/sda1": { + "num": 1, + "fs": "ntfs", + "files": ["System Volume Information"], + } + } + } + } + ) + self._domock_mount_cb(bypath) + value, msg = dsaz.can_dev_be_reformatted( + "/dev/sda", preserve_ntfs=False + ) + self.assertTrue(value) + self.assertIn("safe for", msg.lower()) + def test_one_partition_through_realpath_is_true(self): """A symlink to a device with 1 ntfs partition can be formatted.""" epath = "/dev/disk/cloud/azure_resource" @@ -2749,25 +2896,6 @@ ] -@mock.patch("os.path.isfile", return_value=False) -class TestReprovision(CiTestCase): - def setUp(self): - super(TestReprovision, self).setUp() - tmp = self.tmp_dir() - self.waagent_d = self.tmp_path("/var/lib/waagent", tmp) - self.paths = helpers.Paths({"cloud_dir": tmp}) - dsaz.BUILTIN_DS_CONFIG["data_dir"] = self.waagent_d - - @mock.patch(MOCKPATH + "DataSourceAzure._poll_imds") - def test_reprovision_calls__poll_imds(self, _poll_imds, isfile): - """_reprovision will poll IMDS.""" - isfile.return_value = False - _poll_imds.return_value = construct_ovf_env() - dsa = dsaz.DataSourceAzure({}, distro=mock.Mock(), paths=self.paths) - dsa._reprovision() - _poll_imds.assert_called_with() - - class TestPreprovisioningHotAttachNics(CiTestCase): def setUp(self): super(TestPreprovisioningHotAttachNics, self).setUp() @@ -2805,7 +2933,6 @@ @mock.patch(MOCKPATH + "DataSourceAzure._report_ready") @mock.patch(MOCKPATH + "DataSourceAzure.wait_for_link_up") @mock.patch("cloudinit.sources.helpers.netlink.wait_for_nic_attach_event") - @mock.patch(MOCKPATH + "imds.fetch_metadata_with_api_fallback") @mock.patch(MOCKPATH + "EphemeralDHCPv4", autospec=True) @mock.patch(MOCKPATH + "DataSourceAzure._wait_for_nic_detach") @mock.patch("os.path.isfile") @@ -2814,7 +2941,6 @@ m_isfile, m_detach, m_dhcpv4, - m_imds, m_attach, m_link_up, m_report_ready, @@ -2834,22 +2960,27 @@ } # Simulate two NICs by adding the same one twice. - md = { - "interface": [ - IMDS_NETWORK_METADATA["interface"][0], - IMDS_NETWORK_METADATA["interface"][0], - ] - } - m_isfile.return_value = True m_attach.side_effect = [ "eth0", "eth1", ] - dhcp_ctx = mock.MagicMock(lease=lease) - dhcp_ctx.obtain_lease.return_value = lease - m_dhcpv4.return_value = dhcp_ctx - m_imds.side_effect = [md] + dhcp_ctx_primary = mock.MagicMock(lease=lease) + dhcp_ctx_primary.obtain_lease.return_value = lease + dhcp_ctx_primary._ephipv4 = ephemeral.EphemeralIPv4Network( + distro="ubuntu", + interface="eth0", + ip="10.0.0.4", + prefix_or_mask="32", + broadcast="255.255.255.255", + router="10.0.0.1", + static_routes=[ + ("0.0.0.0/0", "10.0.0.1"), + ("168.63.129.16/32", "10.0.0.1"), + ("169.254.169.254/32", "10.0.0.1"), + ], + ) + m_dhcpv4.side_effect = [dhcp_ctx_primary] dsa._wait_for_pps_savable_reuse() @@ -2858,100 +2989,35 @@ self.assertEqual(1, m_attach.call_count) # DHCP and network metadata calls will only happen on the primary NIC. self.assertEqual(1, m_dhcpv4.call_count) - self.assertEqual(1, m_imds.call_count) # no call to bring link up on secondary nic self.assertEqual(1, m_link_up.call_count) # reset mock to test again with primary nic being eth1 + dhcp_ctx_primary.interface = "eth1" + dhcp_ctx_secondary = mock.MagicMock(lease=lease) + dhcp_ctx_secondary.obtain_lease.return_value = lease + dhcp_ctx_secondary._ephipv4 = ephemeral.EphemeralIPv4Network( + distro="ubuntu", + interface="eth0", + ip="10.0.0.4", + prefix_or_mask="32", + broadcast="255.255.255.255", + router="10.0.0.1", + static_routes=None, + ) m_detach.reset_mock() m_attach.reset_mock() m_dhcpv4.reset_mock() + m_dhcpv4.side_effect = [dhcp_ctx_secondary, dhcp_ctx_primary] m_link_up.reset_mock() m_attach.side_effect = ["eth0", "eth1"] - m_imds.reset_mock() - m_imds.side_effect = [{}, md] dsa = dsaz.DataSourceAzure({}, distro=distro, paths=self.paths) dsa._wait_for_pps_savable_reuse() self.assertEqual(1, m_detach.call_count) self.assertEqual(2, m_attach.call_count) self.assertEqual(2, m_dhcpv4.call_count) - self.assertEqual(2, m_imds.call_count) self.assertEqual(2, m_link_up.call_count) - @mock.patch("cloudinit.url_helper.time.sleep", autospec=True) - @mock.patch("requests.Session.request", autospec=True) - @mock.patch(MOCKPATH + "EphemeralDHCPv4", autospec=True) - @mock.patch(MOCKPATH + "time", autospec=True) - @mock.patch("cloudinit.sources.azure.imds.time", autospec=True) - def test_check_if_nic_is_primary_retries_on_failures( - self, m_imds_time, m_time, m_dhcpv4, m_request, m_sleep - ): - """Retry polling for network metadata on all failures except timeout - and network unreachable errors""" - distro = mock.MagicMock() - distro.get_tmp_exec_path = self.tmp_dir - dsa = dsaz.DataSourceAzure({}, distro=distro, paths=self.paths) - lease = { - "interface": "eth9", - "fixed-address": "192.168.2.9", - "routers": "192.168.2.1", - "subnet-mask": "255.255.255.0", - "unknown-245": "624c3620", - } - - m_req = mock.Mock(content=json.dumps({"not": "empty"})) - errors = ( - [requests.Timeout("Fake connection timeout")] * 5 - + [requests.ConnectionError("Fake Network Unreachable")] * 5 - + [fake_http_error_for_code(410)] * 5 - + [m_req] - ) - m_request.side_effect = errors - m_dhcpv4.return_value.lease = lease - fake_time = 0.0 - fake_time_increment = 5.0 - - def fake_timer(): - nonlocal fake_time, fake_time_increment - fake_time += fake_time_increment - return fake_time - - m_time.side_effect = fake_timer - m_imds_time.side_effect = fake_timer - - is_primary = dsa._check_if_nic_is_primary("eth0") - self.assertEqual(True, is_primary) - assert len(m_request.mock_calls) == len(errors) - - # Re-run tests to verify max http failures. - attempts = 3 - fake_time = 0.0 - fake_time_increment = 300 / attempts - - m_time.side_effect = fake_timer - m_imds_time.side_effect = fake_timer - m_request.reset_mock() - errors = 100 * [fake_http_error_for_code(410)] - m_request.side_effect = errors - - dsa = dsaz.DataSourceAzure({}, distro=distro, paths=self.paths) - - is_primary = dsa._check_if_nic_is_primary("eth1") - self.assertEqual(False, is_primary) - assert len(m_request.mock_calls) == attempts - - # Re-run tests to verify max connection error retries. - m_request.reset_mock() - m_request.side_effect = [ - requests.ConnectionError("Fake Network Unreachable") - ] * 15 - - dsa = dsaz.DataSourceAzure({}, distro=distro, paths=self.paths) - - is_primary = dsa._check_if_nic_is_primary("eth1") - self.assertEqual(False, is_primary) - assert len(m_request.mock_calls) == attempts - @mock.patch("cloudinit.distros.networking.LinuxNetworking.try_set_link_up") def test_wait_for_link_up_returns_if_already_up(self, m_is_link_up): """Waiting for link to be up should return immediately if the link is @@ -3001,8 +3067,7 @@ @mock.patch("cloudinit.net.find_fallback_nic", return_value="eth9") -@mock.patch("cloudinit.net.ephemeral.EphemeralIPv4Network") -@mock.patch("cloudinit.net.ephemeral.maybe_perform_dhcp_discovery") +@mock.patch(MOCKPATH + "EphemeralDHCPv4") @mock.patch( "cloudinit.sources.helpers.netlink.wait_for_media_disconnect_connect" ) @@ -3021,7 +3086,6 @@ m_fetch_reprovisiondata, m_media_switch, m_dhcp, - m_net, m_fallback, ): """The poll_imds will retry DHCP on IMDS timeout.""" @@ -3036,7 +3100,7 @@ "subnet-mask": "255.255.255.0", "unknown-245": "624c3620", } - m_dhcp.return_value = [lease] + m_dhcp.obtain_lease.return_value = [lease] m_media_switch.return_value = None dhcp_ctx = mock.MagicMock(lease=lease) dhcp_ctx.obtain_lease.return_value = lease @@ -3056,7 +3120,6 @@ m_fetch_reprovisiondata, m_media_switch, m_dhcp, - m_net, m_fallback, ): """The poll_imds function should reuse the dhcp ctx if it is already @@ -3072,15 +3135,12 @@ self.assertEqual(0, m_media_switch.call_count) @mock.patch("os.path.isfile") - @mock.patch(MOCKPATH + "EphemeralDHCPv4", autospec=True) def test_poll_imds_does_dhcp_on_retries_if_ctx_present( self, - m_ephemeral_dhcpv4, m_isfile, m_fetch_reprovisiondata, m_media_switch, m_dhcp, - m_net, m_fallback, ): """The poll_imds function should reuse the dhcp ctx if it is already @@ -3106,94 +3166,11 @@ dsa._ephemeral_dhcp_ctx = m_dhcp_ctx dsa._poll_imds() self.assertEqual(1, m_dhcp_ctx.clean_network.call_count) - self.assertEqual(1, m_ephemeral_dhcpv4.call_count) + self.assertEqual(1, m_dhcp.call_count) self.assertEqual(0, m_media_switch.call_count) self.assertEqual(2, m_fetch_reprovisiondata.call_count) -@mock.patch(MOCKPATH + "DataSourceAzure._report_ready", mock.MagicMock()) -@mock.patch(MOCKPATH + "subp.subp", mock.MagicMock()) -@mock.patch(MOCKPATH + "util.write_file", mock.MagicMock()) -@mock.patch( - "cloudinit.sources.helpers.netlink.wait_for_media_disconnect_connect" -) -@mock.patch("cloudinit.net.ephemeral.EphemeralIPv4Network", autospec=True) -@mock.patch("cloudinit.net.ephemeral.maybe_perform_dhcp_discovery") -@mock.patch( - MOCKPATH + "imds.fetch_reprovision_data", side_effect=[b"ovf data"] -) -class TestAzureDataSourcePreprovisioning(CiTestCase): - def setUp(self): - super(TestAzureDataSourcePreprovisioning, self).setUp() - tmp = self.tmp_dir() - self.waagent_d = self.tmp_path("/var/lib/waagent", tmp) - self.paths = helpers.Paths({"cloud_dir": tmp}) - dsaz.BUILTIN_DS_CONFIG["data_dir"] = self.waagent_d - - def test_poll_imds_returns_ovf_env( - self, m_fetch_reprovisiondata, m_dhcp, m_net, m_media_switch - ): - """The _poll_imds method should return the ovf_env.xml.""" - m_media_switch.return_value = None - m_dhcp.return_value = [ - { - "interface": "eth9", - "fixed-address": "192.168.2.9", - "routers": "192.168.2.1", - "subnet-mask": "255.255.255.0", - } - ] - dsa = dsaz.DataSourceAzure({}, distro=mock.Mock(), paths=self.paths) - dsa._ephemeral_dhcp_ctx = None - self.assertTrue(len(dsa._poll_imds()) > 0) - self.assertEqual(m_dhcp.call_count, 1) - m_net.assert_any_call( - dsa.distro, - broadcast="192.168.2.255", - interface="eth9", - ip="192.168.2.9", - prefix_or_mask="255.255.255.0", - router="192.168.2.1", - static_routes=None, - ) - self.assertEqual(m_net.call_count, 1) - - def test__reprovision_calls__poll_imds( - self, m_fetch_reprovisiondata, m_dhcp, m_net, m_media_switch - ): - """The _reprovision method should call poll IMDS.""" - m_media_switch.return_value = None - m_dhcp.return_value = [ - { - "interface": "eth9", - "fixed-address": "192.168.2.9", - "routers": "192.168.2.1", - "subnet-mask": "255.255.255.0", - "unknown-245": "624c3620", - } - ] - hostname = "myhost" - username = "myuser" - content = construct_ovf_env(username=username, hostname=hostname) - m_fetch_reprovisiondata.side_effect = [content] - dsa = dsaz.DataSourceAzure({}, distro=mock.Mock(), paths=self.paths) - dsa._ephemeral_dhcp_ctx = None - md, _ud, cfg, _d = dsa._reprovision() - self.assertEqual(md["local-hostname"], hostname) - self.assertEqual(cfg["system_info"]["default_user"]["name"], username) - self.assertEqual(m_dhcp.call_count, 1) - m_net.assert_any_call( - dsa.distro, - broadcast="192.168.2.255", - interface="eth9", - ip="192.168.2.9", - prefix_or_mask="255.255.255.0", - router="192.168.2.1", - static_routes=None, - ) - self.assertEqual(m_net.call_count, 1) - - class TestRemoveUbuntuNetworkConfigScripts(CiTestCase): with_logs = True @@ -3451,10 +3428,31 @@ assert mock_sleep.mock_calls == [mock.call(1)] assert mock_report_diagnostic_event.mock_calls == [ mock.call( + "Bringing up ephemeral networking with iface=None: " + "[('dummy0', '9e:65:d6:19:19:01', None, None), " + "('enP3', '00:11:22:33:44:02', 'unknown_accel', '0x3'), " + "('eth0', '00:11:22:33:44:00', 'hv_netvsc', '0x3'), " + "('eth2', '00:11:22:33:44:01', 'unknown', '0x3'), " + "('eth3', '00:11:22:33:44:02', " + "'unknown_with_unknown_vf', '0x3'), " + "('lo', '00:00:00:00:00:00', None, None)]", + logger_func=dsaz.LOG.debug, + ), + mock.call( "Command failed: cmd=['failed', 'cmd'] " "stderr='test_stderr' stdout='test_stdout' exit_code=4", logger_func=dsaz.LOG.error, - ) + ), + mock.call( + "Obtained DHCP lease on interface 'fakeEth0' " + "(primary=True driver='fake_driver' router='10.0.0.1' " + "routes=[('0.0.0.0/0', '10.0.0.1'), " + "('168.63.129.16/32', '10.0.0.1'), " + "('169.254.169.254/32', '10.0.0.1')] " + "lease={'interface': 'fakeEth0'} " + "imds_routed=True wireserver_routed=True)", + logger_func=dsaz.LOG.debug, + ), ] @pytest.mark.parametrize( @@ -3551,6 +3549,64 @@ assert error_reasons == [error_reason] * 3 +class TestCheckIfPrimary: + @pytest.mark.parametrize( + "static_routes", + [ + [("168.63.129.16/32", "10.0.0.1")], + [("169.254.169.254/32", "10.0.0.1")], + ], + ) + def test_primary(self, azure_ds, static_routes): + ephipv4 = ephemeral.EphemeralIPv4Network( + distro="ubuntu", + interface="eth0", + ip="10.0.0.4", + prefix_or_mask="32", + broadcast="255.255.255.255", + router="10.0.0.1", + static_routes=static_routes, + ) + + assert azure_ds._check_if_primary(ephipv4) is True + + def test_primary_via_wireserver_specified_in_option_245(self, azure_ds): + ephipv4 = ephemeral.EphemeralIPv4Network( + distro="ubuntu", + interface="eth0", + ip="10.0.0.4", + prefix_or_mask="32", + broadcast="255.255.255.255", + router="10.0.0.1", + static_routes=[("1.2.3.4/32", "10.0.0.1")], + ) + azure_ds._wireserver_endpoint = "1.2.3.4" + + assert azure_ds._check_if_primary(ephipv4) is True + + @pytest.mark.parametrize( + "static_routes", + [ + [], + [("0.0.0.0/0", "10.0.0.1")], + [("10.10.10.10/16", "10.0.0.1")], + ], + ) + def test_secondary(self, azure_ds, static_routes): + ephipv4 = ephemeral.EphemeralIPv4Network( + distro="ubuntu", + interface="eth0", + ip="10.0.0.4", + prefix_or_mask="32", + broadcast="255.255.255.255", + router="10.0.0.1", + static_routes=static_routes, + ) + azure_ds._wireserver_endpoint = "1.2.3.4" + + assert azure_ds._check_if_primary(ephipv4) is False + + class TestInstanceId: def test_metadata(self, azure_ds, mock_dmi_read_dmi_data): azure_ds.metadata = {"instance-id": "test-id"} @@ -3573,7 +3629,7 @@ mock_azure_get_metadata_from_fabric, mock_azure_report_failure_to_fabric, mock_net_dhcp_maybe_perform_dhcp_discovery, - mock_net_dhcp_EphemeralIPv4Network, + mock_ephemeral_ipv4_network, mock_dmi_read_dmi_data, mock_get_interfaces, mock_get_interface_mac, @@ -3582,6 +3638,7 @@ mock_netlink, mock_readurl, mock_subp_subp, + mock_timestamp, mock_util_ensure_dir, mock_util_find_devs_with, mock_util_load_file, @@ -3601,9 +3658,7 @@ self.mock_net_dhcp_maybe_perform_dhcp_discovery = ( mock_net_dhcp_maybe_perform_dhcp_discovery ) - self.mock_net_dhcp_EphemeralIPv4Network = ( - mock_net_dhcp_EphemeralIPv4Network - ) + self.mock_ephemeral_ipv4_network = mock_ephemeral_ipv4_network self.mock_dmi_read_dmi_data = mock_dmi_read_dmi_data self.mock_get_interfaces = mock_get_interfaces self.mock_get_interface_mac = mock_get_interface_mac @@ -3612,6 +3667,7 @@ self.mock_netlink = mock_netlink self.mock_readurl = mock_readurl self.mock_subp_subp = mock_subp_subp + self.mock_timestmp = mock_timestamp self.mock_util_ensure_dir = mock_util_ensure_dir self.mock_util_find_devs_with = mock_util_find_devs_with self.mock_util_load_file = mock_util_load_file @@ -3715,13 +3771,78 @@ assert len(self.mock_kvp_report_failure_to_host.mock_calls) == 0 assert len(self.mock_kvp_report_success_to_host.mock_calls) == 1 + @pytest.mark.parametrize("pps_type", ["Savable", "Running"]) + def test_stale_pps(self, pps_type): + imds_md_source = copy.deepcopy(self.imds_md) + imds_md_source["extended"]["compute"]["ppsType"] = pps_type + + nl_sock = mock.MagicMock() + self.mock_netlink.create_bound_netlink_socket.return_value = nl_sock + self.mock_readurl.side_effect = [ + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), + mock.MagicMock(contents=construct_ovf_env().encode()), + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), + ] + self.mock_azure_get_metadata_from_fabric.return_value = [] + + self.azure_ds._check_and_get_data() + + assert self.mock_readurl.mock_calls == [ + mock.call( + "http://169.254.169.254/metadata/instance?" + "api-version=2021-08-01&extended=true", + exception_cb=mock.ANY, + headers={"Metadata": "true"}, + infinite=True, + log_req_resp=True, + timeout=30, + ), + mock.call( + "http://169.254.169.254/metadata/reprovisiondata?" + "api-version=2019-06-01", + exception_cb=mock.ANY, + headers={"Metadata": "true"}, + log_req_resp=False, + infinite=True, + timeout=30, + ), + mock.call( + "http://169.254.169.254/metadata/instance?" + "api-version=2021-08-01&extended=true", + exception_cb=mock.ANY, + headers={"Metadata": "true"}, + infinite=True, + log_req_resp=True, + timeout=30, + ), + ] + + # Verify DMI usage. + assert self.mock_dmi_read_dmi_data.mock_calls == [ + mock.call("chassis-asset-tag"), + mock.call("system-uuid"), + mock.call("system-uuid"), + ] + + # Verify reports via KVP. + assert len(self.mock_kvp_report_success_to_host.mock_calls) == 1 + + assert self.mock_kvp_report_failure_to_host.mock_calls == [ + mock.call( + errors.ReportableErrorImdsInvalidMetadata( + key="extended.compute.ppsType", value=pps_type + ), + ), + ] + def test_running_pps(self): - self.imds_md["extended"]["compute"]["ppsType"] = "Running" + imds_md_source = copy.deepcopy(self.imds_md) + imds_md_source["extended"]["compute"]["ppsType"] = "Running" nl_sock = mock.MagicMock() self.mock_netlink.create_bound_netlink_socket.return_value = nl_sock self.mock_readurl.side_effect = [ - mock.MagicMock(contents=json.dumps(self.imds_md).encode()), + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), mock.MagicMock(contents=construct_ovf_env().encode()), mock.MagicMock(contents=json.dumps(self.imds_md).encode()), ] @@ -3824,7 +3945,8 @@ assert len(self.mock_kvp_report_success_to_host.mock_calls) == 2 def test_savable_pps(self): - self.imds_md["extended"]["compute"]["ppsType"] = "Savable" + imds_md_source = copy.deepcopy(self.imds_md) + imds_md_source["extended"]["compute"]["ppsType"] = "Savable" nl_sock = mock.MagicMock() self.mock_netlink.create_bound_netlink_socket.return_value = nl_sock @@ -3833,8 +3955,7 @@ "ethAttached1" ) self.mock_readurl.side_effect = [ - mock.MagicMock(contents=json.dumps(self.imds_md).encode()), - mock.MagicMock(contents=json.dumps(self.imds_md).encode()), + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), mock.MagicMock(contents=construct_ovf_env().encode()), mock.MagicMock(contents=json.dumps(self.imds_md).encode()), ] @@ -3853,15 +3974,6 @@ timeout=30, ), mock.call( - "http://169.254.169.254/metadata/instance?" - "api-version=2021-08-01&extended=true", - exception_cb=mock.ANY, - headers={"Metadata": "true"}, - infinite=True, - log_req_resp=True, - timeout=30, - ), - mock.call( "http://169.254.169.254/metadata/reprovisiondata?" "api-version=2019-06-01", exception_cb=mock.ANY, @@ -3884,7 +3996,11 @@ # Verify DHCP is setup twice. assert self.mock_wrapping_setup_ephemeral_networking.mock_calls == [ mock.call(timeout_minutes=20), - mock.call(iface="ethAttached1", timeout_minutes=20), + mock.call( + iface="ethAttached1", + timeout_minutes=20, + report_failure_if_not_primary=False, + ), ] assert self.mock_net_dhcp_maybe_perform_dhcp_discovery.mock_calls == [ mock.call( @@ -3968,7 +4084,8 @@ ], ) def test_savable_pps_early_unplug(self, fabric_side_effect): - self.imds_md["extended"]["compute"]["ppsType"] = "Savable" + imds_md_source = copy.deepcopy(self.imds_md) + imds_md_source["extended"]["compute"]["ppsType"] = "Savable" nl_sock = mock.MagicMock() self.mock_netlink.create_bound_netlink_socket.return_value = nl_sock @@ -3977,10 +4094,7 @@ "ethAttached1" ) self.mock_readurl.side_effect = [ - mock.MagicMock(contents=json.dumps(self.imds_md).encode()), - mock.MagicMock( - contents=json.dumps(self.imds_md["network"]).encode() - ), + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), mock.MagicMock(contents=construct_ovf_env().encode()), mock.MagicMock(contents=json.dumps(self.imds_md).encode()), ] @@ -3989,7 +4103,7 @@ ) # Fake DHCP teardown failure. - ipv4_net = self.mock_net_dhcp_EphemeralIPv4Network + ipv4_net = self.mock_ephemeral_ipv4_network ipv4_net.return_value.__exit__.side_effect = [ subp.ProcessExecutionError( cmd=["failed", "cmd"], @@ -4013,15 +4127,6 @@ timeout=30, ), mock.call( - "http://169.254.169.254/metadata/instance?" - "api-version=2021-08-01&extended=true", - exception_cb=mock.ANY, - headers={"Metadata": "true"}, - infinite=True, - log_req_resp=True, - timeout=30, - ), - mock.call( "http://169.254.169.254/metadata/reprovisiondata?" "api-version=2019-06-01", exception_cb=mock.ANY, @@ -4044,7 +4149,11 @@ # Verify DHCP is setup twice. assert self.mock_wrapping_setup_ephemeral_networking.mock_calls == [ mock.call(timeout_minutes=20), - mock.call(iface="ethAttached1", timeout_minutes=20), + mock.call( + iface="ethAttached1", + timeout_minutes=20, + report_failure_if_not_primary=False, + ), ] assert self.mock_net_dhcp_maybe_perform_dhcp_discovery.mock_calls == [ mock.call( @@ -4105,10 +4214,11 @@ @pytest.mark.parametrize("pps_type", ["Savable", "Running", "None"]) def test_recovery_pps(self, pps_type): self.patched_reported_ready_marker_path.write_text("") - self.imds_md["extended"]["compute"]["ppsType"] = pps_type + imds_md_source = copy.deepcopy(self.imds_md) + imds_md_source["extended"]["compute"]["ppsType"] = pps_type self.mock_readurl.side_effect = [ - mock.MagicMock(contents=json.dumps(self.imds_md).encode()), + mock.MagicMock(contents=json.dumps(imds_md_source).encode()), mock.MagicMock(contents=construct_ovf_env().encode()), mock.MagicMock(contents=json.dumps(self.imds_md).encode()), ] @@ -4291,8 +4401,52 @@ assert len(self.mock_kvp_report_failure_to_host.mock_calls) == 1 assert len(self.mock_kvp_report_success_to_host.mock_calls) == 1 + def test_imds_failure_results_in_provisioning_failure(self): + self.mock_readurl.side_effect = url_helper.UrlError( + requests.ConnectionError( + "Failed to establish a new connection: " + "[Errno 101] Network is unreachable" + ) + ) + + self.mock_azure_get_metadata_from_fabric.return_value = [] + + self.azure_ds._check_and_get_data() + + assert self.mock_readurl.mock_calls == [ + mock.call( + "http://169.254.169.254/metadata/instance?" + "api-version=2021-08-01&extended=true", + timeout=30, + headers={"Metadata": "true"}, + exception_cb=mock.ANY, + infinite=True, + log_req_resp=True, + ), + ] + + # Verify DHCP is setup once. + assert self.mock_wrapping_setup_ephemeral_networking.mock_calls == [ + mock.call(timeout_minutes=20) + ] + + # Verify reporting ready once. + assert self.mock_azure_get_metadata_from_fabric.mock_calls == [] + + # Verify netlink. + assert self.mock_netlink.mock_calls == [] + + # Verify no reported_ready marker written. + assert self.wrapped_util_write_file.mock_calls == [] + assert self.patched_reported_ready_marker_path.exists() is False + + # Verify reports via KVP. + assert len(self.mock_kvp_report_failure_to_host.mock_calls) == 1 + assert len(self.mock_kvp_report_success_to_host.mock_calls) == 0 + class TestGetMetadataFromImds: + @pytest.mark.parametrize("route_configured_for_imds", [False, True]) @pytest.mark.parametrize("report_failure", [False, True]) @pytest.mark.parametrize( "exception,reported_error_type", @@ -4314,30 +4468,51 @@ mock_azure_report_failure_to_fabric, mock_imds_fetch_metadata_with_api_fallback, mock_kvp_report_failure_to_host, + mock_time, monkeypatch, report_failure, reported_error_type, + route_configured_for_imds, ): monkeypatch.setattr( azure_ds, "_is_ephemeral_networking_up", lambda: True ) + azure_ds._route_configured_for_imds = route_configured_for_imds mock_imds_fetch_metadata_with_api_fallback.side_effect = exception + mock_time.return_value = 0.0 + max_connection_errors = None if route_configured_for_imds else 11 assert ( azure_ds.get_metadata_from_imds(report_failure=report_failure) == {} ) assert mock_imds_fetch_metadata_with_api_fallback.mock_calls == [ - mock.call(retry_deadline=mock.ANY) + mock.call( + max_connection_errors=max_connection_errors, + retry_deadline=mock.ANY, + ) ] + expected_duration = 300 + assert ( + mock_imds_fetch_metadata_with_api_fallback.call_args[1][ + "retry_deadline" + ] + == expected_duration + ) + reported_error = mock_kvp_report_failure_to_host.call_args[0][0] assert isinstance(reported_error, reported_error_type) assert reported_error.supporting_data["exception"] == repr(exception) assert mock_kvp_report_failure_to_host.mock_calls == [ mock.call(reported_error) ] - if report_failure: + + connection_error = isinstance( + exception, url_helper.UrlError + ) and isinstance(exception.cause, requests.ConnectionError) + report_skipped = not route_configured_for_imds and connection_error + if report_failure and not report_skipped: assert mock_azure_report_failure_to_fabric.mock_calls == [ mock.call(endpoint=mock.ANY, error=reported_error) ] diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_azure_helper.py cloud-init-23.4.3/tests/unittests/sources/test_azure_helper.py --- cloud-init-23.3.3/tests/unittests/sources/test_azure_helper.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_azure_helper.py 2024-02-02 21:21:52.000000000 +0000 @@ -1648,6 +1648,3 @@ azure_helper.OvfEnvXml.parse_text(ovf) assert str(exc_info.value) == error - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_common.py cloud-init-23.4.3/tests/unittests/sources/test_common.py --- cloud-init-23.3.3/tests/unittests/sources/test_common.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_common.py 2024-02-02 21:21:52.000000000 +0000 @@ -36,6 +36,7 @@ from tests.unittests import helpers as test_helpers DEFAULT_LOCAL = [ + AliYun.DataSourceAliYunLocal, Azure.DataSourceAzure, CloudSigma.DataSourceCloudSigma, ConfigDrive.DataSourceConfigDrive, @@ -44,6 +45,7 @@ Hetzner.DataSourceHetzner, IBMCloud.DataSourceIBMCloud, LXD.DataSourceLXD, + MAAS.DataSourceMAAS, NoCloud.DataSourceNoCloud, OpenNebula.DataSourceOpenNebula, Oracle.DataSourceOracle, diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_configdrive.py cloud-init-23.4.3/tests/unittests/sources/test_configdrive.py --- cloud-init-23.3.3/tests/unittests/sources/test_configdrive.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_configdrive.py 2024-02-02 21:21:52.000000000 +0000 @@ -1076,6 +1076,3 @@ cfg_ds._network_config = openstack.convert_net_json( cfg_ds.network_json, known_macs=KNOWN_MACS ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_digitalocean.py cloud-init-23.4.3/tests/unittests/sources/test_digitalocean.py --- cloud-init-23.3.3/tests/unittests/sources/test_digitalocean.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_digitalocean.py 2024-02-02 21:21:52.000000000 +0000 @@ -421,6 +421,3 @@ sorted(["45.55.249.133", "10.17.0.5"]), sorted([i["address"] for i in eth0["subnets"]]), ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_gce.py cloud-init-23.4.3/tests/unittests/sources/test_gce.py --- cloud-init-23.3.3/tests/unittests/sources/test_gce.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_gce.py 2024-02-02 21:21:52.000000000 +0000 @@ -489,6 +489,3 @@ ds = DataSourceGCE.DataSourceGCE(sys_cfg={}, distro=None, paths=None) ds._get_data() assert m_dhcp.call_count == 0 - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_ibmcloud.py cloud-init-23.4.3/tests/unittests/sources/test_ibmcloud.py --- cloud-init-23.3.3/tests/unittests/sources/test_ibmcloud.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_ibmcloud.py 2024-02-02 21:21:52.000000000 +0000 @@ -421,6 +421,3 @@ self.assertEqual("ibmcloud", self.ds.cloud_name) self.assertEqual("ibmcloud", self.ds.platform_type) self.assertEqual("plat (src)", self.ds.subplatform) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_init.py cloud-init-23.4.3/tests/unittests/sources/test_init.py --- cloud-init-23.3.3/tests/unittests/sources/test_init.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_init.py 2024-02-02 21:21:52.000000000 +0000 @@ -1091,6 +1091,3 @@ cloud_name="azure", region="!chinaeast", platform="platform" ), ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_maas.py cloud-init-23.4.3/tests/unittests/sources/test_maas.py --- cloud-init-23.3.3/tests/unittests/sources/test_maas.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_maas.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,26 +1,18 @@ # This file is part of cloud-init. See LICENSE file for license information. -import os -import shutil -import tempfile from copy import copy from unittest import mock +import pytest import yaml -from cloudinit import url_helper +from cloudinit import helpers, settings, url_helper from cloudinit.sources import DataSourceMAAS -from tests.unittests.helpers import CiTestCase, populate_dir +from tests.unittests.helpers import populate_dir -class TestMAASDataSource(CiTestCase): - def setUp(self): - super(TestMAASDataSource, self).setUp() - # Make a temp directoy for tests to use. - self.tmp = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, self.tmp) - - def test_seed_dir_valid(self): +class TestMAASDataSource: + def test_seed_dir_valid(self, tmpdir): """Verify a valid seeddir is read as such.""" userdata = b"valid01-userdata" @@ -31,20 +23,20 @@ "public-keys": "ssh-rsa AAAAB3Nz...aC1yc2E= keyname", } - my_d = os.path.join(self.tmp, "valid") + my_d = tmpdir.join("valid").strpath populate_dir(my_d, data) ud, md, vd = DataSourceMAAS.read_maas_seed_dir(my_d) - self.assertEqual(userdata, ud) + assert userdata == ud for key in ("instance-id", "local-hostname"): - self.assertEqual(data["meta-data/" + key], md[key]) + assert data["meta-data/" + key] == md[key] # verify that 'userdata' is not returned as part of the metadata - self.assertFalse(("user-data" in md)) - self.assertIsNone(vd) + assert "user-data" not in md + assert vd is None - def test_seed_dir_valid_extra(self): + def test_seed_dir_valid_extra(self, tmpdir): """Verify extra files do not affect seed_dir validity.""" userdata = b"valid-extra-userdata" @@ -55,19 +47,19 @@ "foo": "bar", } - my_d = os.path.join(self.tmp, "valid_extra") + my_d = tmpdir.join("valid_extra").strpath populate_dir(my_d, data) ud, md, _vd = DataSourceMAAS.read_maas_seed_dir(my_d) - self.assertEqual(userdata, ud) + assert userdata == ud for key in ("instance-id", "local-hostname"): - self.assertEqual(data["meta-data/" + key], md[key]) + assert data["meta-data/" + key] == md[key] # additional files should not just appear as keys in metadata atm - self.assertFalse(("foo" in md)) + assert "foo" not in md - def test_seed_dir_invalid(self): + def test_seed_dir_invalid(self, tmpdir): """Verify that invalid seed_dir raises MAASSeedDirMalformed.""" valid = { @@ -76,47 +68,37 @@ "user-data": "", } - my_based = os.path.join(self.tmp, "valid_extra") + my_based = tmpdir.join("valid_extra").strpath # missing 'userdata' file my_d = "%s-01" % my_based invalid_data = copy(valid) del invalid_data["local-hostname"] populate_dir(my_d, invalid_data) - self.assertRaises( - DataSourceMAAS.MAASSeedDirMalformed, - DataSourceMAAS.read_maas_seed_dir, - my_d, - ) + with pytest.raises(DataSourceMAAS.MAASSeedDirMalformed): + DataSourceMAAS.read_maas_seed_dir(my_d) # missing 'instance-id' my_d = "%s-02" % my_based invalid_data = copy(valid) del invalid_data["instance-id"] populate_dir(my_d, invalid_data) - self.assertRaises( - DataSourceMAAS.MAASSeedDirMalformed, - DataSourceMAAS.read_maas_seed_dir, - my_d, - ) + with pytest.raises(DataSourceMAAS.MAASSeedDirMalformed): + DataSourceMAAS.read_maas_seed_dir(my_d) - def test_seed_dir_none(self): + def test_seed_dir_none(self, tmpdir): """Verify that empty seed_dir raises MAASSeedDirNone.""" - my_d = os.path.join(self.tmp, "valid_empty") - self.assertRaises( - DataSourceMAAS.MAASSeedDirNone, - DataSourceMAAS.read_maas_seed_dir, - my_d, - ) + my_d = tmpdir.join("valid_extra").strpath + with pytest.raises(DataSourceMAAS.MAASSeedDirNone): + DataSourceMAAS.read_maas_seed_dir(my_d) - def test_seed_dir_missing(self): + def test_seed_dir_missing(self, tmpdir): """Verify that missing seed_dir raises MAASSeedDirNone.""" - self.assertRaises( - DataSourceMAAS.MAASSeedDirNone, - DataSourceMAAS.read_maas_seed_dir, - os.path.join(self.tmp, "nonexistantdirectory"), - ) + with pytest.raises(DataSourceMAAS.MAASSeedDirNone): + DataSourceMAAS.read_maas_seed_dir( + tmpdir.join("doesnotexist").strpath + ) def mock_read_maas_seed_url(self, data, seed, version="19991231"): """mock up readurl to appear as a web server at seed has provided data. @@ -141,7 +123,7 @@ mock_readurl.side_effect = my_readurl return DataSourceMAAS.read_maas_seed_url(seed, version=version) - def test_seed_url_valid(self): + def test_seed_url_valid(self, tmpdir): """Verify that valid seed_url is read as such.""" valid = { "meta-data/instance-id": "i-instanceid", @@ -154,14 +136,12 @@ my_ver = "1999-99-99" ud, md, vd = self.mock_read_maas_seed_url(valid, my_seed, my_ver) - self.assertEqual(valid["meta-data/instance-id"], md["instance-id"]) - self.assertEqual( - valid["meta-data/local-hostname"], md["local-hostname"] - ) - self.assertEqual(valid["meta-data/public-keys"], md["public-keys"]) - self.assertEqual(valid["user-data"], ud) + assert valid["meta-data/instance-id"] == md["instance-id"] + assert valid["meta-data/local-hostname"] == md["local-hostname"] + assert valid["meta-data/public-keys"] == md["public-keys"] + assert valid["user-data"] == ud # vendor-data is yaml, which decodes a string - self.assertEqual(valid["meta-data/vendor-data"].decode(), vd) + assert valid["meta-data/vendor-data"].decode() == vd def test_seed_url_vendor_data_dict(self): expected_vd = {"key1": "value1"} @@ -174,12 +154,76 @@ valid, "http://example.com/foo" ) - self.assertEqual(valid["meta-data/instance-id"], md["instance-id"]) - self.assertEqual(expected_vd, vd) + assert valid["meta-data/instance-id"] == md["instance-id"] + assert expected_vd == vd + + @pytest.mark.parametrize( + "initramfs_file, cmdline_value, expected", + ( + pytest.param( + None, + "no net in cmdline", + False, + id="no_maas_local_when_missing_run_files_and_cmdline", + ), + pytest.param( + "some initramfs cfg", + "no net in cmdline", + False, + id="no_maas_local_when_run_files_and_no_cmdline", + ), + pytest.param( + "some initframfs cfg", + "ip=dhcp cmdline config", + True, + id="maas_local_when_run_and_ip_cmdline_present", + ), + ), + ) + @pytest.mark.usefixtures("fake_filesystem") + @mock.patch("cloudinit.util.get_cmdline") + def tests_wb_local_stage_detects_datasource_on_initramfs_network( + self, cmdline, initramfs_file, cmdline_value, expected, tmpdir + ): + """Only detect local MAAS datasource net config is seen in initframfs + + MAAS Ephemeral launches provide initramfs network configuration to the + instance being launched. Assert that MAASDataSourceLocal can only be + discovered and detected when initramfs net config is applicable. + + The datasource needs to also prioritize the network config on disk + before initramfs configuration. Because the ephemeral instance + launches provide a custom cloud-config-url that will provide + network configuration to the VM which is more complete than the + initramfs config. + """ + cmdline.return_value = cmdline_value + ds = DataSourceMAAS.DataSourceMAASLocal( + settings.CFG_BUILTIN, None, helpers.Paths({"cloud_dir": tmpdir}) + ) + tmpdir.mkdir("run") + + # Create valid maas seed dir so parent DataSourceMAAS.get_data succeeds + tmpdir.mkdir("seed") + seed_dir = tmpdir.join("seed/maas") + seed_dir.mkdir() + userdata = b"valid01-userdata" + data = { + "meta-data/instance-id": "i-valid01", + "meta-data/local-hostname": "valid01-hostname", + "user-data": userdata, + "public-keys": "ssh-rsa AAAAB3Nz...aC1yc2E= keyname", + } + populate_dir(seed_dir.strpath, data) + + if initramfs_file: + klibc_net_cfg = tmpdir.join("run/net-eno.conf") + klibc_net_cfg.write(initramfs_file) + assert expected == ds.get_data() @mock.patch("cloudinit.sources.DataSourceMAAS.url_helper.OauthUrlHelper") -class TestGetOauthHelper(CiTestCase): +class TestGetOauthHelper: base_cfg = { "consumer_key": "FAKE_CONSUMER_KEY", "token_key": "FAKE_TOKEN_KEY", @@ -200,7 +244,7 @@ m_helper.assert_has_calls([mock.call(**self.base_cfg)]) -class TestGetIdHash(CiTestCase): +class TestGetIdHash: v1_cfg = { "consumer_key": "CKEY", "token_key": "TKEY", @@ -213,7 +257,7 @@ def test_v1_expected(self): """Test v1 id generated as expected working behavior from config.""" result = DataSourceMAAS.get_id_from_ds_cfg(self.v1_cfg.copy()) - self.assertEqual(self.v1_id, result) + assert self.v1_id == result def test_v1_extra_fields_are_ignored(self): """Test v1 id ignores unused entries in config.""" @@ -221,7 +265,4 @@ cfg["consumer_secret"] = "BOO" cfg["unrelated"] = "HI MOM" result = DataSourceMAAS.get_id_from_ds_cfg(cfg) - self.assertEqual(self.v1_id, result) - - -# vi: ts=4 expandtab + assert self.v1_id == result diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_opennebula.py cloud-init-23.4.3/tests/unittests/sources/test_opennebula.py --- cloud-init-23.3.3/tests/unittests/sources/test_opennebula.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_opennebula.py 2024-02-02 21:21:52.000000000 +0000 @@ -1066,6 +1066,3 @@ for k, v in variables.items(): data += "%s='%s'\n" % (k.upper(), v.replace(r"'", r"'\''")) populate_dir(path, {"context.sh": data}) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_ovf.py cloud-init-23.4.3/tests/unittests/sources/test_ovf.py --- cloud-init-23.3.3/tests/unittests/sources/test_ovf.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_ovf.py 2024-02-02 21:21:52.000000000 +0000 @@ -198,7 +198,6 @@ class TestDatasourceOVF(CiTestCase): - with_logs = True def setUp(self): @@ -452,9 +451,12 @@ rpctool = "vmware-rpctool" with_logs = True rpctool_path = "/not/important/vmware-rpctool" + vmtoolsd_path = "/not/important/vmtoolsd" - def test_without_vmware_rpctool_returns_notfound(self, m_subp, m_which): - m_which.return_value = None + def test_without_vmware_rpctool_and_without_vmtoolsd_returns_notfound( + self, m_subp, m_which + ): + m_which.side_effect = [None, None] self.assertEqual(NOT_FOUND, dsovf.transport_vmware_guestinfo()) self.assertEqual( 0, @@ -462,9 +464,21 @@ "subp should not be called if no rpctool in path.", ) + def test_without_vmware_rpctool_and_with_vmtoolsd_returns_found( + self, m_subp, m_which + ): + m_which.side_effect = [self.vmtoolsd_path, None] + m_subp.side_effect = [(fill_properties({}), "")] + self.assertNotEqual(NOT_FOUND, dsovf.transport_vmware_guestinfo()) + self.assertEqual( + 1, + m_subp.call_count, + "subp should be called once bc/ rpctool missing.", + ) + def test_notfound_on_exit_code_1(self, m_subp, m_which): """If vmware-rpctool exits 1, then must return not found.""" - m_which.return_value = self.rpctool_path + m_which.side_effect = [None, self.rpctool_path] m_subp.side_effect = subp.ProcessExecutionError( stdout="", stderr="No value found", exit_code=1, cmd=["unused"] ) @@ -483,14 +497,14 @@ rpctool would exit 1 with 'No value found' on stderr. But cover the case where it exited 0 and just wrote nothing to stdout. """ - m_which.return_value = self.rpctool_path + m_which.side_effect = [None, self.rpctool_path] m_subp.return_value = ("", "") self.assertEqual(NOT_FOUND, dsovf.transport_vmware_guestinfo()) self.assertEqual(1, m_subp.call_count) def test_notfound_and_warns_on_unexpected_exit_code(self, m_subp, m_which): """If vmware-rpctool exits non zero or 1, warnings should be logged.""" - m_which.return_value = self.rpctool_path + m_which.side_effect = [None, self.rpctool_path] m_subp.side_effect = subp.ProcessExecutionError( stdout=None, stderr="No value found", exit_code=2, cmd=["unused"] ) @@ -504,12 +518,37 @@ def test_found_when_guestinfo_present(self, m_subp, m_which): """When there is a ovf info, transport should return it.""" - m_which.return_value = self.rpctool_path + m_which.side_effect = [None, self.rpctool_path] content = fill_properties({}) m_subp.return_value = (content, "") self.assertEqual(content, dsovf.transport_vmware_guestinfo()) self.assertEqual(1, m_subp.call_count) + def test_vmware_rpctool_fails_and_vmtoolsd_fails(self, m_subp, m_which): + """When vmware-rpctool fails and vmtoolsd fails""" + m_which.side_effect = [self.vmtoolsd_path, self.rpctool_path] + m_subp.side_effect = [ + subp.ProcessExecutionError( + stdout="", stderr="No value found", exit_code=1, cmd=["unused"] + ), + subp.ProcessExecutionError( + stdout="", stderr="No value found", exit_code=1, cmd=["unused"] + ), + ] + self.assertEqual(NOT_FOUND, dsovf.transport_vmware_guestinfo()) + self.assertEqual(2, m_subp.call_count) + + def test_vmware_rpctool_fails_and_vmtoolsd_success(self, m_subp, m_which): + """When vmware-rpctool fails but vmtoolsd succeeds""" + m_which.side_effect = [self.vmtoolsd_path, self.rpctool_path] + m_subp.side_effect = [ + subp.ProcessExecutionError( + stdout="", stderr="No value found", exit_code=1, cmd=["unused"] + ), + (fill_properties({}), ""), + ] + self.assertNotEqual(NOT_FOUND, dsovf.transport_vmware_guestinfo()) + self.assertEqual(2, m_subp.call_count) + # -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_smartos.py cloud-init-23.4.3/tests/unittests/sources/test_smartos.py --- cloud-init-23.3.3/tests/unittests/sources/test_smartos.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_smartos.py 2024-02-02 21:21:52.000000000 +0000 @@ -1409,6 +1409,3 @@ client.get(key) self.assertIsNone(self.mdata_proc.exitcode) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/test_vmware.py cloud-init-23.4.3/tests/unittests/sources/test_vmware.py --- cloud-init-23.3.3/tests/unittests/sources/test_vmware.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/test_vmware.py 2024-02-02 21:21:52.000000000 +0000 @@ -16,6 +16,7 @@ from cloudinit import dmi, helpers, safeyaml, settings, util from cloudinit.sources import DataSourceVMware from cloudinit.sources.helpers.vmware.imc import guestcust_util +from cloudinit.subp import ProcessExecutionError from tests.unittests.helpers import ( CiTestCase, FilesystemMockingTestCase, @@ -100,7 +101,6 @@ def test_no_data_access_method(self): ds = get_ds(self.tmp) - ds.vmware_rpctool = None with mock.patch( "cloudinit.sources.DataSourceVMware.is_vmware_platform", return_value=False, @@ -252,7 +252,6 @@ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ds = get_ds(self.tmp) - ds.vmware_rpctool = None ret = ds.get_data() self.assertTrue(ret) self.assertEqual(m_fn_call_count, m_fn.call_count) @@ -381,7 +380,6 @@ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ds = get_ds(self.tmp) - ds.vmware_rpctool = "vmware-rpctool" ret = ds.get_data() self.assertTrue(ret) self.assertEqual(m_fn_call_count, m_fn.call_count) @@ -400,7 +398,9 @@ self.assertEqual(system_type, PRODUCT_NAME) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_subplatform(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_subplatform(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ds = self.assert_get_data_ok(m_fn, m_fn_call_count=4) self.assertEqual( @@ -413,17 +413,53 @@ ) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_userdata_only(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_with_vmware_rpctool(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] + m_fn.side_effect = [VMW_METADATA_YAML, "", "", ""] + self.assert_get_data_ok(m_fn, m_fn_call_count=4) + + @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") + @mock.patch("cloudinit.sources.DataSourceVMware.exec_vmware_rpctool") + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_non_zero_exit_code_fallback_to_vmtoolsd( + self, m_which_fn, m_exec_vmware_rpctool_fn, m_fn + ): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] + m_exec_vmware_rpctool_fn.side_effect = ProcessExecutionError( + exit_code=1 + ) + m_fn.side_effect = [VMW_METADATA_YAML, "", "", ""] + self.assert_get_data_ok(m_fn, m_fn_call_count=4) + + @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") + @mock.patch("cloudinit.sources.DataSourceVMware.exec_vmware_rpctool") + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_vmware_rpctool_not_found_fallback_to_vmtoolsd( + self, m_which_fn, m_exec_vmware_rpctool_fn, m_fn + ): + m_which_fn.side_effect = ["vmtoolsd", None] + m_fn.side_effect = [VMW_METADATA_YAML, "", "", ""] + self.assert_get_data_ok(m_fn, m_fn_call_count=4) + + @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_userdata_only(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] m_fn.side_effect = ["", VMW_USERDATA_YAML, "", ""] self.assert_get_data_ok(m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_vendordata_only(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_vendordata_only(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] m_fn.side_effect = ["", "", VMW_VENDORDATA_YAML, ""] self.assert_get_data_ok(m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_metadata_single_ssh_key(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_metadata_single_ssh_key(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) metadata["public_keys"] = VMW_SINGLE_KEY metadata_yaml = safeyaml.dumps(metadata) @@ -431,7 +467,9 @@ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_metadata_multiple_ssh_keys(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_metadata_multiple_ssh_keys(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) metadata["public_keys"] = VMW_MULTIPLE_KEYS metadata_yaml = safeyaml.dumps(metadata) @@ -439,19 +477,25 @@ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_metadata_base64(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_base64(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) m_fn.side_effect = [data, "base64", "", ""] self.assert_get_data_ok(m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_metadata_b64(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_b64(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) m_fn.side_effect = [data, "b64", "", ""] self.assert_get_data_ok(m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_metadata_gzip_base64(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_gzip_base64(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] data = VMW_METADATA_YAML.encode("utf-8") data = gzip.compress(data) data = base64.b64encode(data) @@ -459,7 +503,9 @@ self.assert_get_data_ok(m_fn, m_fn_call_count=4) @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") - def test_get_data_metadata_gz_b64(self, m_fn): + @mock.patch("cloudinit.sources.DataSourceVMware.which") + def test_get_data_metadata_gz_b64(self, m_which_fn, m_fn): + m_which_fn.side_effect = ["vmtoolsd", "vmware-rpctool"] data = VMW_METADATA_YAML.encode("utf-8") data = gzip.compress(data) data = base64.b64encode(data) @@ -494,7 +540,6 @@ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ds = get_ds(self.tmp) - ds.vmware_rpctool = "vmware-rpctool" ret = ds.get_data() self.assertFalse(ret) @@ -1217,8 +1262,4 @@ ds = DataSourceVMware.DataSourceVMware( settings.CFG_BUILTIN, None, helpers.Paths({"run_dir": temp_dir}) ) - ds.vmware_rpctool = "vmware-rpctool" return ds - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/vmware/test_custom_script.py cloud-init-23.4.3/tests/unittests/sources/vmware/test_custom_script.py --- cloud-init-23.3.3/tests/unittests/sources/vmware/test_custom_script.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/vmware/test_custom_script.py 2024-02-02 21:21:52.000000000 +0000 @@ -109,6 +109,3 @@ content, "This is the script to run post cust" ) self.assertTrue(os.path.exists(markerFile)) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/vmware/test_guestcust_util.py cloud-init-23.4.3/tests/unittests/sources/vmware/test_guestcust_util.py --- cloud-init-23.3.3/tests/unittests/sources/vmware/test_guestcust_util.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/vmware/test_guestcust_util.py 2024-02-02 21:21:52.000000000 +0000 @@ -107,6 +107,3 @@ ["vmware-rpctool", "info-set guestinfo.gc.status Successful"], rcs=[0], ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/sources/vmware/test_vmware_config_file.py cloud-init-23.4.3/tests/unittests/sources/vmware/test_vmware_config_file.py --- cloud-init-23.3.3/tests/unittests/sources/vmware/test_vmware_config_file.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/sources/vmware/test_vmware_config_file.py 2024-02-02 21:21:52.000000000 +0000 @@ -625,6 +625,3 @@ cf._insertKey("CLOUDINIT|USERDATA", "test-userdata") conf = Config(cf) self.assertEqual("test-userdata", conf.user_data_name) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test__init__.py cloud-init-23.4.3/tests/unittests/test__init__.py --- cloud-init-23.3.3/tests/unittests/test__init__.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test__init__.py 2024-02-02 21:21:52.000000000 +0000 @@ -341,6 +341,3 @@ assert logging.WARN == lvl assert url in msg assert False is os.path.exists(fpath) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_apport.py cloud-init-23.4.3/tests/unittests/test_apport.py --- cloud-init-23.3.3/tests/unittests/test_apport.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_apport.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,27 +1,15 @@ import os +import sys +from importlib import reload import pytest +from cloudinit import apport from tests.unittests.helpers import mock M_PATH = "cloudinit.apport." -@pytest.fixture() -def apport(request, mocker, paths): - """Mock apport.hookutils before importing cloudinit.apport. - - This avoids our optional import dependency on apport, providing tests with - mocked apport.hookutils function call counts. - """ - m_hookutils = mock.Mock() - mocker.patch.dict("sys.modules", {"apport.hookutils": m_hookutils}) - mocker.patch(M_PATH + "read_cfg_paths", return_value=paths) - from cloudinit import apport - - yield apport - - class TestApport: @pytest.mark.parametrize( "instance_data,choice_idx,expected_report", @@ -45,10 +33,10 @@ ), ) def test_attach_cloud_info( - self, instance_data, choice_idx, expected_report, apport, paths + self, instance_data, choice_idx, expected_report, mocker, paths ): """Prompt for cloud name when instance-data.json is not-json/absent.""" - + mocker.patch(M_PATH + "read_cfg_paths", return_value=paths) instance_data_file = paths.get_runpath("instance_data") if instance_data is None: assert not os.path.exists(instance_data_file) @@ -66,11 +54,16 @@ else: assert ui.choice.call_count == 0 - def test_attach_user_data(self, apport, paths): + def test_attach_user_data(self, mocker, paths): user_data_file = paths.get_ipath_cur("userdata_raw") ui = mock.Mock() ui.yesno.return_value = True report = object() + m_hookutils = mock.Mock() + + with mock.patch.dict(sys.modules, {"apport.hookutils": m_hookutils}): + reload(sys.modules["cloudinit.apport"]) + mocker.patch(M_PATH + "read_cfg_paths", return_value=paths) apport.attach_user_data(report, ui) assert [ mock.call(report, user_data_file, "user_data.txt"), @@ -104,8 +97,34 @@ ), ), ) - def test_add_bug_tags_assigns_proper_tags(self, report, tags, apport): + def test_add_bug_tags_assigns_proper_tags(self, report, tags): """Tags are assigned based on non-empty project report key values.""" apport.add_bug_tags(report) assert report.get("Tags", "") == tags + + @mock.patch(M_PATH + "os.path.exists", return_value=True) + def test_attach_ubuntu_pro_info(self, m_exists): + m_hookutils = mock.Mock() + with mock.patch.dict(sys.modules, {"apport.hookutils": m_hookutils}): + reload(sys.modules["cloudinit.apport"]) + report = {} + apport.attach_ubuntu_pro_info(report) + + assert [ + mock.call(report, "/var/log/ubuntu-advantage.log"), + ] == m_hookutils.attach_file_if_exists.call_args_list + assert report.get("Tags", "") == "ubuntu-pro" + + @mock.patch(M_PATH + "os.path.exists", return_value=False) + def test_attach_ubuntu_pro_info_log_non_present(self, m_exists): + m_hookutils = mock.Mock() + with mock.patch.dict(sys.modules, {"apport.hookutils": m_hookutils}): + reload(sys.modules["cloudinit.apport"]) + report = {} + apport.attach_ubuntu_pro_info(report) + + assert [ + mock.call(report, "/var/log/ubuntu-advantage.log"), + ] == m_hookutils.attach_file_if_exists.call_args_list + assert report.get("Tags", "") == "" diff -Nru cloud-init-23.3.3/tests/unittests/test_atomic_helper.py cloud-init-23.4.3/tests/unittests/test_atomic_helper.py --- cloud-init-23.3.3/tests/unittests/test_atomic_helper.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_atomic_helper.py 2024-02-02 21:21:52.000000000 +0000 @@ -63,6 +63,3 @@ def check_perms(self, path, perms): file_stat = os.stat(path) self.assertEqual(perms, stat.S_IMODE(file_stat.st_mode)) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_builtin_handlers.py cloud-init-23.4.3/tests/unittests/test_builtin_handlers.py --- cloud-init-23.3.3/tests/unittests/test_builtin_handlers.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_builtin_handlers.py 2024-02-02 21:21:52.000000000 +0000 @@ -11,6 +11,7 @@ from cloudinit import atomic_helper, handlers, helpers, util from cloudinit.cmd.devel import read_cfg_paths +from cloudinit.handlers.boot_hook import BootHookPartHandler from cloudinit.handlers.cloud_config import CloudConfigPartHandler from cloudinit.handlers.jinja_template import ( JinjaLoadError, @@ -25,6 +26,7 @@ ) from cloudinit.settings import PER_ALWAYS, PER_INSTANCE, PER_ONCE from tests.unittests.helpers import CiTestCase, mock, skipUnlessJinja +from tests.unittests.util import FakeDataSource INSTANCE_DATA_FILE = "instance-data-sensitive.json" @@ -429,4 +431,25 @@ self.do_test_frequency(PER_ONCE) -# vi: ts=4 expandtab +@pytest.mark.allow_all_subp +@pytest.mark.usefixtures("fake_filesystem") +class TestBootHookHandler: + def test_handle_part(self, paths, tmpdir, capfd): + paths.get_ipath = paths.get_ipath_cur + datasource = FakeDataSource(paths=paths) + handler = BootHookPartHandler(paths=paths, datasource=datasource) + # Setup /dev/null file for supb because no data param present + tmpdir.mkdir("/dev/") + tmpdir.join("dev/null").write("") + assert handler.boothook_dir == f"{tmpdir}/cloud_dir/instance/boothooks" + payload = f"#!/bin/sh\necho id:$INSTANCE_ID | tee {tmpdir}/boothook\n" + handler.handle_part( + data="dontcare", + ctype="text/cloud-boothook", + filename="part-001", + payload=payload, + frequency=None, + ) + assert payload == util.load_file(f"{handler.boothook_dir}/part-001") + assert "id:i-testing\n" == util.load_file(f"{tmpdir}/boothook") + assert "id:i-testing\n" == capfd.readouterr().out diff -Nru cloud-init-23.3.3/tests/unittests/test_cli.py cloud-init-23.4.3/tests/unittests/test_cli.py --- cloud-init-23.3.3/tests/unittests/test_cli.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_cli.py 2024-02-02 21:21:52.000000000 +0000 @@ -7,9 +7,8 @@ import pytest -from cloudinit import helpers +from cloudinit import helpers, log from cloudinit.cmd import main as cli -from cloudinit.util import load_file, load_json from tests.unittests import helpers as test_helpers mock = test_helpers.mock @@ -30,6 +29,7 @@ if not sysv_args: sysv_args = ["cloud-init"] try: + log.setup_logging() return cli.main(sysv_args=sysv_args) except SystemExit as e: return e.code @@ -62,14 +62,18 @@ cli.status_wrapper(name, myargs, data_d, link_d) assert [] == my_action.call_args_list - def test_status_wrapper_init_local_writes_fresh_status_info(self, tmpdir): + @mock.patch("cloudinit.cmd.main.atomic_helper.write_json") + def test_status_wrapper_init_local_writes_fresh_status_info( + self, + m_json, + tmpdir, + ): """When running in init-local mode, status_wrapper writes status.json. Old status and results artifacts are also removed. """ data_d = tmpdir.join("data") link_d = tmpdir.join("link") - status_link = link_d.join("status.json") # Write old artifacts which will be removed or updated. for _dir in data_d, link_d: test_helpers.populate_dir( @@ -85,7 +89,15 @@ myargs = FakeArgs(("ignored_name", myaction), True, "bogusmode") cli.status_wrapper("init", myargs, data_d, link_d) # No errors reported in status - status_v1 = load_json(load_file(status_link))["v1"] + status_v1 = m_json.call_args_list[1][0][1]["v1"] + assert status_v1.keys() == { + "datasource", + "init-local", + "init", + "modules-config", + "modules-final", + "stage", + } assert ["an error"] == status_v1["init-local"]["errors"] assert "SomeDatasource" == status_v1["datasource"] assert False is os.path.exists( @@ -95,7 +107,10 @@ link_d.join("result.json") ), "unexpected result.json link found" - def test_status_wrapper_init_local_honor_cloud_dir(self, mocker, tmpdir): + @mock.patch("cloudinit.cmd.main.atomic_helper.write_json") + def test_status_wrapper_init_local_honor_cloud_dir( + self, m_json, mocker, tmpdir + ): """When running in init-local mode, status_wrapper honors cloud_dir.""" cloud_dir = tmpdir.join("cloud") paths = helpers.Paths({"cloud_dir": str(cloud_dir)}) @@ -111,8 +126,9 @@ myargs = FakeArgs(("ignored_name", myaction), True, "bogusmode") cli.status_wrapper("init", myargs, link_d=link_d) # No explicit data_d + # Access cloud_dir directly - status_v1 = load_json(load_file(data_d.join("status.json")))["v1"] + status_v1 = m_json.call_args_list[1][0][1]["v1"] assert ["an_error"] == status_v1["init-local"]["errors"] assert "SomeDatasource" == status_v1["datasource"] assert False is os.path.exists( @@ -235,7 +251,7 @@ assert ( "Error:\n" "Expected one of --config-file, --system or --docs arguments\n" - == err + in err ) @pytest.mark.parametrize( diff -Nru cloud-init-23.3.3/tests/unittests/test_data.py cloud-init-23.4.3/tests/unittests/test_data.py --- cloud-init-23.3.3/tests/unittests/test_data.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_data.py 2024-02-02 21:21:52.000000000 +0000 @@ -475,6 +475,7 @@ EXPIRE_APPLIES_TO_HASHED_USERS=False, NETPLAN_CONFIG_ROOT_READ_ONLY=True, NOCLOUD_SEED_URL_APPEND_FORWARD_SLASH=False, + APT_DEB822_SOURCE_LIST_FILE=True, ) def test_shellscript(self, init_tmp, tmpdir, caplog): """Raw text starting #!/bin/sh is treated as script.""" @@ -505,6 +506,7 @@ "EXPIRE_APPLIES_TO_HASHED_USERS": False, "NETPLAN_CONFIG_ROOT_READ_ONLY": True, "NOCLOUD_SEED_URL_APPEND_FORWARD_SLASH": False, + "APT_DEB822_SOURCE_LIST_FILE": True, }, "system_info": { "default_user": {"name": "ubuntu"}, @@ -897,6 +899,3 @@ "key3": "builtin3", "keyconfd1": "kconfd1", } - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_ds_identify.py cloud-init-23.4.3/tests/unittests/test_ds_identify.py --- cloud-init-23.3.3/tests/unittests/test_ds_identify.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_ds_identify.py 2024-02-02 21:21:52.000000000 +0000 @@ -6,6 +6,8 @@ from textwrap import dedent from uuid import uuid4 +import pytest + from cloudinit import atomic_helper, safeyaml, subp, util from cloudinit.sources import DataSourceIBMCloud as ds_ibm from cloudinit.sources import DataSourceOracle as ds_oracle @@ -186,6 +188,10 @@ "err": "No dmidecode program. ERROR.", }, { + "name": "is_disabled", + "ret": 1, + }, + { "name": "get_kenv_field", "ret": 1, "err": "No kenv program. ERROR.", @@ -301,6 +307,65 @@ for var in expected_vars: self.assertIn("{0}=".format(var), err) + @pytest.mark.xfail(reason="GH-4796") + def test_maas_not_detected_1(self): + """Don't incorrectly identify maas + + In ds-identify the function check_config() attempts to parse yaml keys + in bash, but it sometimes introduces false positives. The maas + datasource uses check_config() and the existence of a "MAAS" key to + identify itself (which is a very poor identifier - clouds should have + stricter identifiers). Since the MAAS datasource is at the begining of + the list, this is particularly troublesome and more concerning than + NoCloud false positives, for example. + config = "LXD-kvm-not-MAAS-1" + self._test_ds_found(config) + """ + + def test_maas_not_detected_2(self): + """Don't incorrectly identify maas + + The bug reported in 4794 combined with the previously existing bug + reported in 4796 made for very loose MAAS false-positives. + + In ds-identify the function check_config() attempts to parse yaml keys + in bash, but it sometimes introduces false positives. The maas + datasource uses check_config() and the existence of a "MAAS" key to + identify itself (which is a very poor identifier - clouds should have + stricter identifiers). Since the MAAS datasource is at the begining of + the list, this is particularly troublesome and more concerning than + NoCloud false positives, for example. + """ + config = "LXD-kvm-not-MAAS-2" + self._test_ds_found(config) + + def test_maas_not_detected_3(self): + """Don't incorrectly identify maas + + The bug reported in 4794 combined with the previously existing bug + reported in 4796 made for very loose MAAS false-positives. + + In ds-identify the function check_config() attempts to parse yaml keys + in bash, but it sometimes introduces false positives. The maas + datasource uses check_config() and the existence of a "MAAS" key to + identify itself (which is a very poor identifier - clouds should have + stricter identifiers). Since the MAAS datasource is at the begining of + the list, this is particularly troublesome and more concerning than + NoCloud false positives, for example. + """ + config = "LXD-kvm-not-MAAS-3" + self._test_ds_found(config) + + def test_azure_invalid_configuration(self): + """Don't detect incorrect config when invalid datasource_list provided + + If unparsable list is provided we just ignore it. Some users + might assume that since the rest of the configuration is yaml that + multi-line yaml lists are valid (they aren't). When this happens, just + run ds-identify and figure it out for ourselves which platform to run. + """ + self._test_ds_found("Azure-parse-invalid") + def test_azure_dmi_detection_from_chassis_asset_tag(self): """Azure datasource is detected from DMI chassis-asset-tag""" self._test_ds_found("Azure-dmi-detection") @@ -869,7 +934,8 @@ def test_vmware_guestinfo_no_data(self): """VMware: guestinfo transport no data""" - self._test_ds_not_found("VMware-GuestInfo-NoData") + self._test_ds_not_found("VMware-GuestInfo-NoData-Rpctool") + self._test_ds_not_found("VMware-GuestInfo-NoData-Vmtoolsd") def test_vmware_guestinfo_no_virt_id(self): """VMware: guestinfo transport fails if no virt id""" @@ -1074,6 +1140,15 @@ os.path.join(P_SEED_DIR, "azure", "ovf-env.xml"): "present\n", }, }, + "Azure-parse-invalid": { + "ds": "Azure", + "files": { + P_CHASSIS_ASSET_TAG: "7783-7084-3265-9085-8269-3286-77\n", + "etc/cloud/cloud.cfg.d/91-azure_datasource.cfg": ( + "datasource_list:\n - Azure" + ), + }, + }, "Ec2-hvm": { "ds": "Ec2", "mocks": [{"name": "detect_virt", "RET": "kvm", "ret": 0}], @@ -1114,6 +1189,38 @@ "mocks": [{"name": "is_socket_file", "ret": 1}, MOCK_VIRT_IS_KVM], "no_mocks": ["dscheck_LXD"], # Don't default mock dscheck_LXD }, + "LXD-kvm-not-MAAS-1": { + "ds": "LXD", + "files": { + P_BOARD_NAME: "LXD\n", + "etc/cloud/cloud.cfg.d/92-broken-maas.cfg": ( + "datasource:\n MAAS:\n metadata_urls: [ 'blah.com' ]" + ), + }, + # /dev/lxd/sock does not exist and KVM virt-type + "mocks": [{"name": "is_socket_file", "ret": 1}, MOCK_VIRT_IS_KVM], + "no_mocks": ["dscheck_LXD"], # Don't default mock dscheck_LXD + }, + "LXD-kvm-not-MAAS-2": { + "ds": "LXD", + "files": { + P_BOARD_NAME: "LXD\n", + "etc/cloud/cloud.cfg.d/92-broken-maas.cfg": ("#MAAS: None"), + }, + # /dev/lxd/sock does not exist and KVM virt-type + "mocks": [{"name": "is_socket_file", "ret": 1}, MOCK_VIRT_IS_KVM], + "no_mocks": ["dscheck_LXD"], # Don't default mock dscheck_LXD + }, + "LXD-kvm-not-MAAS-3": { + "ds": "LXD", + "files": { + P_BOARD_NAME: "LXD\n", + "etc/cloud/cloud.cfg.d/92-broken-maas.cfg": ("MAAS: None"), + }, + # /dev/lxd/sock does not exist and KVM virt-type + "mocks": [{"name": "is_socket_file", "ret": 1}, MOCK_VIRT_IS_KVM], + "no_mocks": ["dscheck_LXD"], # Don't default mock dscheck_LXD + }, "LXD-kvm-qemu-kernel-gt-5.10": { # LXD host > 5.10 kvm launch virt==qemu "ds": "LXD", "files": {P_BOARD_NAME: "LXD\n"}, @@ -1653,6 +1760,11 @@ "ret": 0, "out": "/usr/bin/vmware-rpctool", }, + { + "name": "vmware_has_vmtoolsd", + "ret": 1, + "out": "/usr/bin/vmtoolsd", + }, ], "files": { # Setup vmware customization enabled @@ -1769,7 +1881,7 @@ MOCK_VIRT_IS_VMWARE, ], }, - "VMware-GuestInfo-NoData": { + "VMware-GuestInfo-NoData-Rpctool": { "ds": "VMware", "mocks": [ { @@ -1778,15 +1890,49 @@ "out": "/usr/bin/vmware-rpctool", }, { - "name": "vmware_rpctool_guestinfo_metadata", + "name": "vmware_has_vmtoolsd", + "ret": 1, + "out": "/usr/bin/vmtoolsd", + }, + { + "name": "vmware_guestinfo_metadata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_userdata", + "name": "vmware_guestinfo_userdata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_vendordata", + "name": "vmware_guestinfo_vendordata", + "ret": 1, + }, + MOCK_VIRT_IS_VMWARE, + ], + }, + "VMware-GuestInfo-NoData-Vmtoolsd": { + "ds": "VMware", + "policy_dmi": POLICY_FOUND_ONLY, + "mocks": [ + { + "name": "vmware_has_rpctool", + "ret": 1, + "out": "/usr/bin/vmware-rpctool", + }, + { + "name": "vmware_has_vmtoolsd", + "ret": 0, + "out": "/usr/bin/vmtoolsd", + }, + { + "name": "vmware_guestinfo_metadata", + "ret": 1, + }, + { + "name": "vmware_guestinfo_userdata", + "ret": 1, + }, + { + "name": "vmware_guestinfo_vendordata", "ret": 1, }, MOCK_VIRT_IS_VMWARE, @@ -1801,16 +1947,16 @@ "out": "/usr/bin/vmware-rpctool", }, { - "name": "vmware_rpctool_guestinfo_metadata", + "name": "vmware_guestinfo_metadata", "ret": 0, "out": "---", }, { - "name": "vmware_rpctool_guestinfo_userdata", + "name": "vmware_guestinfo_userdata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_vendordata", + "name": "vmware_guestinfo_vendordata", "ret": 1, }, ], @@ -1820,20 +1966,25 @@ "mocks": [ { "name": "vmware_has_rpctool", - "ret": 0, + "ret": 1, "out": "/usr/bin/vmware-rpctool", }, { - "name": "vmware_rpctool_guestinfo_metadata", + "name": "vmware_has_vmtoolsd", + "ret": 0, + "out": "/usr/bin/vmtoolsd", + }, + { + "name": "vmware_guestinfo_metadata", "ret": 0, "out": "---", }, { - "name": "vmware_rpctool_guestinfo_userdata", + "name": "vmware_guestinfo_userdata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_vendordata", + "name": "vmware_guestinfo_vendordata", "ret": 1, }, MOCK_VIRT_IS_VMWARE, @@ -1848,16 +1999,21 @@ "out": "/usr/bin/vmware-rpctool", }, { - "name": "vmware_rpctool_guestinfo_metadata", + "name": "vmware_has_vmtoolsd", "ret": 1, + "out": "/usr/bin/vmtoolsd", }, { - "name": "vmware_rpctool_guestinfo_userdata", + "name": "vmware_guestinfo_metadata", + "ret": 1, + }, + { + "name": "vmware_guestinfo_userdata", "ret": 0, "out": "---", }, { - "name": "vmware_rpctool_guestinfo_vendordata", + "name": "vmware_guestinfo_vendordata", "ret": 1, }, MOCK_VIRT_IS_VMWARE, @@ -1868,19 +2024,24 @@ "mocks": [ { "name": "vmware_has_rpctool", - "ret": 0, + "ret": 1, "out": "/usr/bin/vmware-rpctool", }, { - "name": "vmware_rpctool_guestinfo_metadata", + "name": "vmware_has_vmtoolsd", + "ret": 0, + "out": "/usr/bin/vmtoolsd", + }, + { + "name": "vmware_guestinfo_metadata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_userdata", + "name": "vmware_guestinfo_userdata", "ret": 1, }, { - "name": "vmware_rpctool_guestinfo_vendordata", + "name": "vmware_guestinfo_vendordata", "ret": 0, "out": "---", }, @@ -1909,5 +2070,3 @@ }, }, } - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_features.py cloud-init-23.4.3/tests/unittests/test_features.py --- cloud-init-23.3.3/tests/unittests/test_features.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_features.py 2024-02-02 21:21:52.000000000 +0000 @@ -20,6 +20,7 @@ EXPIRE_APPLIES_TO_HASHED_USERS=False, NETPLAN_CONFIG_ROOT_READ_ONLY=True, NOCLOUD_SEED_URL_APPEND_FORWARD_SLASH=False, + APT_DEB822_SOURCE_LIST_FILE=True, ): assert { "ERROR_ON_USER_DATA_FAILURE": True, @@ -27,4 +28,5 @@ "EXPIRE_APPLIES_TO_HASHED_USERS": False, "NETPLAN_CONFIG_ROOT_READ_ONLY": True, "NOCLOUD_SEED_URL_APPEND_FORWARD_SLASH": False, + "APT_DEB822_SOURCE_LIST_FILE": True, } == features.get_features() diff -Nru cloud-init-23.3.3/tests/unittests/test_helpers.py cloud-init-23.4.3/tests/unittests/test_helpers.py --- cloud-init-23.3.3/tests/unittests/test_helpers.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_helpers.py 2024-02-02 21:21:52.000000000 +0000 @@ -64,6 +64,3 @@ == test_helpers.cloud_init_project_dir("test") == str(Path(self._get_top_level_dir_alt_implementation(), "test")) ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_log.py cloud-init-23.4.3/tests/unittests/test_log.py --- cloud-init-23.3.3/tests/unittests/test_log.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_log.py 2024-02-02 21:21:52.000000000 +0000 @@ -7,25 +7,24 @@ import logging import time -from cloudinit import log as ci_logging -from cloudinit import util +from cloudinit import log, util from cloudinit.analyze.dump import CLOUD_INIT_ASCTIME_FMT from tests.unittests.helpers import CiTestCase class TestCloudInitLogger(CiTestCase): def setUp(self): - # set up a logger like cloud-init does in setupLogging, but instead + # set up a logger like cloud-init does in setup_logging, but instead # of sys.stderr, we'll plug in a StringIO() object so we can see # what gets logged logging.Formatter.converter = time.gmtime self.ci_logs = io.StringIO() self.ci_root = logging.getLogger() console = logging.StreamHandler(self.ci_logs) - console.setFormatter(logging.Formatter(ci_logging.DEF_CON_FORMAT)) - console.setLevel(ci_logging.DEBUG) + console.setFormatter(logging.Formatter(log.DEFAULT_LOG_FORMAT)) + console.setLevel(logging.DEBUG) self.ci_root.addHandler(console) - self.ci_root.setLevel(ci_logging.DEBUG) + self.ci_root.setLevel(logging.DEBUG) self.LOG = logging.getLogger("test_cloudinit_logger") def test_logger_uses_gmtime(self): @@ -62,14 +61,14 @@ class TestDeprecatedLogs: def test_deprecated_log_level(self, caplog): - ci_logging.setupLogging() - log = ci_logging.getLogger() - log.deprecated("deprecated message") + logger = logging.getLogger() + log.setup_logging() + logger.deprecated("deprecated message") assert "DEPRECATED" == caplog.records[0].levelname assert "deprecated message" in caplog.text def test_log_deduplication(self, caplog): - ci_logging.defineDeprecationLogger() + log.define_deprecation_logger() util.deprecate( deprecated="stuff", deprecated_version="19.1", @@ -87,3 +86,11 @@ schedule=6, ) assert 2 == len(caplog.records) + + +def test_logger_prints_to_stderr(capsys): + message = "to stdout" + log.setup_basic_logging() + LOG = logging.getLogger() + LOG.warning(message) + assert message in capsys.readouterr().err diff -Nru cloud-init-23.3.3/tests/unittests/test_merging.py cloud-init-23.4.3/tests/unittests/test_merging.py --- cloud-init-23.3.3/tests/unittests/test_merging.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_merging.py 2024-02-02 21:21:52.000000000 +0000 @@ -7,8 +7,15 @@ import re import string +import pytest + from cloudinit import helpers as c_helpers from cloudinit import util +from cloudinit.config.schema import ( + SchemaValidationError, + get_schema, + validate_cloudconfig_schema, +) from cloudinit.handlers import CONTENT_END, CONTENT_START, cloud_config from tests.unittests import helpers @@ -257,4 +264,152 @@ self.assertEqual(c, d) -# vi: ts=4 expandtab +class TestMergingSchema: + @pytest.mark.parametrize( + "config, error_msg", + [ + ({"merge_how": "list()+dict()+str()"}, None), + ({"merge_type": "list()+dict()+str()"}, None), + ({"merge_how": []}, "\\[\\] is too short"), + ( + {"merge_how": {"name": "list", "settings": ["append"]}}, + "is not of type", + ), + ( + {"merge_how": [{"name": "list", "settings": "append"}]}, + "'append' is not of type 'array'", + ), + ( + { + "merge_how": [ + { + "settings": ["recurse_list"], + } + ] + }, + "'name' is a required property", + ), + ( + { + "merge_how": [ + { + "name": "list", + } + ] + }, + "'settings' is a required property", + ), + ( + { + "merge_how": [ + { + "name": "str", + "settings": ["recurse_list"], + "badkey": "append", + } + ] + }, + ( + "Additional properties are not allowed " + "\$'badkey' was unexpected\$" + ), + ), + ( + { + "merge_how": [ + { + "name": "str", + "settings": ["badvalue"], + } + ] + }, + "'badvalue' is not one of", + ), + ( + { + "merge_how": [ + { + "name": "badvalue", + "settings": ["append"], + } + ] + }, + re.escape("'badvalue' is not one of ['list', 'dict', 'str']"), + ), + ( + { + "merge_how": [ + { + "name": "str", + "settings": [ + "append", + "recurse_dict", + "recurse_list", + ], + }, + { + "name": "dict", + "settings": [ + "allow_delete", + "no_replace", + "replace", + "recurse_array", + ], + }, + { + "name": "list", + "settings": [ + "append", + "prepend", + "no_replace", + "replace", + "recurse_str", + ], + }, + ] + }, + None, + ), + ( + { + "merge_type": [ + { + "name": "str", + "settings": [ + "append", + "recurse_dict", + "recurse_list", + ], + }, + { + "name": "dict", + "settings": [ + "allow_delete", + "no_replace", + "replace", + "recurse_array", + ], + }, + { + "name": "list", + "settings": [ + "append", + "prepend", + "no_replace", + "replace", + "recurse_str", + ], + }, + ] + }, + None, + ), + ], + ) + @helpers.skipUnlessJsonSchema() + def test_schema_validation(self, config, error_msg): + if error_msg is None: + validate_cloudconfig_schema(config, get_schema(), strict=True) + else: + with pytest.raises(SchemaValidationError, match=error_msg): + validate_cloudconfig_schema(config, get_schema(), strict=True) diff -Nru cloud-init-23.3.3/tests/unittests/test_net.py cloud-init-23.4.3/tests/unittests/test_net.py --- cloud-init-23.3.3/tests/unittests/test_net.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_net.py 2024-02-02 21:21:52.000000000 +0000 @@ -536,7 +536,7 @@ ( "etc/sysconfig/network/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=static IPADDR=172.19.1.34 @@ -548,7 +548,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -556,7 +556,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -576,7 +576,7 @@ ( "etc/sysconfig/network-scripts/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -594,7 +594,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -602,7 +602,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -697,7 +697,7 @@ ( "etc/sysconfig/network/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=static IPADDR=172.19.1.34 @@ -711,7 +711,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -719,7 +719,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -739,7 +739,7 @@ ( "etc/sysconfig/network-scripts/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -759,7 +759,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -767,7 +767,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -854,7 +854,7 @@ ( "etc/sysconfig/network/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=static IPADDR=172.19.1.34 @@ -869,7 +869,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -877,7 +877,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -897,7 +897,7 @@ ( "etc/sysconfig/network-scripts/ifcfg-eth0", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -921,7 +921,7 @@ ( "etc/resolv.conf", """ -; Created by cloud-init on instance boot automatically, do not edit. +; Created by cloud-init automatically, do not edit. ; nameserver 172.19.0.12 """.lstrip(), @@ -929,7 +929,7 @@ ( "etc/NetworkManager/conf.d/99-cloud-init.conf", """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # [main] dns = none @@ -988,6 +988,58 @@ """.lstrip() NETWORK_CONFIGS = { + "small_v1_suse_dhcp6": { + "expected_sysconfig_opensuse": { + "ifcfg-eth1": textwrap.dedent( + """\ + BOOTPROTO=static + LLADDR=cf:d6:af:48:e8:80 + STARTMODE=auto""" + ), + "ifcfg-eth99": textwrap.dedent( + """\ + BOOTPROTO=dhcp + DHCLIENT6_MODE=managed + LLADDR=c0:d6:9f:2c:e8:80 + IPADDR=192.168.21.3 + NETMASK=255.255.255.0 + STARTMODE=auto""" + ), + }, + "yaml": textwrap.dedent( + """ + version: 1 + config: + # Physical interfaces. + - type: physical + name: eth99 + mac_address: c0:d6:9f:2c:e8:80 + subnets: + - type: dhcp4 + - type: dhcp6 + - type: static + address: 192.168.21.3/24 + dns_nameservers: + - 8.8.8.8 + - 8.8.4.4 + dns_search: barley.maas sach.maas + routes: + - gateway: 65.61.151.37 + netmask: 0.0.0.0 + network: 0.0.0.0 + metric: 10000 + - type: physical + name: eth1 + mac_address: cf:d6:af:48:e8:80 + - type: nameserver + address: + - 1.2.3.4 + - 5.6.7.8 + search: + - wark.maas + """ + ), + }, "small_v1": { "expected_networkd_eth99": textwrap.dedent( """\ @@ -1932,6 +1984,9 @@ method=auto may-fail=false + [ipv4] + method=disabled + """ ), }, @@ -2044,6 +2099,9 @@ method=auto may-fail=false + [ipv4] + method=disabled + """ ), }, @@ -2091,11 +2149,12 @@ "expected_sysconfig_rhel": { "ifcfg-iface0": textwrap.dedent( """\ - BOOTPROTO=dhcp + BOOTPROTO=none DEVICE=iface0 DHCPV6C=yes IPV6INIT=yes IPV6_AUTOCONF=no + IPV6_FAILURE_FATAL=yes IPV6_FORCE_ACCEPT_RA=yes DEVICE=iface0 NM_CONTROLLED=no @@ -2531,7 +2590,7 @@ ), "ifcfg-eth5": textwrap.dedent( """\ - BOOTPROTO=dhcp + BOOTPROTO=dhcp4 LLADDR=98:bb:9f:2c:e8:8a STARTMODE=manual""" ), @@ -3435,7 +3494,7 @@ ), "route6-bond0": textwrap.dedent( """\ - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # 2001:67c::/32 via 2001:67c:1562::1 dev bond0 3001:67c::/32 via 3001:67c:15::1 metric 10000 dev bond0 @@ -4576,10 +4635,7 @@ with_logs = True scripts_dir = "/etc/sysconfig/network-scripts" - header = ( - "# Created by cloud-init on instance boot automatically, " - "do not edit.\n#\n" - ) + header = "# Created by cloud-init automatically, do not edit.\n#\n" expected_name = "expected_sysconfig_rhel" @@ -4675,7 +4731,7 @@ with open(os.path.join(render_dir, render_file)) as fh: content = fh.read() expected_content = """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp DEVICE=eth1000 @@ -4883,7 +4939,7 @@ nspath = "/etc/sysconfig/network-scripts/" self.assertNotIn(nspath + "ifcfg-lo", found.keys()) expected = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -4913,7 +4969,7 @@ nspath = "/etc/sysconfig/network-scripts/" self.assertNotIn(nspath + "ifcfg-lo", found.keys()) expected_i1 = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -4930,7 +4986,7 @@ """ self.assertEqual(expected_i1, found[nspath + "ifcfg-eth0"]) expected_i2 = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp DEVICE=eth1 @@ -4958,7 +5014,7 @@ nspath = "/etc/sysconfig/network-scripts/" self.assertNotIn(nspath + "ifcfg-lo", found.keys()) expected = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp DEVICE=eth0 @@ -5405,7 +5461,7 @@ expected = { "ifcfg-eth0": textwrap.dedent( """\ - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # BOOTPROTO=none DEFROUTE=yes @@ -5431,7 +5487,7 @@ ), "route-eth0": textwrap.dedent( """\ - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # ADDRESS0=10.54.0.1 GATEWAY0=0.0.0.0 @@ -5440,14 +5496,14 @@ ), "route6-eth0": textwrap.dedent( """\ - # Created by cloud-init on instance boot automatically, do not edit. + # Created by cloud-init automatically, do not edit. # 2a00:1730:fff9:100::1/128 via ::0 dev eth0 ::0/0 via 2a00:1730:fff9:100::1 dev eth0 """ # noqa: E501 ), } - log.setupLogging() + log.setup_logging() found = self._render_and_read(network_config=v2_data) self._compare_files_to_expected(expected, found) @@ -5501,10 +5557,7 @@ with_logs = True scripts_dir = "/etc/sysconfig/network" - header = ( - "# Created by cloud-init on instance boot automatically, " - "do not edit.\n#\n" - ) + header = "# Created by cloud-init automatically, do not edit.\n#\n" expected_name = "expected_sysconfig_opensuse" @@ -5591,7 +5644,7 @@ with open(os.path.join(render_dir, render_file)) as fh: content = fh.read() expected_content = """ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=dhcp4 LLADDR=07-1c-c6-75-a4-be @@ -5709,7 +5762,7 @@ nspath = "/etc/sysconfig/network/" self.assertNotIn(nspath + "ifcfg-lo", found.keys()) expected = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # BOOTPROTO=static IPADDR=10.0.2.15 @@ -5737,9 +5790,9 @@ nspath = "/etc/sysconfig/network/" self.assertNotIn(nspath + "ifcfg-lo", found.keys()) expected = """\ -# Created by cloud-init on instance boot automatically, do not edit. +# Created by cloud-init automatically, do not edit. # -BOOTPROTO=dhcp +BOOTPROTO=dhcp4 STARTMODE=auto """ self.assertEqual(expected, found[nspath + "ifcfg-eth0"]) @@ -5796,6 +5849,12 @@ self._compare_files_to_expected(entry[self.expected_name], found) self._assert_headers(found) + def test_small_config_v1_suse(self): + entry = NETWORK_CONFIGS["small_v1_suse_dhcp6"] + found = self._render_and_read(network_config=yaml.load(entry["yaml"])) + self._compare_files_to_expected(entry[self.expected_name], found) + self._assert_headers(found) + def test_small_config_v2(self): entry = NETWORK_CONFIGS["small_v1"] found = self._render_and_read(network_config=yaml.load(entry["yaml"])) @@ -8984,6 +9043,3 @@ self.assertEqual( "10.1.21.255", bcast_addr("255.255.255.0", "10.1.21.4") ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_netinfo.py cloud-init-23.4.3/tests/unittests/test_netinfo.py --- cloud-init-23.3.3/tests/unittests/test_netinfo.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_netinfo.py 2024-02-02 21:21:52.000000000 +0000 @@ -375,6 +375,3 @@ def test_netdev_info_iproute_json(self, input, expected): out = _netdev_info_iproute_json(json.dumps(input)) assert out == expected - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_pathprefix2dict.py cloud-init-23.4.3/tests/unittests/test_pathprefix2dict.py --- cloud-init-23.3.3/tests/unittests/test_pathprefix2dict.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_pathprefix2dict.py 2024-02-02 21:21:52.000000000 +0000 @@ -41,6 +41,3 @@ ret = util.pathprefix2dict(self.tmp, required=["f1"], optional=["f2"]) self.assertEqual(dirdata, ret) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_registry.py cloud-init-23.4.3/tests/unittests/test_registry.py --- cloud-init-23.3.3/tests/unittests/test_registry.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_registry.py 2024-02-02 21:21:52.000000000 +0000 @@ -28,6 +28,3 @@ self.assertRaises( ValueError, registry.register_item, item_key, mock.Mock() ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_render_cloudcfg.py cloud-init-23.4.3/tests/unittests/test_render_cloudcfg.py --- cloud-init-23.3.3/tests/unittests/test_render_cloudcfg.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_render_cloudcfg.py 1970-01-01 00:00:00.000000000 +0000 @@ -1,116 +0,0 @@ -"""Tests for tools/render-cloudcfg""" - -import sys - -import pytest - -from cloudinit import subp, templater, util -from tests.unittests.helpers import cloud_init_project_dir - -# TODO(Look to align with tools.render-cloudcfg or cloudinit.distos.OSFAMILIES) -DISTRO_VARIANTS = [ - "amazon", - "arch", - "centos", - "debian", - "eurolinux", - "fedora", - "freebsd", - "gentoo", - "mariner", - "netbsd", - "openbsd", - "photon", - "rhel", - "suse", - "ubuntu", - "unknown", -] - - -@pytest.mark.allow_subp_for(sys.executable) -class TestRenderCloudCfg: - cmd = [sys.executable, cloud_init_project_dir("tools/render-cloudcfg")] - tmpl_path = cloud_init_project_dir("config/cloud.cfg.tmpl") - init_path = cloud_init_project_dir("sysvinit/freebsd/dsidentify.tmpl") - - def test_variant_sets_distro_in_cloud_cfg_subp(self, tmpdir): - outfile = tmpdir.join("outcfg").strpath - - subp.subp(self.cmd + ["--variant", "ubuntu", self.tmpl_path, outfile]) - with open(outfile) as stream: - system_cfg = util.load_yaml(stream.read()) - assert system_cfg["system_info"]["distro"] == "ubuntu" - - def test_variant_sets_prefix_in_cloud_cfg_subp(self, tmpdir): - outfile = tmpdir.join("outcfg").strpath - - subp.subp( - self.cmd - + [ - "--variant", - "freebsd", - "--prefix", - "/usr/local", - self.init_path, - outfile, - ] - ) - with open(outfile) as stream: - init_cfg = stream.readlines() - assert 'command="/usr/local/lib/cloud-init/ds-identify"\n' in init_cfg - - @pytest.mark.parametrize("variant", (DISTRO_VARIANTS)) - def test_variant_sets_distro_in_cloud_cfg(self, variant, tmpdir): - """Testing parametrized inputs with imported function saves ~0.5s per - call versus calling as subp - """ - outfile = tmpdir.join("outcfg").strpath - - templater.render_cloudcfg(variant, self.tmpl_path, outfile) - with open(outfile) as stream: - system_cfg = util.load_yaml(stream.read()) - if variant == "unknown": - variant = "ubuntu" # Unknown is defaulted to ubuntu - assert system_cfg["system_info"]["distro"] == variant - - @pytest.mark.parametrize("variant", (DISTRO_VARIANTS)) - def test_variant_sets_default_user_in_cloud_cfg(self, variant, tmpdir): - """Testing parametrized inputs with imported function saves ~0.5s per - call versus calling as subp - """ - outfile = tmpdir.join("outcfg").strpath - templater.render_cloudcfg(variant, self.tmpl_path, outfile) - with open(outfile) as stream: - system_cfg = util.load_yaml(stream.read()) - - default_user_exceptions = { - "amazon": "ec2-user", - "rhel": "cloud-user", - "centos": "cloud-user", - "unknown": "ubuntu", - } - default_user = system_cfg["system_info"]["default_user"]["name"] - assert default_user == default_user_exceptions.get(variant, variant) - - @pytest.mark.parametrize( - "variant,renderers", - ( - ("freebsd", ["freebsd"]), - ("netbsd", ["netbsd"]), - ("openbsd", ["openbsd"]), - ("ubuntu", ["netplan", "eni", "sysconfig"]), - ), - ) - def test_variant_sets_network_renderer_priority_in_cloud_cfg( - self, variant, renderers, tmpdir - ): - """Testing parametrized inputs with imported function saves ~0.5s per - call versus calling as subp - """ - outfile = tmpdir.join("outcfg").strpath - templater.render_cloudcfg(variant, self.tmpl_path, outfile) - with open(outfile) as stream: - system_cfg = util.load_yaml(stream.read()) - - assert renderers == system_cfg["system_info"]["network"]["renderers"] diff -Nru cloud-init-23.3.3/tests/unittests/test_render_template.py cloud-init-23.4.3/tests/unittests/test_render_template.py --- cloud-init-23.3.3/tests/unittests/test_render_template.py 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_render_template.py 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,122 @@ +"""Tests for tools/render-template""" + +import sys + +import pytest + +from cloudinit import subp, templater, util +from tests.unittests.helpers import cloud_init_project_dir + +# TODO(Look to align with tools.render-template or cloudinit.distos.OSFAMILIES) +DISTRO_VARIANTS = [ + "amazon", + "arch", + "centos", + "debian", + "eurolinux", + "fedora", + "freebsd", + "gentoo", + "mariner", + "netbsd", + "openbsd", + "photon", + "rhel", + "suse", + "ubuntu", + "unknown", +] + + +@pytest.mark.allow_subp_for(sys.executable) +class TestRenderCloudCfg: + cmd = [sys.executable, cloud_init_project_dir("tools/render-template")] + tmpl_path = cloud_init_project_dir("config/cloud.cfg.tmpl") + init_path = cloud_init_project_dir("sysvinit/freebsd/dsidentify.tmpl") + + def test_variant_sets_distro_in_cloud_cfg_subp(self, tmpdir): + outfile = tmpdir.join("outcfg").strpath + + subp.subp(self.cmd + ["--variant", "ubuntu", self.tmpl_path, outfile]) + with open(outfile) as stream: + system_cfg = util.load_yaml(stream.read()) + assert system_cfg["system_info"]["distro"] == "ubuntu" + + def test_variant_sets_prefix_in_cloud_cfg_subp(self, tmpdir): + outfile = tmpdir.join("outcfg").strpath + + subp.subp( + self.cmd + + [ + "--variant", + "freebsd", + "--prefix", + "/usr/local", + self.init_path, + outfile, + ] + ) + with open(outfile) as stream: + init_cfg = stream.readlines() + assert 'command="/usr/local/lib/cloud-init/ds-identify"\n' in init_cfg + + @pytest.mark.parametrize("variant", (DISTRO_VARIANTS)) + def test_variant_sets_distro_in_cloud_cfg(self, variant, tmpdir): + """Testing parametrized inputs with imported function saves ~0.5s per + call versus calling as subp + """ + outfile = tmpdir.join("outcfg").strpath + + templater.render_template( + variant, self.tmpl_path, outfile, is_yaml=True + ) + with open(outfile) as stream: + system_cfg = util.load_yaml(stream.read()) + if variant == "unknown": + variant = "ubuntu" # Unknown is defaulted to ubuntu + assert system_cfg["system_info"]["distro"] == variant + + @pytest.mark.parametrize("variant", (DISTRO_VARIANTS)) + def test_variant_sets_default_user_in_cloud_cfg(self, variant, tmpdir): + """Testing parametrized inputs with imported function saves ~0.5s per + call versus calling as subp + """ + outfile = tmpdir.join("outcfg").strpath + templater.render_template( + variant, self.tmpl_path, outfile, is_yaml=True + ) + with open(outfile) as stream: + system_cfg = util.load_yaml(stream.read()) + + default_user_exceptions = { + "amazon": "ec2-user", + "rhel": "cloud-user", + "centos": "cloud-user", + "unknown": "ubuntu", + } + default_user = system_cfg["system_info"]["default_user"]["name"] + assert default_user == default_user_exceptions.get(variant, variant) + + @pytest.mark.parametrize( + "variant,renderers", + ( + ("freebsd", ["freebsd"]), + ("netbsd", ["netbsd"]), + ("openbsd", ["openbsd"]), + ("ubuntu", ["netplan", "eni", "sysconfig"]), + ), + ) + def test_variant_sets_network_renderer_priority_in_cloud_cfg( + self, variant, renderers, tmpdir + ): + """Testing parametrized inputs with imported function saves ~0.5s per + call versus calling as subp + """ + outfile = tmpdir.join("outcfg").strpath + templater.render_template( + variant, self.tmpl_path, outfile, is_yaml=True + ) + with open(outfile) as stream: + system_cfg = util.load_yaml(stream.read()) + + assert renderers == system_cfg["system_info"]["network"]["renderers"] diff -Nru cloud-init-23.3.3/tests/unittests/test_safeyaml.py cloud-init-23.4.3/tests/unittests/test_safeyaml.py --- cloud-init-23.3.3/tests/unittests/test_safeyaml.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_safeyaml.py 2024-02-02 21:21:52.000000000 +0000 @@ -46,7 +46,7 @@ pytest.param( b"{a: [a1, a2], b: [b3]}", {"a": ["a1", "a2"], "b": ["b3"]}, - {"a": 1, "a.0": 1, "a.1": 1, "b": 1}, + {"a": 1, "a.0": 1, "a.1": 1, "b": 1, "b.0": 1}, id="dict_of_lists_oneline", ), pytest.param( @@ -65,11 +65,42 @@ b"a:\n- a1\n- a2\nb:\n- b3", {"a": ["a1", "a2"], "b": ["b3"]}, {"a": 1, "a.0": 2, "a.1": 3, "b": 4, "b.0": 5}, - id="separate_dicts_nestes_lists", + id="separate_dicts_nested_lists", + ), + pytest.param( + b"a:\n- {a1: 1}\n- {a2: 2}\n", + {"a": [{"a1": 1}, {"a2": 2}]}, + {"a": 1, "a.0": 2, "a.0.a1": 2, "a.1": 3, "a.1.a2": 3}, + id="list_of_dict_items", + ), + pytest.param( + b"a:\n- x: ['i', 'ii']\n", + {"a": [{"x": ["i", "ii"]}]}, + {"a": 1, "a.0": 2, "a.0.x": 2, "a.0.x.0": 2, "a.0.x.1": 2}, + id="list_of_dict_items_with_nested_lists", + ), + pytest.param( + b"a:\n- x: [['i', 'ii']]\n- y: ['iii', 'iv']\n", + {"a": [{"x": [["i", "ii"]]}, {"y": ["iii", "iv"]}]}, + { + "a": 1, + "a.0": 2, + "a.0.x": 2, + "a.0.x.0": 2, + "a.0.x.0.0": 2, + "a.0.x.0.1": 2, + "a.1": 3, + "a.1.y": 3, + "a.1.y.0": 3, + "a.1.y.1": 3, + }, + id="list_of_dict_items_with_nested_lists_of_lists", ), ), ) def test_schema_marks_preserved( self, source_yaml, loaded_yaml, schemamarks ): - assert (loaded_yaml, schemamarks) == load_with_marks(source_yaml) + (processed_yaml, yaml_marks) = load_with_marks(source_yaml) + assert loaded_yaml == processed_yaml + assert schemamarks == yaml_marks diff -Nru cloud-init-23.3.3/tests/unittests/test_ssh_util.py cloud-init-23.4.3/tests/unittests/test_ssh_util.py --- cloud-init-23.3.3/tests/unittests/test_ssh_util.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_ssh_util.py 2024-02-02 21:21:52.000000000 +0000 @@ -1598,6 +1598,3 @@ self.execute_and_check(user_bobby, sshd_config, authorized_keys, keys) self.execute_and_check(user_suzie, sshd_config, authorized_keys2, keys) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_stages.py cloud-init-23.4.3/tests/unittests/test_stages.py --- cloud-init-23.3.3/tests/unittests/test_stages.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_stages.py 2024-02-02 21:21:52.000000000 +0000 @@ -1,6 +1,7 @@ # This file is part of cloud-init. See LICENSE file for license information. """Tests related to cloudinit.stages module.""" +import json import os import stat @@ -9,7 +10,7 @@ from cloudinit import sources, stages from cloudinit.event import EventScope, EventType from cloudinit.sources import NetworkConfigSource -from cloudinit.util import write_file +from cloudinit.util import sym_link, write_file from tests.unittests.helpers import mock from tests.unittests.util import TEST_INSTANCE_ID, FakeDataSource @@ -27,6 +28,8 @@ "paths": {"cloud_dir": self.tmpdir, "run_dir": self.tmpdir}, } } + tmpdir.mkdir("instance-uuid") + sym_link(tmpdir.join("instance-uuid"), tmpdir.join("instance")) self.init.datasource = FakeDataSource(paths=self.init.paths) self._real_is_new_instance = self.init.is_new_instance self.init.is_new_instance = mock.Mock(return_value=True) @@ -391,9 +394,12 @@ assert caplog.records[0].levelname == "INFO" assert f"network config is disabled by {disable_file}" in caplog.text + @pytest.mark.parametrize("instance_dir_present", (True, False)) @mock.patch("cloudinit.net.get_interfaces_by_mac") @mock.patch("cloudinit.distros.ubuntu.Distro") - def test_apply_network_on_new_instance(self, m_ubuntu, m_macs): + def test_apply_network_on_new_instance( + self, m_ubuntu, m_macs, instance_dir_present + ): """Call distro apply_network_config methods on is_new_instance.""" net_cfg = { "version": 1, @@ -413,13 +419,26 @@ m_macs.return_value = {"42:42:42:42:42:42": "eth9"} self.init._find_networking_config = fake_network_config - + if not instance_dir_present: + self.tmpdir.join("instance").remove() + self.tmpdir.join("instance-uuid").remove() self.init.apply_network_config(True) networking = self.init.distro.networking networking.apply_network_config_names.assert_called_with(net_cfg) self.init.distro.apply_network_config.assert_called_with( net_cfg, bring_up=True ) + if instance_dir_present: + assert net_cfg == json.loads( + self.tmpdir.join("network-config.json").read() + ) + assert os.path.islink(self.tmpdir.join("network-config.json")) + else: + for path in ( + "instance/network-config.json", + "network-config.json", + ): + assert not self.tmpdir.join(path).exists() @mock.patch("cloudinit.distros.ubuntu.Distro") def test_apply_network_on_same_instance_id(self, m_ubuntu, caplog): @@ -611,31 +630,43 @@ assert 0o640 == stat.S_IMODE(log_file.stat().mode) @pytest.mark.parametrize( - "set_perms,expected_perms", + "input, expected", [ + (0o777, 0o640), (0o640, 0o640), - (0o606, 0o640), - (0o600, 0o600), + (0o606, 0o600), + (0o501, 0o400), ], ) - def test_existing_file_permissions( - self, init, tmpdir, set_perms, expected_perms - ): + def test_existing_file_permissions(self, init, tmpdir, input, expected): """Test file permissions are set as expected. - CIS Hardening requires 640 permissions. If the file has looser - permissions, then hard code 640. If the file has tighter - permissions, then leave them as they are + CIS Hardening requires file mode 0o640 or stricter. Set the + permissions to the subset of 0o640 and the current + mode. See https://bugs.launchpad.net/cloud-init/+bug/1900837. """ log_file = tmpdir.join("cloud-init.log") log_file.ensure() - # Use a mode that will never be made the default so this test will - # always be valid - log_file.chmod(set_perms) + log_file.chmod(input) init._cfg = {"def_log_file": str(log_file)} - - init._initialize_filesystem() - - assert expected_perms == stat.S_IMODE(log_file.stat().mode) + with mock.patch.object(stages.util, "ensure_file") as ensure: + init._initialize_filesystem() + assert expected == ensure.call_args[0][1] + + +@pytest.mark.parametrize( + "mode_1, mode_2, expected", + [ + (0o777, 0o640, 0o640), + (0o640, 0o777, 0o640), + (0o640, 0o541, 0o440), + (0o111, 0o050, 0o010), + (0o631, 0o640, 0o600), + (0o661, 0o640, 0o640), + (0o453, 0o611, 0o411), + ], +) +def test_strictest_permissions(mode_1, mode_2, expected): + assert expected == stages.Init._get_strictest_mode(mode_1, mode_2) diff -Nru cloud-init-23.3.3/tests/unittests/test_subp.py cloud-init-23.4.3/tests/unittests/test_subp.py --- cloud-init-23.3.3/tests/unittests/test_subp.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_subp.py 2024-02-02 21:21:52.000000000 +0000 @@ -348,6 +348,3 @@ "End run command: exit(0)\n", ] self.assertEqual(expected, logs) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_temp_utils.py cloud-init-23.4.3/tests/unittests/test_temp_utils.py --- cloud-init-23.3.3/tests/unittests/test_temp_utils.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_temp_utils.py 2024-02-02 21:21:52.000000000 +0000 @@ -134,6 +134,3 @@ os.rmdir(tdir) # Since the directory is already gone, shutil.rmtree would raise # OSError, but we suppress that - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_templating.py cloud-init-23.4.3/tests/unittests/test_templating.py --- cloud-init-23.3.3/tests/unittests/test_templating.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_templating.py 2024-02-02 21:21:52.000000000 +0000 @@ -167,6 +167,3 @@ ).strip(), expected_result, ) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_url_helper.py cloud-init-23.4.3/tests/unittests/test_url_helper.py --- cloud-init-23.3.3/tests/unittests/test_url_helper.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_url_helper.py 2024-02-02 21:21:52.000000000 +0000 @@ -11,14 +11,12 @@ from cloudinit import util, version from cloudinit.url_helper import ( - NOT_FOUND, REDACTED, UrlError, UrlResponse, dual_stack, oauth_headers, read_file_or_url, - retry_on_url_exc, wait_for_url, ) from tests.unittests.helpers import CiTestCase, mock, skipIf @@ -257,25 +255,6 @@ assert response._response == m_response -class TestRetryOnUrlExc(CiTestCase): - def test_do_not_retry_non_urlerror(self): - """When exception is not UrlError return False.""" - myerror = IOError("something unexcpected") - self.assertFalse(retry_on_url_exc(msg="", exc=myerror)) - - def test_perform_retries_on_not_found(self): - """When exception is UrlError with a 404 status code return True.""" - myerror = UrlError( - cause=RuntimeError("something was not found"), code=NOT_FOUND - ) - self.assertTrue(retry_on_url_exc(msg="", exc=myerror)) - - def test_perform_retries_on_timeout(self): - """When exception is a requests.Timout return True.""" - myerror = UrlError(cause=requests.Timeout("something timed out")) - self.assertTrue(retry_on_url_exc(msg="", exc=myerror)) - - def assert_time(func, max_time=1): """Assert function time is bounded by a max (default=1s) @@ -579,6 +558,3 @@ self.event.set() assert not url assert not response_contents - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tests/unittests/test_util.py cloud-init-23.4.3/tests/unittests/test_util.py --- cloud-init-23.3.3/tests/unittests/test_util.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_util.py 2024-02-02 21:21:52.000000000 +0000 @@ -1457,6 +1457,19 @@ "cc:runcmd%3A%20%5B%20%5B%20ls%2C%20-l%20%5D%20%5D end_cc", {"ssh_import_id": ["smoser"], "runcmd": [["ls", "-l"]]}, ), + # Parse cmdlines that contain an IPv6 with cc: in different + # positions + ("BOOTIF=aa:bb:cc:dd bar", None), + ("BOOTIF=aa:bb:cc:dd cc: end_cc bar", None), + ("BOOTIF=aa:bb:cc:dd cc: ssh_pwauth: true", {"ssh_pwauth": True}), + ( + "BOOTIF=aa:bb:cc:dd cc: ssh_pwauth: true end_cc", + {"ssh_pwauth": True}, + ), + ( + "cc: ssh_pwauth: true end_cc BOOTIF=aa:bb:cc:dd", + {"ssh_pwauth": True}, + ), ], ) def test_read_conf_from_cmdline_config(self, expected_cfg, cmdline): @@ -3137,30 +3150,6 @@ ) -class TestComparePermissions: - @pytest.mark.parametrize( - "perm1,perm2,expected", - [ - (0o777, 0o777, 0), - (0o000, 0o000, 0), - (0o421, 0o421, 0), - (0o1640, 0o1640, 0), - (0o1407, 0o1600, 1), - (0o1600, 0o1407, -1), - (0o407, 0o600, 1), - (0o600, 0o407, -1), - (0o007, 0o700, 1), - (0o700, 0o007, -1), - (0o077, 0o100, 1), - (0o644, 0o640, 1), - (0o640, 0o600, 1), - (0o600, 0o400, 1), - ], - ) - def test_compare_permissions(self, perm1, perm2, expected): - assert util.compare_permission(perm1, perm2) == expected - - class TestMaybeB64Decode: """Test the maybe_b64decode helper function.""" diff -Nru cloud-init-23.3.3/tests/unittests/test_version.py cloud-init-23.4.3/tests/unittests/test_version.py --- cloud-init-23.3.3/tests/unittests/test_version.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tests/unittests/test_version.py 2024-02-02 21:21:52.000000000 +0000 @@ -27,6 +27,3 @@ def test_package_version_skipped(self): """If _PACKAGED_VERSION is not modified, then return __VERSION__.""" self.assertEqual("17.2", version.version_string()) - - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/.github-cla-signers cloud-init-23.4.3/tools/.github-cla-signers --- cloud-init-23.3.3/tools/.github-cla-signers 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/.github-cla-signers 2024-02-02 21:21:52.000000000 +0000 @@ -25,6 +25,7 @@ brianphaley CalvoM candlerb +catmsred cawamata cclauss chifac08 @@ -32,6 +33,7 @@ ciprianbadescu citrus-it cjp256 +CodeBleu Conan-Kudo cvstealth dankenigsberg @@ -62,6 +64,7 @@ Indrranil irishgordo ITJamie +itsaviral2609 ixjhuang izzyleung j5awry @@ -83,6 +86,7 @@ klausenbusk KsenijaS landon912 +ld9379435 licebmi linitio lkundrak @@ -103,6 +107,7 @@ megian michaelrommel mitechie +MjMoshiri mxwebdev nazunalika netcho @@ -113,10 +118,13 @@ olivierlemasle omBratteng onitake +orndorffgrant Oursin outscale-mdr +phsm phunyguy qubidt +r00ta RedKrieg renanrodrigo rhansen diff -Nru cloud-init-23.3.3/tools/Z99-cloud-locale-test.sh cloud-init-23.4.3/tools/Z99-cloud-locale-test.sh --- cloud-init-23.3.3/tools/Z99-cloud-locale-test.sh 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/Z99-cloud-locale-test.sh 2024-02-02 21:21:52.000000000 +0000 @@ -100,4 +100,3 @@ locale 2>&1 | locale_warn unset locale_warn -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/Z99-cloudinit-warnings.sh cloud-init-23.4.3/tools/Z99-cloudinit-warnings.sh --- cloud-init-23.3.3/tools/Z99-cloudinit-warnings.sh 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/Z99-cloudinit-warnings.sh 2024-02-02 21:21:52.000000000 +0000 @@ -28,5 +28,3 @@ cloud_init_warnings 1>&2 unset cloud_init_warnings - -# vi: syntax=sh ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/build-on-openbsd cloud-init-23.4.3/tools/build-on-openbsd --- cloud-init-23.3.3/tools/build-on-openbsd 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/build-on-openbsd 2024-02-02 21:21:52.000000000 +0000 @@ -10,13 +10,14 @@ py3-configobj py3-jinja2 py3-jsonschema + py3-netifaces py3-oauthlib py3-requests py3-setuptools py3-yaml sudo-- " -[ -f "$depschecked" ] || pkg_add ${pkgs} || fail "install packages" +[ -f "$depschecked" ] || pkg_add "${pkgs}" || fail "install packages" touch $depschecked diff -Nru cloud-init-23.3.3/tools/ccfg-merge-debug cloud-init-23.4.3/tools/ccfg-merge-debug --- cloud-init-23.3.3/tools/ccfg-merge-debug 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/ccfg-merge-debug 2024-02-02 21:21:52.000000000 +0000 @@ -26,7 +26,7 @@ if args.verbose: level = (logging.WARN, logging.INFO, logging.DEBUG)[min(args.verbose, 2)] - logging.setupBasicLogging(level) + logging.setup_basic_logging(level) outfile = args.output if args.output == "-": @@ -86,5 +86,3 @@ if __name__ == "__main__": main() - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/cloudconfig-schema cloud-init-23.4.3/tools/cloudconfig-schema --- cloud-init-23.3.3/tools/cloudconfig-schema 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/cloudconfig-schema 2024-02-02 21:21:52.000000000 +0000 @@ -31,5 +31,3 @@ if __name__ == '__main__': call_entry_point("cloudinit.config.schema.main") - -# vi: ts=4 expandtab syntax=python diff -Nru cloud-init-23.3.3/tools/ds-identify cloud-init-23.4.3/tools/ds-identify --- cloud-init-23.3.3/tools/ds-identify 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/ds-identify 2024-02-02 21:21:52.000000000 +0000 @@ -330,8 +330,8 @@ [ -n "$dev" -a "$ftype" = "iso9660" ] && isodevs="${isodevs},${dev}=$label" - DI_FS_LABELS="${labels%${delim}}" - DI_FS_UUIDS="${uuids%${delim}}" + DI_FS_LABELS="${labels%"${delim}"}" + DI_FS_UUIDS="${uuids%"${delim}"}" DI_ISO9660_DEVS="${isodevs#,}" } @@ -381,7 +381,7 @@ labels="${labels}${label}${delim}" done - DI_FS_LABELS="${labels%${delim}}" + DI_FS_LABELS="${labels%"${delim}"}" DI_ISO9660_DEVS="${isodevs#,}" } @@ -462,7 +462,7 @@ } is_socket_file() { - [ -S $1 ] && return 0 || return 1 + [ -S "$1" ] && return 0 || return 1 } read_kernel_cmdline() { @@ -590,8 +590,8 @@ case "$DI_KERNEL_CMDLINE" in *cc:*datasource_list*) t=${DI_KERNEL_CMDLINE##*datasource_list} - t=${t%%$cb*} - t=${t##*$ob} + t=${t%%"${cb}"*} + t=${t##*"${ob}"} parse_yaml_array "$t" dslist=${_RET} ;; @@ -687,7 +687,7 @@ local delim="-delim-" # shellcheck disable=2018,2019 out=$(echo "$1${delim}$2" | tr A-Z a-z) - [ "${out#*${delim}}" = "${out%${delim}*}" ] + [ "${out#*"${delim}"}" = "${out%"${delim}"*}" ] } check_seed_dir() { @@ -713,7 +713,7 @@ # over the top of /var/lib/cloud, but the mount might not be done yet. local wdir="/writable/system-data" [ -d "${PATH_ROOT}$wdir" ] || return 1 - local sdir="${PATH_ROOT}$wdir${PATH_VAR_LIB_CLOUD#${PATH_ROOT}}" + local sdir="${PATH_ROOT}$wdir${PATH_VAR_LIB_CLOUD#"${PATH_ROOT}"}" local PATH_VAR_LIB_CLOUD="$sdir" check_seed_dir "$@" } @@ -863,7 +863,7 @@ is_ds_enabled() { local name="$1" pad=" ${DI_DSLIST} " - [ "${pad#* $name }" != "${pad}" ] + [ "${pad#* "${name}" }" != "${pad}" ] } check_configdrive_v2() { @@ -966,14 +966,64 @@ return 1 } +vmware_has_rpctool() { + command -v vmware-rpctool >/dev/null 2>&1 +} + +vmware_rpctool_guestinfo() { + vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]" +} + +vmware_rpctool_guestinfo_err() { + vmware-rpctool "info-get guestinfo.${1}" 2>&1 | grep "[[:alnum:]]" +} + +vmware_has_vmtoolsd() { + command -v vmtoolsd >/dev/null 2>&1 +} + +vmware_vmtoolsd_guestinfo() { + vmtoolsd --cmd "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]" +} + +vmware_vmtoolsd_guestinfo_err() { + vmtoolsd --cmd "info-get guestinfo.${1}" 2>&1 | grep "[[:alnum:]]" +} + +vmware_guestinfo() { + vmware_rpctool_guestinfo "${1}" || vmware_vmtoolsd_guestinfo "${1}" +} + +vmware_guestinfo_err() { + vmware_rpctool_guestinfo_err "${1}" || vmware_vmtoolsd_guestinfo_err "${1}" +} + +vmware_guestinfo_ovfenv_err() { + vmware_guestinfo_err "ovfEnv" +} + +vmware_guestinfo_metadata() { + vmware_guestinfo "metadata" +} + +vmware_guestinfo_userdata() { + + vmware_guestinfo "userdata" +} + +vmware_guestinfo_vendordata() { + + vmware_guestinfo "vendordata" +} + ovf_vmware_transport_guestinfo() { [ "${DI_VIRT}" = "vmware" ] || return 1 - command -v vmware-rpctool >/dev/null 2>&1 || return 1 + vmware_has_rpctool || vmware_has_vmtoolsd || return 1 local out="" ret="" - out=$(vmware-rpctool "info-get guestinfo.ovfEnv" 2>&1) + out=$(vmware_guestinfo_ovfenv_err) ret=$? if [ $ret -ne 0 ]; then - debug 1 "Running on vmware but rpctool query returned $ret: $out" + debug 1 "Running on vmware but query returned $ret: $out" return 1 fi case "$out" in @@ -1022,7 +1072,7 @@ has_ovf_cdrom() { # DI_ISO9660_DEVS is =label,=label2 # like /dev/sr0=OVF-TRANSPORT,/dev/other=with spaces - if [ "${DI_ISO9660_DEVS#${UNAVAILABLE}:}" = "${DI_ISO9660_DEVS}" ]; then + if [ "${DI_ISO9660_DEVS#"${UNAVAILABLE}":}" = "${DI_ISO9660_DEVS}" ]; then local oifs="$IFS" # shellcheck disable=2086 { IFS=","; set -- ${DI_ISO9660_DEVS}; IFS="$oifs"; } @@ -1033,6 +1083,23 @@ return 1 } +is_disabled() { + if [ -f /etc/cloud/cloud-init.disabled ]; then + debug 1 "disabled by marker file /etc/cloud-init.disabled" + return 0 + fi + if [ "${KERNEL_CMDLINE:-}" = "cloud-init=disabled" ]; then + debug 1 "disabled by KERNEL_CMDLINE environment variable" + return 0 + fi + case "$DI_KERNEL_CMDLINE" in + *cloud-init=disabled*) + debug 1 "disabled by kernel command line cloud-init=disabled" + return 0 + esac + return 1 +} + dscheck_OVF() { check_seed_dir ovf ovf-env.xml && return "${DS_FOUND}" @@ -1054,19 +1121,9 @@ } dscheck_Azure() { - # http://paste.ubuntu.com/23630873/ - # $ grep /sr0 /run/blkid/blkid.tab - # /dev/sr0 - # is_azure_chassis && return $DS_FOUND check_seed_dir azure ovf-env.xml && return ${DS_FOUND} - [ "${DI_VIRT}" = "microsoft" ] || return ${DS_NOT_FOUND} - - has_fs_with_label "rd_rdfe_*" && return ${DS_FOUND} - return ${DS_NOT_FOUND} } @@ -1091,7 +1148,7 @@ # 4. kernel command line case " ${DI_KERNEL_CMDLINE} " in *\ $key=*\ ) - val=${DI_KERNEL_CMDLINE##*$key=} + val=${DI_KERNEL_CMDLINE##*"${key}"=} val=${val%% *}; _RET=${val:-$default} return 0 @@ -1443,26 +1500,6 @@ [ -n "${VMX_GUESTINFO_VENDORDATA:-}" ] } -vmware_has_rpctool() { - command -v vmware-rpctool >/dev/null 2>&1 -} - -vmware_rpctool_guestinfo() { - vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]" -} - -vmware_rpctool_guestinfo_metadata() { - vmware_rpctool_guestinfo "metadata" -} - -vmware_rpctool_guestinfo_userdata() { - vmware_rpctool_guestinfo "userdata" -} - -vmware_rpctool_guestinfo_vendordata() { - vmware_rpctool_guestinfo "vendordata" -} - dscheck_VMware() { # Checks to see if there is valid data for the VMware datasource. # The data transports are checked in the following order: @@ -1490,16 +1527,16 @@ return "${DS_NOT_FOUND}" fi - # Do not proceed if the vmware-rpctool command is not present. - if ! vmware_has_rpctool; then + # Do not proceed if neither the vmware-rpctool or vmtoolsd command exists. + if ! vmware_has_rpctool && ! vmware_has_vmtoolsd; then return "${DS_NOT_FOUND}" fi # Activate the VMware datasource only if any of the fields used # by the datasource are present in the guestinfo table. - if { vmware_rpctool_guestinfo_metadata || \ - vmware_rpctool_guestinfo_userdata || \ - vmware_rpctool_guestinfo_vendordata; } >/dev/null 2>&1; then + if { vmware_guestinfo_metadata || \ + vmware_guestinfo_userdata || \ + vmware_guestinfo_vendordata; } >/dev/null 2>&1; then return "${DS_FOUND}" fi @@ -1511,10 +1548,7 @@ } collect_info() { - read_uname_info - read_virt read_pid1_product_name - read_kernel_cmdline read_config read_datasource_list read_dmi_sys_vendor @@ -1629,7 +1663,7 @@ local keyname="${1:-_unset}" local line="" hash="#" key="" val="" while read line; do - line=${line%%${hash}*} + line=${line%%"${hash}"*} key="${line%%:*}" # no : in the line. @@ -1789,6 +1823,12 @@ read_uptime debug 1 "[up ${_RET}s]" "ds-identify $*" + read_uname_info + read_virt + read_kernel_cmdline + if is_disabled; then + return 2 + fi collect_info if [ "$DI_LOG" = "stderr" ]; then @@ -1935,5 +1975,3 @@ main|print_info|noop) "${DI_MAIN}" "$@";; *) error "unexpected value for DI_MAIN"; exit 1;; esac - -# vi: syntax=sh ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/mock-meta.py cloud-init-23.4.3/tools/mock-meta.py --- cloud-init-23.3.3/tools/mock-meta.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/mock-meta.py 2024-02-02 21:21:52.000000000 +0000 @@ -491,5 +491,3 @@ if __name__ == "__main__": run_server() - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/read-dependencies cloud-init-23.4.3/tools/read-dependencies --- cloud-init-23.3.3/tools/read-dependencies 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/read-dependencies 2024-02-02 21:21:52.000000000 +0000 @@ -210,8 +210,11 @@ dep = line.split(";")[0] # remove version requirements - if re.search("[>=.<]+", dep): - dep_names.append(re.split(r"[>=.<]+", dep)[0].strip()) + version_comparison = re.compile(r"[~!>=.<]+") + if version_comparison.search(dep): + dep_names.append( + version_comparison.split(dep)[0].strip() + ) else: dep_names.append(dep) return dep_names @@ -352,5 +355,3 @@ parser = get_parser() args = parser.parse_args() sys.exit(main(args.distro)) - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/render-cloudcfg cloud-init-23.4.3/tools/render-cloudcfg --- cloud-init-23.3.3/tools/render-cloudcfg 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/render-cloudcfg 1970-01-01 00:00:00.000000000 +0000 @@ -1,79 +0,0 @@ -#!/usr/bin/env python3 - -import os -import sys -import argparse - - -def main(): - _tdir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) - sys.path.insert(0, _tdir) - from cloudinit import templater, util # pylint: disable=E0401 - - VARIANTS = [ - "almalinux", - "alpine", - "amazon", - "arch", - "centos", - "cloudlinux", - "debian", - "dragonfly", - "eurolinux", - "fedora", - "freebsd", - "gentoo", - "mariner", - "miraclelinux", - "netbsd", - "openbsd", - "openeuler", - "OpenCloudOS", - "openmandriva", - "photon", - "rhel", - "suse", - "rocky", - "TencentOS", - "ubuntu", - "unknown", - "virtuozzo", - ] - parser = argparse.ArgumentParser() - platform = util.system_info() - parser.add_argument( - "--variant", - default=platform["variant"], - action="store", - help="define the variant.", - choices=VARIANTS, - ) - parser.add_argument( - "--prefix", - action="store", - default=sys.prefix, - help="define the prefix. Default: " + sys.prefix, - ) - parser.add_argument( - "template", - nargs="?", - action="store", - default="./config/cloud.cfg.tmpl", - help="Path to the cloud.cfg template", - ) - parser.add_argument( - "output", - nargs="?", - action="store", - default="-", - help="Output file. Use '-' to write to stdout", - ) - - args = parser.parse_args(sys.argv[1:]) - templater.render_cloudcfg( - args.variant, args.template, args.output, prefix=args.prefix - ) - - -if __name__ == "__main__": - main() diff -Nru cloud-init-23.3.3/tools/render-template cloud-init-23.4.3/tools/render-template --- cloud-init-23.3.3/tools/render-template 1970-01-01 00:00:00.000000000 +0000 +++ cloud-init-23.4.3/tools/render-template 2024-02-02 21:21:52.000000000 +0000 @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 + +import os +import sys +import argparse + + +def main(): + _tdir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) + sys.path.insert(0, _tdir) + from cloudinit import templater, util # pylint: disable=E0401 + + VARIANTS = [ + "almalinux", + "alpine", + "amazon", + "arch", + "benchmark", + "centos", + "cloudlinux", + "debian", + "dragonfly", + "eurolinux", + "fedora", + "freebsd", + "gentoo", + "mariner", + "miraclelinux", + "netbsd", + "openbsd", + "openeuler", + "OpenCloudOS", + "openmandriva", + "photon", + "rhel", + "suse", + "rocky", + "TencentOS", + "ubuntu", + "unknown", + "virtuozzo", + ] + parser = argparse.ArgumentParser() + platform = util.system_info() + parser.add_argument( + "--variant", + default=platform["variant"], + action="store", + help="define the variant.", + choices=VARIANTS, + ) + parser.add_argument( + "--prefix", + action="store", + default=sys.prefix, + help="define the prefix. Default: " + sys.prefix, + ) + parser.add_argument( + "--is-yaml", + action="store_true", + default=False, + help=( + "Rendered template is not expected to be valid yaml by default." + " Add this flag for extra validation." + ), + ) + parser.add_argument( + "template", + nargs="?", + action="store", + default="./config/cloud.cfg.tmpl", + help="Path to the cloud.cfg template", + ) + parser.add_argument( + "output", + nargs="?", + action="store", + default="-", + help="Output file. Use '-' to write to stdout", + ) + + args = parser.parse_args(sys.argv[1:]) + templater.render_template( + args.variant, args.template, args.output, args.is_yaml, prefix=args.prefix + ) + + +if __name__ == "__main__": + main() diff -Nru cloud-init-23.3.3/tools/run-container cloud-init-23.4.3/tools/run-container --- cloud-init-23.3.3/tools/run-container 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/run-container 2024-02-02 21:21:52.000000000 +0000 @@ -251,7 +251,7 @@ case "$OS_NAME" in centos|rocky*) yum_install "$@";; opensuse*) zypper_install "$@";; - debian|ubuntu) apt_install "$@";; + debian|ubuntu) apt_install "$@" -y;; *) error "Do not know how to install packages on ${OS_NAME}"; return 1;; esac @@ -572,5 +572,3 @@ prep|os_info|wait_inside|pytest) _n=$1; shift; "$_n" "$@";; *) main "$@";; esac - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/run-lint cloud-init-23.4.3/tools/run-lint --- cloud-init-23.3.3/tools/run-lint 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/run-lint 2024-02-02 21:21:52.000000000 +0000 @@ -1,5 +1,7 @@ #!/bin/bash +# This file runs flake8 for compatibility's sake. As soon as we move off python 3.6, this should be changed to use ruff. + CR=" " pycheck_dirs=( "cloudinit/" "tests/" "tools/" "setup.py" ) @@ -11,7 +13,7 @@ files=( "$@" ) fi -cmd=( "python3" -m "ruff" "${files[@]}" ) +cmd=( "python3" -m "flake8" "${files[@]}" ) echo "Running: " "${cmd[@]}" 1>&2 exec "${cmd[@]}" diff -Nru cloud-init-23.3.3/tools/tox-venv cloud-init-23.4.3/tools/tox-venv --- cloud-init-23.3.3/tools/tox-venv 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/tox-venv 2024-02-02 21:21:52.000000000 +0000 @@ -181,5 +181,3 @@ fi echo "inside tox:$env running: $*" 1>&2 debian_chroot="tox:$env" exec "$@" - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/uncloud-init cloud-init-23.4.3/tools/uncloud-init --- cloud-init-23.3.3/tools/uncloud-init 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/uncloud-init 2024-02-02 21:21:52.000000000 +0000 @@ -139,4 +139,3 @@ #fi doexec "$@" -# vi: ts=4 noexpandtab diff -Nru cloud-init-23.3.3/tools/validate-yaml.py cloud-init-23.4.3/tools/validate-yaml.py --- cloud-init-23.3.3/tools/validate-yaml.py 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/validate-yaml.py 2024-02-02 21:21:52.000000000 +0000 @@ -23,5 +23,3 @@ sys.exit(1) else: sys.exit(0) - -# vi: ts=4 expandtab diff -Nru cloud-init-23.3.3/tools/xkvm cloud-init-23.4.3/tools/xkvm --- cloud-init-23.3.3/tools/xkvm 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tools/xkvm 2024-02-02 21:21:52.000000000 +0000 @@ -705,5 +705,3 @@ else main "$@" fi - -# vi: ts=4 expandtab syntax=sh diff -Nru cloud-init-23.3.3/tox.ini cloud-init-23.4.3/tox.ini --- cloud-init-23.3.3/tox.ini 2023-10-24 14:53:43.000000000 +0000 +++ cloud-init-23.4.3/tox.ini 2024-02-02 21:21:52.000000000 +0000 @@ -27,7 +27,7 @@ mypy==0.950 pylint==2.13.9 pytest==7.0.1 -ruff==0.0.282 +ruff==0.0.285 types-jsonschema==4.4.2 types-oauthlib==3.1.6 types-passlib==1.7.7.12

Page not found