Docs: clarify -p vs PYTEST_PLUGINS plugin loading

[Dooopinder]

---

Summary

This PR clarifies the difference between -p and PYTEST_PLUGINS in the plugin documentation and fixes an incorrect command example.

Details

• Fixes the incorrect example pytest --disable-plugin-autoload -p NAME,NAME2
  → replaced with the correct repeated form -p NAME -p NAME2
• Clarifies the different use-cases:
  • -p early-loads (or disables) a plugin by name or entry point for a single invocation
  • PYTEST_PLUGINS loads plugin modules during startup, commonly used when testing plugins
• Adds guidance to avoid duplicate plugin registration when plugin autoloading is disabled
• Adds concrete examples for both mechanisms

closes #13388
Changelog: not needed (documentation-only change).

[nicoddemus]

Thanks!

[patchback]

Backport to 9.0.x: 💚 backport PR created

✅ Backport PR branch: patchback/backports/9.0.x/6d0aff14ef1aea1636b543c173124a18e61772ee/pr-14113

Backported as #14117

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[Dooopinder]

No Problem happy to help

[hseg]

Sorry it took me so long to comment on this, thanks for taking this up.
IMO, the documented difference is still unclear.
It seems the following are the differences:

• -p supports blacklisting (with -p no:..)
• -p takes as arguments package names or entry points, PYTEST_PLUGINS takes as arguments module names

However, what difference does the clause

is processed early during startup

supposed to indicate? Is PYTEST_PLUGINS processed later, for example?
The use-case difference is also unconvincing -- couldn't the "testsuite-global config override" usecase for PYTEST_PLUGINS be served by setting PYTEST_ADDOPTS=-p ...?

Speculating, is the difference that -p is restricted to adding installed plugins (and so can't eg use plugins in the package under development), while PYTEST_PLUGINS hooks directly into the plugin loading infra and can (with an appropriate PYTHONPATH) load arbitrary modules?

[Dooopinder]

If you think this distinction should be phrased more narrowly or with less emphasis, I’m very happy to adjust the wording and work on it more if the maintainers also feel the same .

[hseg]

The thing is that I don't understand the difference 100% myself, so I'm not confident with my suggestions (note my hedge in calling the above a speculation).
But if I understood correctly, perhaps the way to write this as:

.. note::
    <they both explicitly control, as before>

    :option:`-p` acts on entry-points -- it loads (or disables with `-p
    no:<name>`) plugins by looking them up in the list of installed entry
    points.

    :envvar:`PYTEST_PLUGINS` acts on modules -- it loads the specified modules
    from :envvar:`PYTHONPATH`.

    Thus, :option:`-p` is usually used by users of the plugins, while
    :envvar`:PYTEST_PLUGINS` is usually useful for plugin development (where the
    plugin module might not yet be installed on the test system).

    <don't double-load>

    <examples>

What do you think?

[hseg]

Not sure my understanding is correct -- but then, I'm not sure I understand my test case (pytest-recording) well either.
It seems I never needed to add pytest_recording itself to PYTEST_PLUGINS to test it. At least, the test results don't get any worse when I remove it.
Am confused and tired, I think I'll leave it here for today.

[Dooopinder]

The intent is not to draw a hard architectural line, but to make two things explicit: -p is early, per-invocation control (including blocking), while PYTEST_PLUGINS is explicit module loading during startup.
That’s why PYTEST_PLUGINS commonly appears in test suites and plugin development.
If removing pytest_recording didn’t change results, it likely means it was already being loaded via another path, so the env var was redundant.
This is also why the note warns against mixing mechanisms — it’s easy to think you’re controlling loading when you’re not.

[hseg]

Again I find myself confused. At the end of the day, what does PYTEST_PLUGINS actually do? And how does -p's earliness affect things?

[Dooopinder]

Would it help if I shared a couple of minimal test cases you can run locally, when I have some free time to put them together?

[hseg]

Perhaps - that would be appreciated, thanks!

[Dooopinder]

Hi @hseg — I built a minimal repro and read through the issue you opened.
It shows that -p can succeed while loading a top-level package with no hooks, while only the entry-point submodule actually activates the plugin.
This seems like a UX issue (silent success), so I think adding a short warning note in the docs for this edge case would clear up the confusion — what do you think? repo

[hseg]

Hrm. Yeah, the script I posted to Dooopinder/p_dash_proof#2 shows that -p only validates that the module exists, which is a problem since the correct spelling to get the module to load is -p plugins.live_plugin.

TBH, I consider this a misfeature, that should actually be solved within pytest itself. Documenting this is a useful stopgap, but is not a long-term solution.
Not sure what a correct solution would be. Perhaps checking whether the module is listed in the package entrypoints?

Additionally, this testsuite still doesn't manage to distinguish PYTEST_PLUGINS from -p for me, which was my initial problem.

[hseg]

Sorry for the delay in response, was swamped the past few days.

[Dooopinder]

no issue man in regards to delay and i also think after reading your comment not the test repo will try later that a good discussion i think is required to properly tackle this with the maintainers because again their input would help more