fix: avoid copy constructor instantiation in shared_ptr fallback cast

[virtuald]

Description

Original bug at https://github.com/pytorch/pytorch/actions/runs/24007816974/job/70014373231.

I haven't manually validated this fix yet, it might be all slop. I don't think the tests are really in the right shape to test/validate the problem... I'll take a closer look tonight, but for now Here's GPT-5.4's explanation:

---

Problem and fix

Summary

A regression in pybind11's std::shared_ptr<T> caster caused the reference_internal fallback path to instantiate copy/move constructor helpers even when no copy or move was actually needed.

This broke downstream projects such as PyTorch when building with Clang warning settings like:

• -Werror=deprecated-copy-with-user-provided-dtor
• -Werror=deprecated-copy-with-dtor

What regressed

In include/pybind11/cast.h, the fallback path for returning std::shared_ptr<T> to Python when T is bound with neither std::shared_ptr holder nor py::smart_holder was changed to:

if (parent) {
    return type_caster_base<type>::cast(
        srcs, return_value_policy::reference_internal, parent);
}

That looks reasonable, but type_caster_base<type>::cast(...) forwards through code that eagerly instantiates:

• make_copy_constructor(...)
• make_move_constructor(...)

via type_caster_base::cast(const cast_sources &, ...).

Even though the reference_internal path does not use those constructors at runtime, their instantiation is enough for Clang to diagnose implicit copy construction for types that have a user-declared destructor.

Downstream impact

PyTorch hit this in CI after updating pybind11. The failing patterns came directly from code such as:

• torch/csrc/jit/python/python_tracer.cpp
  • m.def("_get_tracing_state", []() { return getTracingState(); });
• torch/csrc/jit/python/script_init.cpp
  • methods returning std::shared_ptr<InterfaceType> from CompilationUnit

The relevant types include user-declared destructors, e.g.:

• TracingState::~TracingState();
• InterfaceType::~InterfaceType() override = default;

Clang then reports errors originating from pybind11/detail/type_caster_base.h:1682, where the copy-constructor helper lambda is instantiated.

Root cause

The bug is not in lifetime handling itself. The lifetime model is correct: for this path, pybind11 should create a Python object referring to the existing C++ object and tie its lifetime to parent using reference_internal.

The regression is that the implementation routed through a helper that always prepares copy/move constructor callbacks, even though this fallback path is strictly non-owning and does not need them.

Fix

Use type_caster_generic::cast(...) directly for the parent/reference_internal fallback and pass null copy/move constructor callbacks:

if (parent) {
    return type_caster_generic::cast(
        srcs, return_value_policy::reference_internal, parent, nullptr, nullptr);
}

This preserves the intended behavior:

• no ownership transfer
• reference_internal keep-alive behavior remains intact
• no unnecessary copy/move constructor instantiation

Tests added

Two kinds of tests were added:

1. Existing smart-holder property aliasing coverage

In tests/test_class_sh_property.py, the original aliasing test was kept and a stronger multi-read version was added:

• verifies repeated property reads alias the same underlying member
• ensures updates through one Python handle are visible through another

2. Direct downstream-style PyTorch regression reproducer

New files:

• tests/pybind11_pytorch_regressions.cpp
• tests/test_pytorch_downstream_regressions.py

These extract the failing PyTorch patterns into a minimal pybind11 test module:

• a TracingState-like type returned from a free function as std::shared_ptr<T>
• an InterfaceType-like object returned from a CompilationUnit method as std::shared_ptr<T>

For the dedicated test target, Clang is given:

• -Werror=deprecated-copy-with-user-provided-dtor
• -Werror=deprecated-copy-with-dtor

This ensures the regression is caught at compile time.

Suggested changelog entry:

• Fix std::shared_ptr fallback casting to avoid unnecessary copy-constructor instantiation in reference_internal paths

[rwgk]

Thanks for the reproducer! I can confirm that it reproduces the compilation error on my workstation (Ubuntu clang version 18.1.3 (1ubuntu1)).

Looks good to me. My suggestions are just to make things more intuitive and easier to discover.

[virtuald]

If you're happy then I'm happy too

[rwgk]

I'm happy, but older clang versions need some more convincing: commit 87bf519

For easy reference: clang 5 and 11 failures observed here: https://github.com/pybind/pybind11/actions/runs/24018953764/job/70043719610?pr=6028