Make `to_numpy` conversions consistent with NumPy `dtype`

[marcosdiezgarcia]

Moved discussion in #462 into this separate issue.

Other PythonCall issues related to NumPy dtype and Numpy arrays that may be relevant:

• Use "native" types when possible #319
• dtype of NumPy array created from Julia array is object on Julia nightly #439
• Working with type instabilities, coming from PyJulia #441
• numpy functions don't treat Any[...] arrays like Python lists #486

My system details (click to expand)

Julia

julia> versioninfo()
Julia Version 1.12.1
Commit ba1e628ee49 (2025-10-17 13:02 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 48 × AMD EPYC 7V13 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, znver3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 48 virtual cores)

julia> Pkg.status()
Status `~/temp/Project.toml`
  [992eb4ea] CondaPkg v0.2.33
  [6099a3de] PythonCall v0.9.28

Python Environment

• python 3.13.9
• numpy 2.3.4

I noticed a difference in how the PythonCall to_numpy() function and NumPy treat property values of dtype as follows:

julia> using PythonCall

julia> Py(rand(10)).to_numpy().dtype
Python: dtype('float64')

julia> Py([[1,2,3], [4,5,6]]).to_numpy().dtype
Python: dtype('O')

I would expect the latter to be dtype('int64') to match Python:

>>> import numpy
>>> a = numpy.array([[1,2,3], [4,5,6]])
>>> a.dtype
dtype('int64')

While Julia does provide Matrix{Int64} and to_numpy() outputs dtype('int64') as I would expect

julia> [1 2 3; 4 5 6] |> typeof
Matrix{Int64} (alias for Array{Int64, 2})

julia> Py([1 2 3; 4 5 6]).to_numpy().dtype
Python: dtype('int64')

I am working on a project where I need to use PythonCall to deal with Vector{Vector{Int64}} instances and changing into Matrix{Int64} is not an option.

In Julia, using PythonCall to_numpy(), it is clear that dtype('int64') is only output for Vector{Int64} not Vector{Vector{Int64}} nor Vector{Vector{Vector{Int64}}} regardless of how many levels of nesting:

julia> Py([1, 2, 3]).to_numpy().dtype
Python: dtype('int64')

julia> Py([[1,2,3], [4,5,6]]).to_numpy().dtype
Python: dtype('O')

julia> Py([[[1,2,3], [4,5,6]], [[7,8,9], [10,11,12]]]).to_numpy().dtype
Python: dtype('O')

Whereas in Python, dtype('int64') is output for all the above:

>>> numpy.array([1, 2, 3]).dtype
dtype('int64')

>>> numpy.array([[1,2,3], [4,5,6]]).dtype
dtype('int64')

>>> numpy.array([[[1,2,3], [4,5,6]], [[7,8,9], [10,11,12]]]).dtype
dtype('int64')

Thus the value of the property .dtype in NumPy is defined based on the innermost elements in the array, whereas in Julia the value of .dtype upon using to_numpy() is not based on the innermost elements (i.e. 1, 2, 3, etc) but on the whole structure containing them (i.e Vector{Int64} for the 2nd array, and Vector{Vector{Int64}} for the 3rd array).

I was expecting the same behaviour from Python's NumPy and the conversions from to_numpy() given by PythonCall, but it turns out the conversion does not agree with NumPy on dtype property values.

[marcosdiezgarcia]

I found a simple fix to JlWrap/array.jl (line 370) in PythonCall 0.9.28, which resolves the conversion issue I described above:

try:
    import numpy
-   arr = numpy.array(arr, dtype=dtype)
+   arr = numpy.array(list(arr), dtype=dtype)
except ImportError:
    pass

Now I get what I was looking for:

julia> Py([1, 2, 3]).to_numpy().dtype
Python: dtype('int64')

julia> Py([[1, 2, 3], [4, 5, 6]]).to_numpy().dtype
Python: dtype('int64')

julia> Py([[[1, 2, 3], [4, 5, 6]]]).to_numpy().dtype
Python: dtype('int64')

This also works if the underlying elements are of other type like floats or strings, e.g.:

julia> Py([[1.0,2.0,3.0], [4.1,5.1,6.1]]).to_numpy()
Python:
array([[1. , 2. , 3. ],
       [4.1, 5.1, 6.1]])

julia> Py([[1.0,2.0,3.0], [4.1,5.1,6.1]]).to_numpy().dtype
Python: dtype('float64')

julia> Py([["a","b","c"], ["d","f","g"]]).to_numpy()
Python:
array([['a', 'b', 'c'],
       ['d', 'f', 'g']], dtype='<U1')

The suggested fix also works for Vector{SparseVector{Int}}, which was also discussed in #462 :

julia> v
5-element SparseVector{Int64, Int64} with 4 stored entries:
  [1]  =  1
  [3]  =  -5
  [4]  =  2
  [5]  =  3

julia> w
5-element SparseVector{Int64, Int64} with 3 stored entries:
  [1]  =  10
  [2]  =  20
  [5]  =  30

julia> [v, w]
2-element Vector{SparseVector{Int64, Int64}}:
 sparsevec([1, 3, 4, 5], [1, -5, 2, 3], 5)
 sparsevec([1, 2, 5], [10, 20, 30], 5)

julia> Py([v, w]).to_numpy()
Python:
array([[ 1,  0, -5,  2,  3],
       [10, 20,  0,  0, 30]])

julia> Py([v, w]).to_numpy().dtype
Python: dtype('int64')

[cjdoris]

As @MilesCranmer already explained elsewhere there is no inconsistency here.

Since Python has no inbuilt notion of multidimensional arrays, numpy allows you to construct an array from a list of lists as a convenience. However the resulting array is not a list of lists, there is no nesting, it is a flat 2D array whose elements are int64s.

On the other hand Julia does have multidimensional arrays natively, and so PythonCall's to_numpy() function preserves the dimension
That is, Vector{Vector{Int64}} is a 1D array so to_numpy() is a 1D array also. Its elements are themselves arrays (i.e. vectors of int64s) and not int64s, so the dtype is (correctly) object.

This is a design decision but we're sticking with it. It would be more surprising if a 1D Julia array got converted to a 2D numpy array.

You said that you absolutely cannot use Matrix{Int64} in your application. Can you explain why? I strongly suspect there is an XY issue here.

[marcosdiezgarcia]

You said that you absolutely cannot use Matrix{Int64} in your application. Can you explain why?

I am interacting with another Julia package functionality, entirely outside my control, whose output is Vector{Vector{Int64}} and call it from Python via PythonCall/juliacall in my code.

The fix or workaround I mentioned does the job.

I understand that there are design decisions behind how PythonCall deals with multi-dimensional arrays taking advantage of Julia types. I was wondering if (1) the suggested fix would cause any "inconvenience/friction" with the rest of PythonCall that would prevent it from being adopted, and (2) there was some other way to get dtype('int64') for Vector{Vector{Int64}} (Matrix{Int64} not being an option).

[MilesCranmer]

I am interacting with another Julia package functionality, entirely outside my control, whose output is Vector{Vector{Int64}} and call it from Python via PythonCall/juliacall in my code.

Why not just convert it to a Matrix before the .to_numpy() code?

[marcosdiezgarcia]

I just thought first of the list() call. But yes, actually converting to Matrix does also work.

julia> m = reduce(hcat, [[1,2,3], [4,5,6]])
3×2 Matrix{Int64}:
 1  4
 2  5
 3  6

julia> Py(m).to_numpy().dtype
Python: dtype('int64')

Thanks for suggesting it and the helpful discussions.

[marcosdiezgarcia]

Using the list() call as I suggested handles also the case of Vector{Vector{Vector{Int}}}, but the conversion to Matrix using reduce and hcat still leads to dtype('O'):

julia> m = reduce(hcat, [[[1,2,3], [4,5,6]], [[7,8,9], [10,11,12]]])
2×2 Matrix{Vector{Int64}}:
 [1, 2, 3]  [7, 8, 9]
 [4, 5, 6]  [10, 11, 12]

julia> Py(m).to_numpy().dtype
Python: dtype('O')

As discussed, this is expected since Vector{Int64} are considered as objects.

[MilesCranmer]

Try this:

julia> to_array(x) = x isa AbstractVector ? stack(map(to_array, x)) : x
to_array (generic function with 1 method)

julia> to_array([1])
1-element Vector{Int64}:
 1

julia> to_array([[[[[[[[1.0]]]]]]]])
1×1×1×1×1×1×1×1 Array{Float64, 8}:
[:, :, 1, 1, 1, 1, 1, 1] =
 1.0

This will collapse anything to a single Array{T,N} type, after which you can do your Py().to_numpy call and get a regular numpy array