Get process output from Malt, instead of capturing it ourselves #84

giordano · 2026-01-19T16:00:14Z

julia> using Malt, ParallelTestRunner

julia> testsuite = Dict(
           "abort" => quote
               @info("I'm about to crash")
               abort() = ccall(:abort, Nothing, ())
               abort()
           end,
           "works" => quote
               println("This will work")
               @test true
           end,
           "silent" => quote
               @test true
           end,
       );

julia> runtests(ParallelTestRunner, ["--jobs=1"]; testsuite);
Running 1 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │          │ ──────────────── CPU ──────────────── │
Test  (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
abort     (13) |         crashed at 2026-01-19T15:58:30.948
works     (14) │     0.07 │   0.00 │  0.0 │       5.40 │  1017.21 │
silent    (14) │     0.00 │   0.00 │  0.0 │       0.00 │  1017.21 │

Output generated during execution of 'abort':
┌ [ Info: I'm about to crash
│ 
│ [61478] signal 6 (-6): Aborted
│ in expression starting at none:1
│ unknown function (ip: 0x7f70552973dc) at /usr/lib/x86_64-linux-gnu/libc.so.6
│ gsignal at /usr/lib/x86_64-linux-gnu/libc.so.6 (unknown line)
│ abort at /usr/lib/x86_64-linux-gnu/libc.so.6 (unknown line)
│ abort at ./REPL[15]:4
│ unknown function (ip: 0x7f704df313cf) at (unknown file)
│ macro expansion at ./REPL[15]:5 [inlined]
│ macro expansion at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:280 [inlined]
│ macro expansion at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
│ macro expansion at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:280 [inlined]
│ macro expansion at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
│ macro expansion at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:279 [inlined]
│ macro expansion at ./timing.jl:689 [inlined]
│ top-level scope at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:278
│ jl_toplevel_eval_flex at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/toplevel.c:1024
│ ijl_toplevel_eval at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/toplevel.c:1047
│ ijl_toplevel_eval_in at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/toplevel.c:1092
│ eval at ./boot.jl:489
│ inner at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:271
│ runtest at /home/mose/.julia/dev/ParallelTestRunner/src/ParallelTestRunner.jl:302
│ unknown function (ip: 0x7f704df23460) at (unknown file)
│ jl_apply at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
│ jl_f_invokelatest at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/builtins.c:881
│ #handle##0 at /home/mose/.julia/dev/Malt/src/worker.jl:120
│ unknown function (ip: 0x7f704df2323f) at (unknown file)
│ jl_apply at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
│ start_task at /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/src/task.c:1249
└ Allocations: 2975744 (Pool: 2975695; Big: 49); GC: 4

Output generated during execution of 'works':
[ This will work

Test Summary: | Pass  Error  Total  Time
  Overall     |    2      1      3  7.0s
    abort     |           1      1  4.6s
    works     |    1             1  0.0s
    silent    |    1             1  0.0s
    FAILURE

Error in testset abort:
Error During Test at none:1
  Got exception outside of a @test
  Malt.TerminatedWorkerException()
ERROR: Test run finished with errors

Note: still needs tests (can adapt the example above). Tests added

Requires JuliaPluto/Malt.jl#103. Fix #83.

This enables us to get the output generated by crashes jobs.

…sues

giordano · 2026-01-19T17:12:00Z

So, the remaining failures now are only due to buffering: the tests at

ParallelTestRunner.jl/test/runtests.jl

Lines 236 to 238 in 608f947

    
           "color" => quote 
        
               printstyled("Roses Are Red"; color=:red) 
        
           end

ParallelTestRunner.jl/test/runtests.jl

Lines 248 to 251 in 608f947

    
           "no color" => quote 
        
               print("Violets are ") 
        
               printstyled("blue"; color=:blue) 
        
           end

rely on the fact that the previous IO capturing mechanism was able to capture also from print-like function, without a newline which would flush the stream.

Unless someone knows a magic way to run julia unbuffered like python -u without using external tools like unbuffer/stdbuf (but decade-old issues like JuliaLang/julia#13050 don't inspire much hope), only thing I can think of is to append a println to those tests and live with the fact that lone prints without newlines/flushing won't be captured anymore. But that's hopefully a non-super-common situation, and I'd argue the ability of getting the output of crashed workers overcomes that limitation.

giordano · 2026-01-19T17:19:34Z

@vchuravy @maleadt with the caveat that we need to wait for the upstream PR JuliaPluto/Malt.jl#103, I'd appreciate some feedback about the design of this PR and its tradeoffs.

src/ParallelTestRunner.jl

giordano · 2026-01-26T09:56:36Z

This is now ready for review!

vchuravy · 2026-01-26T13:31:44Z

src/ParallelTestRunner.jl


+# Adapted from `Malt._stdio_loop`
+function stdio_loop(worker::PTRWorker)
+    @async while !eof(worker.w.stdout) && Malt.isrunning(worker)


Obligatory: Why @async and not @spawn

No specific reason, I had followed the original Malt._stdio_loop. I pushed a change to replace @async -> Threads.@spawn

vchuravy · 2026-01-26T13:32:32Z

src/ParallelTestRunner.jl

-    wrkr = Malt.Worker(; exename, exeflags, env)
-    WORKER_IDS[wrkr.proc_pid] = length(WORKER_IDS) + 1
+    io = IOBuffer()
+    wrkr = PTRWorker(Malt.Worker(; exename, exeflags, env, monitor_stdout=false, monitor_stderr=false), io)


Can we fold this into the constructor?

Done! Also included the worker id as a field of the worker (with a global atomic counter), no need to keep a dictionary around for doing the mapping.

vchuravy · 2026-01-26T13:34:18Z

src/ParallelTestRunner.jl

+            break
+        end
+    end
+    @async while !eof(worker.w.stderr) && Malt.isrunning(worker)


Slight order weirdness... What happens if Malt.isrunning is true before we have consumed all the text?

Uhm, to this one I don't have a good answer

giordano · 2026-01-26T18:35:03Z

It's so nice to be able to see the output of a segfault 🥳

giordano added 2 commits January 19, 2026 15:57

Get process output from Malt, instead of capturing it ourselves

a292217

This enables us to get the output generated by crashes jobs.

DROP ME: temporarily point to my fork of Malt

eaa38d8

giordano mentioned this pull request Jan 19, 2026

Add monitor_std{out,err} arguments to Worker constructor JuliaPluto/Malt.jl#103

Merged

giordano added 2 commits January 19, 2026 16:45

Almost completely fix handling of colours

e562950

Pass --color option directly to workers to completely fix colour is…

6227e30

…sues

giordano force-pushed the mg/crash-output branch from 81c7d58 to 6227e30 Compare January 19, 2026 17:01

Append newlines to flush streams in colorful output tests

e550df5

giordano force-pushed the mg/crash-output branch from 83db39f to ca5c3ae Compare January 19, 2026 17:48

Add tests about capturing output of crashed workers

02219b7

giordano force-pushed the mg/crash-output branch from ca5c3ae to 02219b7 Compare January 19, 2026 17:48

giordano commented Jan 20, 2026

View reviewed changes

src/ParallelTestRunner.jl Outdated Show resolved Hide resolved

giordano added 2 commits January 20, 2026 10:56

Use readavailable instead of readline

a33635e

Add more tests about output capturing

dbdf9d6

vchuravy reviewed Jan 20, 2026

View reviewed changes

src/ParallelTestRunner.jl Outdated Show resolved Hide resolved

giordano and others added 5 commits January 20, 2026 14:56

Replace isopen -> !eof

c295ac5

Fix Malt revision

c5078e1

Create a custom PTRWorker worker type to adapt to new upstream changes

6415b44

Update docstrings

431743f

Malt v1.4.0 is now registered

cc0c408

giordano requested review from maleadt and vchuravy January 26, 2026 09:56

vchuravy reviewed Jan 26, 2026

View reviewed changes

vchuravy approved these changes Jan 26, 2026

View reviewed changes

giordano added 2 commits January 26, 2026 14:17

Fold initialisation of PTRWorker in its constructor

140d737

@async -> Threads.@spawn in stdio_loop

bd8dca0

Include worker ID in PTRWorker structure

bcf2c50

vchuravy approved these changes Jan 26, 2026

View reviewed changes

giordano merged commit 39c1c38 into main Jan 26, 2026
42 of 43 checks passed

giordano deleted the mg/crash-output branch January 26, 2026 15:56

giordano restored the mg/crash-output branch January 26, 2026 15:57

giordano mentioned this pull request Jan 26, 2026

Enable Reactant compilation of + tracing through Breeze timestepping NumericalEarth/Breeze.jl#429

Open

giordano deleted the mg/crash-output branch January 27, 2026 01:50

maleadt mentioned this pull request Feb 2, 2026

Add feature to Malt to capture worker output #61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get process output from Malt, instead of capturing it ourselves #84

Get process output from Malt, instead of capturing it ourselves #84

Uh oh!

giordano commented Jan 19, 2026 •

edited

Loading

Uh oh!

giordano commented Jan 19, 2026

Uh oh!

giordano commented Jan 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

giordano commented Jan 26, 2026

Uh oh!

vchuravy Jan 26, 2026

Uh oh!

giordano Jan 26, 2026

Uh oh!

vchuravy Jan 26, 2026

Uh oh!

giordano Jan 26, 2026

Uh oh!

vchuravy Jan 26, 2026

Uh oh!

giordano Jan 26, 2026

Uh oh!

Uh oh!

giordano commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Get process output from Malt, instead of capturing it ourselves #84

Get process output from Malt, instead of capturing it ourselves #84

Uh oh!

Conversation

giordano commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

giordano commented Jan 19, 2026

Uh oh!

giordano commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

giordano commented Jan 26, 2026

Uh oh!

vchuravy Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

giordano Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

vchuravy Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

giordano Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

vchuravy Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

giordano Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

giordano commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

giordano commented Jan 19, 2026 •

edited

Loading

giordano commented Jan 19, 2026 •

edited

Loading