fix(runtime): package Foo; lexical scoping + DBI parity Phase 10#551
Merged
fix(runtime): package Foo; lexical scoping + DBI parity Phase 10#551
Conversation
Full suite ran cleanly in 192s (no Gofer STORE/set_err infinite loop this time), so we now have real numbers: 5566/5944 passing (94%), 76/200 files failing, 378 failing subtests. Key finding: the per-file failure distribution is extremely skewed. t/10examp.t alone accounts for ~25% of all failures because the file dies at test 50 with "Undefined subroutine &main::test_dir" after `do "./t/lib.pl"` -- a PerlOnJava interpreter bug, not a DBI issue. Isolated repros work. Revised plan: - Drop the "Full Phase 10" big-scope plan (Profile + Kids + Executed + swap_inner_handle + take_imp_data, then flip $DBI::PurePerl = 0). Keeping the flag true means all current skip paths stay intact, so that plan would only land net-zero until every XS-only feature is reimplemented. Deferred indefinitely. - Phase 10 (new scope): debug the t/10examp.t `do "./t/lib.pl"` symbol-table issue. Up to ~965 subtests (193 x 5 wrappers). - Phase 11: file locking for DBM tests (~95 subtests). - Phase 12: execute_array (~25 subtests). - Phase 13: small triage fix-ups (~75 subtests). No code changes in this commit -- plan update only. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Perl 5 lexically scopes `package Foo;` to the enclosing block / sub /
eval / file, not just to the file. PerlOnJava's ScopedSymbolTable
already scoped it correctly at COMPILE time via packageStack, but both
backends were emitting an unscoped runtime update of
InterpreterState.currentPackage:
- JVM backend: InterpreterState.setCurrentPackageStatic(name)
- Interpreter: SET_PACKAGE opcode
Only the block form `package Foo { BLOCK }` was correctly scoped (via
PUSH_PACKAGE + DynamicVariableManager). Bare `package Foo;` leaked into
the caller's scope for the entire rest of the process.
The leak was latent until we switched to upstream DBI 1.647, which
pulls in Carp::caller_info. Carp does:
{
package DB;
@call_info{...} = caller($i);
}
as a standard debugger-compatibility shim. After any Test::More call
fired, the runtime current-package tracker was stuck at "DB". Later
`do "./t/lib.pl"` in t/10examp.t then compiled the loaded file in
"DB", installing `sub test_dir` as `DB::test_dir` instead of
`main::test_dir`. Line 174's `test_dir()` call failed with "Undefined
subroutine &main::test_dir".
Fix:
- New InterpreterState.setCurrentPackageLocal(name): push the current
package scalar onto DynamicVariableManager, then set the new value.
Scope teardown (localTeardown / POP_LOCAL_LEVEL) auto-restores.
- EmitOperator.handlePackageOperator (JVM): emit the scoped variant.
- CompileOperator (interpreter): always emit PUSH_PACKAGE; the
`isScoped` annotation is redundant since all `package X;` are scoped
in Perl 5.
Impact on `jcpan -t DBI` full suite:
- Before: 5566/5944 passing (94%), 378 fail
- After: 6210/6600 passing (94%), 390 fail
- Delta: +644 more subtests pass (+656 executed overall)
Per-file: `t/10examp.t` and its 4 wrappers each went from ~49/242
executed subtests to ~200+/242.
`make` passes; no unit-test regressions.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fglock
changed the title
…t of scope
Triaged t/14utf8.t, t/02dbidrv.t, t/06attrs.t, t/08keeperr.t, t/19fhtrace.t.
Each remaining failure maps to one of:
- PerlOnJava infra gaps out of DBI scope:
- t/14utf8.t: Encode::_utf8_on flag not preserved across hash keys.
- t/19fhtrace.t: PerlIO ':via' and ':scalar' custom layers not
implemented.
- Work already tracked elsewhere:
- t/02dbidrv.t: $dbh->DESTROY -> $drh err propagation, covered by
a separate DESTROY PR.
- Deep PerlOnJava interpreter issue flagged for future dive:
- t/06attrs.t and t/08keeperr.t both affected by a bug in
DBI::PurePerl's _install_method eval-STRING wrapper where
`$h->{dbi_pp_last_method} = $method_name;` persists inside
the wrapper (verified with injected print) but vanishes after
the wrapper returns. Minimal reproductions outside DBI all work
correctly - the bug only triggers inside DBI's actual wrapper
with full DBI initialisation. Not cost-effective to debug for
~8 subtests. Documented for future investigation.
Docs-only commit; no behaviour change.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Investigated the DBI::PurePerl _install_method wrapper failures
flagged in the previous triage commit. Traced the exact point of
divergence between jperl and real-perl-plus-PurePerl.
Root cause is a reproducible PerlOnJava bug in list-form `local`
assignment inside eval-STRING-compiled subroutines:
my $h = { x => 0 };
my $sub = eval q{ sub { local ($h->{x}) = 99; print "x=$h->{x}\n" } };
$sub->(); # prints "x=0" on jperl (WRONG); "x=99" on real perl
The scope-entry/exit machinery works (values restore on sub exit)
but the right-hand-side assignment is silently dropped. Scalar
form `local $h->{x} = 99;` (no parens) works correctly. Same bug
affects `local ($a[0]) = 99;`.
Why this breaks DBI: every DBI::PurePerl method wrapper is built
via eval STRING and contains `local ($h->{dbi_pp_call_depth}) = $call_depth;`.
Because the assignment is a no-op, nested wrapper entries all see
call_depth=0, and the innermost wrapper (set_err) incorrectly fires
the "failed" error message instead of letting it bubble to the
outermost (do). Result: "set_err failed" instead of "do failed".
Compare the same trace on real perl vs jperl:
real perl jperl
[CALLDEPTH do] h.cd=0 [CALLDEPTH do] h.cd=0
[CALLDEPTH prepare] h.cd=1 [CALLDEPTH prepare] h.cd=0 <- wrong
[CALLDEPTH set_err] h.cd=2 [CALLDEPTH set_err] h.cd=0 <- wrong
err: "db do failed: ..." err: "db set_err failed: ..."
Adds:
- dev/known-bugs/local_list_assign_eval_string.pl - minimal repro
demonstrating the bug with hash-element and array-element
list-form local assignments inside eval STRING.
- Updated Phase 13 triage section in dbi_test_parity.md to
replace the "deep interaction" hand-wave with the precise root
cause, reproduction, affected code surface, and expected impact
if fixed.
Likely fix area: JVM emitter / bytecode compiler path for
LIST_ASSIGN targeting LOCAL HASH_ELEMENT / ARRAY_ELEMENT, in the
eval-STRING compilation mode (file-scope compilation works). No
code change in this commit -- bug report only.
If fixed, expect +10-15 DBI suite subtests plus latent benefit
for any CPAN module using eval-STRING-generated accessors that
localize hash/array elements (Moose/MouseX/similar).
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
`local ($h->{key}) = value;` and `local ($a[i]) = value;` in the
bytecode-interpreter backend silently dropped the right-hand-side
assignment. The `local` scope-entry/exit machinery fired correctly
(the value was restored on scope exit), but no SET happened in
between -- effectively a no-op. Scalar form `local $h->{key} = v;`
(no outer parens) was unaffected; the JVM backend was unaffected
for file-scope code but hit the bug via eval STRING (which compiles
to bytecode).
Root cause: `CompileAssignment.handleLocalListAssignment` iterated
over the LHS list and only emitted bytecode when an element was
`OperatorNode("$", IdentifierNode)` (i.e. a plain `$name`). Elements
that were `BinaryOperatorNode` -- hash-element (`$h->{k}`), array-
element (`$a[i]`), arrow chains (`$obj->method->{k}`) -- fell
through both the size==1 special case and the main loop with no
bytecode emitted.
Fix: added a `BinaryOperatorNode` branch in both paths:
- size==1 case: compile element -> PUSH_LOCAL_VARIABLE ->
SET_SCALAR from RHS (matches the existing scalar-context
`local EXPR = RHS` handler at the top of handleLocalAssignment).
- multi-element loop: compile element -> PUSH_LOCAL_VARIABLE ->
ARRAY_GET RHS[i] -> SET_SCALAR.
Why this matters for DBI: DBI::PurePerl's `_install_method` wraps
every method in an eval-STRING'd sub containing
`local ($h->{dbi_pp_call_depth}) = $call_depth;`. Because the
assignment was dropped, every nested wrapper saw call_depth=0,
so error/warning messages fired from the innermost wrapper (set_err)
instead of bubbling to the outermost (do). Users saw
"DBD::Foo::db set_err failed: ..." instead of "db do failed: ...".
Impact on `jcpan -t DBI` full suite:
- 6210 -> 6256 passing (+46 subtests)
- 76 -> 64 failed files (-12)
Latent bug also affecting any CPAN module with eval-STRING-generated
subs that localize hash/array elements (DBI::PurePerl was the most
visible; Moose/MouseX-style accessor generators may also benefit).
Regression test added at dev/known-bugs/local_list_assign_eval_string.pl
(renamed for history; still present as the reproducer). The script
now passes on both JVM and --interpreter backends.
`make` passes; no unit-test regressions.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CPAN's DB_File, BerkeleyDB, SDBM_File, GDBM_File, NDBM_File, ODBM_File
are pure-XS modules with no pure-Perl fallback. In PerlOnJava, our
XSLoader::load was silently returning success for them, so `require
DB_File` appeared to work but the XS helpers like DB_File::constant
were never defined. The first real use (e.g. `tie %h, 'DB_File', ...`)
triggered infinite AUTOLOAD -> constant -> AUTOLOAD recursion ending
in StackOverflowError.
CPAN test runners like DBI's t/50dbm_simple.t probe optional DBM
backends with:
@dbm_types = grep { eval { require "$_.pm" } } @dbms;
plan skip_all => "No DBM modules available" unless @dbm_types;
This pattern requires `require` to FAIL for unavailable backends.
Our silent success broke it.
Fix:
- XS_ONLY_NOT_SUPPORTED blacklist in both XSLoader.pm and
XSLoader.java (kept in sync). XSLoader::load dies with a clear
"XS module not supported on PerlOnJava" message, which the
caller's `eval` catches and the backend probe falls through.
- installEndBlockStubs("BerkeleyDB"): registers a no-op Perl sub
for BerkeleyDB::Term::close_everything. The module's END block
is registered at compile time, BEFORE our XSLoader::load dies
at runtime, so the END queue still fires at interpreter shutdown.
Without the stub, the program exits non-zero and prove counts it
as a failed test program even if it SKIPped all subtests.
Impact on `jcpan -t DBI` full suite:
- Failing subtests: 344 -> 144 (-200)
- Failing files: 64 -> 48 (-16)
Per-file wins:
t/50dbm_simple.t + variants: 16/38 fail x 5 -> SKIP x 5
t/52dbm_complex.t: partial crash -> SKIP
t/53sqlengine_adv.t: crash -> SKIP
t/49dbd_file.t (base): 9/65 fail -> 65/65 pass
The "passing" column drops from 6256 -> 5992 because ~464 subtests
that formerly ran (and mostly failed) inside these files now skip
entirely — the correct outcome for CPAN-style backend probing.
make passes; no unit-test regressions.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four cumulative fixes exposed by the DBI test suite:
Overall
jcpan -t DBIprogressionpackage Foo;scoping)local (...) = vlist-form)Per-file improvements this PR brings to the DBM/Gofer/SqlEngine family: tests that were noisily crashing now cleanly SKIP or pass:
t/50dbm_simple.t+ 4 variants: 16 failures each → SKIP with "No DBM modules available"t/52dbm_complex.t,t/53sqlengine_adv.t: crash → SKIPt/49dbd_file.t(base): 9/65 → 65/65 passPhase 10: runtime-scoped
package Foo;{ package DB; ... }insideCarp::caller_infoleaked "DB" intoInterpreterState.currentPackageforever. Subsequentdo FILEcompiled loaded files inDBnamespace, sosub test_dirbecameDB::test_dir— invisible tomain::.Fix: new
InterpreterState.setCurrentPackageLocal(name)pushes the current package ontoDynamicVariableManagerbefore setting. JVM emitter + interpreter compiler updated to always emit the scoped variant.Phase 10b: list-form
localon hash/array elementslocal ($h->{k}) = v;andlocal ($a[i]) = v;in the bytecode-interpreter backend silently dropped the RHS assignment.CompileAssignment.handleLocalListAssignmentonly emitted bytecode forOperatorNode("$" + IdentifierNode)elements;BinaryOperatorNodefell through with no bytecode.Fix: added
BinaryOperatorNodebranch in both single-element and multi-element paths, emittingPUSH_LOCAL_VARIABLE+SET_SCALAR(+ARRAY_GETfor multi-element lists).Why this breaks DBI: DBI::PurePerl's wrappers contain
local ($h->{dbi_pp_call_depth}) = $call_depth;. The no-op meant nested wrappers saw call_depth=0, so error messages fired from the innermost wrapper (set_err) instead of bubbling to the outermost (do).Phase 11: XSLoader rejects known pure-XS modules cleanly
CPAN's DB_File, BerkeleyDB, SDBM_File, GDBM_File, NDBM_File, ODBM_File are pure-XS with no pure-Perl fallback. PerlOnJava's XSLoader was silently returning success, so
require DB_Fileappeared to work but XS helpers likeDB_File::constantwere never defined, and first use triggered infinite AUTOLOAD recursion →StackOverflowError.CPAN test runners probe with
eval { require "$_.pm" }and rely onrequirefailing for unavailable backends. Silent success broke that.Fix:
XS_ONLY_NOT_SUPPORTEDblacklist inXSLoader.pmandXSLoader.java(kept in sync): dies with clear error for blacklisted modules.installEndBlockStubs("BerkeleyDB"): registers no-opBerkeleyDB::Term::close_everythingso the module's already-registered END block doesn't abort at shutdown and taint the program's exit status.Files changed
src/main/java/org/perlonjava/backend/bytecode/InterpreterState.java— Phase 10 scoped-package helper.src/main/java/org/perlonjava/backend/jvm/EmitOperator.java— Phase 10 JVM emission.src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java— Phase 10 interpreter emission.src/main/java/org/perlonjava/backend/bytecode/CompileAssignment.java— Phase 10b BinaryOperatorNode handling.src/main/java/org/perlonjava/runtime/perlmodule/XSLoader.java— Phase 11 blacklist + END stubs.src/main/perl/lib/XSLoader.pm— Phase 11 blacklist (fallback path).dev/modules/dbi_test_parity.md— baseline updates, phase docs, triage.dev/known-bugs/local_list_assign_eval_string.pl— regression repro for Phase 10b.Test plan
makepasses onfeature/dbi-phase10-profile(no unit-test regressions).jcpan -t DBIrun: 5566 → 5992 passing (+426), 378 → 144 failing (-234).t/10examp.t: 49 → 208+ passing per variant.t/06attrs.t: 164 → 166 passing.t/08keeperr.t: 88 → 91 passing.t/50dbm_simple.t: crashes → SKIP cleanly.t/49dbd_file.t(base): 56 → 65 passing.Generated with Devin