Update Spock Tests to run with `STRICT_TEST` mode by default by mhlidd · Pull Request #11032 · DataDog/dd-trace-java

mhlidd · 2026-04-01T20:02:33Z

What Does This Do

This PR updates Spock tests to run with STRICT_TEST by default. This means that all Environment Variables that are queried but not documented will fail the test. We fail the test by throwing an error when querying such environment variable, but if the test has automatic error handling we also throw an error during test cleanup to log the missing configurations from supported-configurations.json.

If a test is using fake configurations, the test itself can override the strictness mode with TEST to bypass this mechanism. Note that this is being done w/ many Instrumentation tests as they create fake decorators that register fake configs.

Motivation

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

pr-commenter · 2026-04-05T14:26:44Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775497568	1775503219
git_commit_sha	`b664cc0`	`64af647`
release_version	1.61.0-SNAPSHOT~b664cc0f2a	1.61.0-SNAPSHOT~64af64756a

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775505076	1775505076
ci_job_id	1570881599	1570881599
ci_pipeline_id	106256212	106256212
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-0k3buref 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-0k3buref 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053875
Total [baseline] (8.835 s) : 0, 8835228
Agent [candidate] (1.058 s) : 0, 1057872
Total [candidate] (8.873 s) : 0, 8873297
section iast
Agent [baseline] (1.224 s) : 0, 1223539
Total [baseline] (9.578 s) : 0, 9577752
Agent [candidate] (1.224 s) : 0, 1223749
Total [candidate] (9.553 s) : 0, 9553081

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	iast	1.224 s	169.664 ms (16.1%)
Total	tracing	8.835 s	-
Total	iast	9.578 s	742.524 ms (8.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.224 s	165.877 ms (15.7%)
Total	tracing	8.873 s	-
Total	iast	9.553 s	679.785 ms (7.7%)

gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.228 ms) : 0, 1228
BytebuddyAgent [baseline] (632.124 ms) : 0, 632124
BytebuddyAgent [candidate] (633.795 ms) : 0, 633795
AgentMeter [baseline] (29.381 ms) : 0, 29381
AgentMeter [candidate] (29.287 ms) : 0, 29287
GlobalTracer [baseline] (248.854 ms) : 0, 248854
GlobalTracer [candidate] (248.548 ms) : 0, 248548
AppSec [baseline] (31.97 ms) : 0, 31970
AppSec [candidate] (31.987 ms) : 0, 31987
Debugger [baseline] (59.071 ms) : 0, 59071
Debugger [candidate] (59.142 ms) : 0, 59142
Remote Config [baseline] (602.367 µs) : 0, 602
Remote Config [candidate] (598.592 µs) : 0, 599
Telemetry [baseline] (7.997 ms) : 0, 7997
Telemetry [candidate] (8.1 ms) : 0, 8100
Flare Poller [baseline] (6.48 ms) : 0, 6480
Flare Poller [candidate] (8.958 ms) : 0, 8958
section iast
crashtracking [baseline] (1.224 ms) : 0, 1224
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (801.09 ms) : 0, 801090
BytebuddyAgent [candidate] (801.354 ms) : 0, 801354
AgentMeter [baseline] (11.379 ms) : 0, 11379
AgentMeter [candidate] (11.369 ms) : 0, 11369
GlobalTracer [baseline] (239.315 ms) : 0, 239315
GlobalTracer [candidate] (239.159 ms) : 0, 239159
AppSec [baseline] (33.58 ms) : 0, 33580
AppSec [candidate] (30.835 ms) : 0, 30835
Debugger [baseline] (55.452 ms) : 0, 55452
Debugger [candidate] (58.423 ms) : 0, 58423
Remote Config [baseline] (519.942 µs) : 0, 520
Remote Config [candidate] (523.801 µs) : 0, 524
Telemetry [baseline] (13.994 ms) : 0, 13994
Telemetry [candidate] (14.34 ms) : 0, 14340
Flare Poller [baseline] (4.762 ms) : 0, 4762
Flare Poller [candidate] (4.403 ms) : 0, 4403
IAST [baseline] (26.079 ms) : 0, 26079
IAST [candidate] (25.788 ms) : 0, 25788

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062609
Total [baseline] (11.119 s) : 0, 11119344
Agent [candidate] (1.054 s) : 0, 1054231
Total [candidate] (11.046 s) : 0, 11046307
section appsec
Agent [baseline] (1.251 s) : 0, 1251281
Total [baseline] (11.22 s) : 0, 11220023
Agent [candidate] (1.246 s) : 0, 1245501
Total [candidate] (11.159 s) : 0, 11158878
section iast
Agent [baseline] (1.242 s) : 0, 1241543
Total [baseline] (11.419 s) : 0, 11419494
Agent [candidate] (1.253 s) : 0, 1253106
Total [candidate] (11.361 s) : 0, 11361354
section profiling
Agent [baseline] (1.185 s) : 0, 1185153
Total [baseline] (11.135 s) : 0, 11135239
Agent [candidate] (1.192 s) : 0, 1191604
Total [candidate] (11.095 s) : 0, 11094726

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.063 s	-
Agent	appsec	1.251 s	188.672 ms (17.8%)
Agent	iast	1.242 s	178.934 ms (16.8%)
Agent	profiling	1.185 s	122.544 ms (11.5%)
Total	tracing	11.119 s	-
Total	appsec	11.22 s	100.68 ms (0.9%)
Total	iast	11.419 s	300.15 ms (2.7%)
Total	profiling	11.135 s	15.895 ms (0.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.246 s	191.27 ms (18.1%)
Agent	iast	1.253 s	198.875 ms (18.9%)
Agent	profiling	1.192 s	137.373 ms (13.0%)
Total	tracing	11.046 s	-
Total	appsec	11.159 s	112.572 ms (1.0%)
Total	iast	11.361 s	315.047 ms (2.9%)
Total	profiling	11.095 s	48.419 ms (0.4%)

gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (637.346 ms) : 0, 637346
BytebuddyAgent [candidate] (631.466 ms) : 0, 631466
AgentMeter [baseline] (29.521 ms) : 0, 29521
AgentMeter [candidate] (29.384 ms) : 0, 29384
GlobalTracer [baseline] (250.169 ms) : 0, 250169
GlobalTracer [candidate] (248.776 ms) : 0, 248776
AppSec [baseline] (31.97 ms) : 0, 31970
AppSec [candidate] (32.087 ms) : 0, 32087
Debugger [baseline] (60.052 ms) : 0, 60052
Debugger [candidate] (59.942 ms) : 0, 59942
Remote Config [baseline] (596.386 µs) : 0, 596
Remote Config [candidate] (624.784 µs) : 0, 625
Telemetry [baseline] (8.068 ms) : 0, 8068
Telemetry [candidate] (8.047 ms) : 0, 8047
Flare Poller [baseline] (7.297 ms) : 0, 7297
Flare Poller [candidate] (6.63 ms) : 0, 6630
section appsec
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.209 ms) : 0, 1209
BytebuddyAgent [baseline] (661.375 ms) : 0, 661375
BytebuddyAgent [candidate] (660.756 ms) : 0, 660756
AgentMeter [baseline] (12.115 ms) : 0, 12115
AgentMeter [candidate] (11.998 ms) : 0, 11998
GlobalTracer [baseline] (250.228 ms) : 0, 250228
GlobalTracer [candidate] (248.516 ms) : 0, 248516
IAST [baseline] (24.79 ms) : 0, 24790
IAST [candidate] (24.553 ms) : 0, 24553
AppSec [baseline] (185.528 ms) : 0, 185528
AppSec [candidate] (183.416 ms) : 0, 183416
Debugger [baseline] (66.636 ms) : 0, 66636
Debugger [candidate] (66.103 ms) : 0, 66103
Remote Config [baseline] (610.685 µs) : 0, 611
Remote Config [candidate] (612.965 µs) : 0, 613
Telemetry [baseline] (8.727 ms) : 0, 8727
Telemetry [candidate] (8.557 ms) : 0, 8557
Flare Poller [baseline] (3.6 ms) : 0, 3600
Flare Poller [candidate] (3.506 ms) : 0, 3506
section iast
crashtracking [baseline] (1.244 ms) : 0, 1244
crashtracking [candidate] (1.26 ms) : 0, 1260
BytebuddyAgent [baseline] (812.764 ms) : 0, 812764
BytebuddyAgent [candidate] (825.217 ms) : 0, 825217
AgentMeter [baseline] (11.598 ms) : 0, 11598
AgentMeter [candidate] (12.18 ms) : 0, 12180
GlobalTracer [baseline] (242.213 ms) : 0, 242213
GlobalTracer [candidate] (241.915 ms) : 0, 241915
IAST [baseline] (26.246 ms) : 0, 26246
IAST [candidate] (26.147 ms) : 0, 26147
AppSec [baseline] (30.823 ms) : 0, 30823
AppSec [candidate] (33.531 ms) : 0, 33531
Debugger [baseline] (60.804 ms) : 0, 60804
Debugger [candidate] (56.724 ms) : 0, 56724
Remote Config [baseline] (524.471 µs) : 0, 524
Remote Config [candidate] (537.91 µs) : 0, 538
Telemetry [baseline] (14.306 ms) : 0, 14306
Telemetry [candidate] (14.356 ms) : 0, 14356
Flare Poller [baseline] (4.632 ms) : 0, 4632
Flare Poller [candidate] (4.249 ms) : 0, 4249
section profiling
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (691.708 ms) : 0, 691708
BytebuddyAgent [candidate] (697.007 ms) : 0, 697007
AgentMeter [baseline] (9.17 ms) : 0, 9170
AgentMeter [candidate] (9.127 ms) : 0, 9127
GlobalTracer [baseline] (206.953 ms) : 0, 206953
GlobalTracer [candidate] (207.936 ms) : 0, 207936
AppSec [baseline] (32.653 ms) : 0, 32653
AppSec [candidate] (32.622 ms) : 0, 32622
Debugger [baseline] (65.999 ms) : 0, 65999
Debugger [candidate] (66.229 ms) : 0, 66229
Remote Config [baseline] (575.743 µs) : 0, 576
Remote Config [candidate] (564.679 µs) : 0, 565
Telemetry [baseline] (7.924 ms) : 0, 7924
Telemetry [candidate] (7.958 ms) : 0, 7958
Flare Poller [baseline] (3.565 ms) : 0, 3565
Flare Poller [candidate] (3.635 ms) : 0, 3635
ProfilingAgent [baseline] (94.159 ms) : 0, 94159
ProfilingAgent [candidate] (93.771 ms) : 0, 93771
Profiling [baseline] (94.726 ms) : 0, 94726
Profiling [candidate] (94.338 ms) : 0, 94338

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775497568	1775503219
git_commit_sha	`b664cc0`	`64af647`
release_version	1.61.0-SNAPSHOT~b664cc0f2a	1.61.0-SNAPSHOT~64af64756a

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775505636	1775505636
ci_job_id	1570881600	1570881600
ci_pipeline_id	106256212	106256212
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-gtpvqzto 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-gtpvqzto 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 15 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-219.611µs; -52.199µs] or [-11.240%; -2.672%]	unstable [-1070.987µs; -204.938µs] or [-17.654%; -3.378%]	unstable [+0.470op/s; +434.530op/s] or [+0.026%; +24.422%]	1.818ms	5.429ms	1996.781op/s	1.954ms	6.067ms	1779.281op/s
scenario:load:petclinic:iast:high_load	worse [+454.953µs; +1308.686µs] or [+2.614%; +7.520%]	worse [+0.575ms; +2.010ms] or [+2.030%; +7.102%]	unstable [-43.452op/s; +18.327op/s] or [-16.537%; +6.975%]	18.284ms	29.600ms	250.188op/s	17.402ms	28.308ms	262.750op/s
scenario:load:petclinic:code_origins:high_load	better [-1.486ms; -0.595ms] or [-8.079%; -3.235%]	better [-2.022ms; -0.601ms] or [-6.822%; -2.028%]	unstable [-16.834op/s; +45.147op/s] or [-6.762%; +18.133%]	17.350ms	28.333ms	263.125op/s	18.390ms	29.645ms	248.969op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.23 ms) : 1218, 1241
.   : milestone, 1230,
iast (3.246 ms) : 3199, 3293
.   : milestone, 3246,
iast_FULL (6.143 ms) : 6080, 6206
.   : milestone, 6143,
iast_GLOBAL (3.726 ms) : 3660, 3791
.   : milestone, 3726,
profiling (2.555 ms) : 2529, 2581
.   : milestone, 2555,
tracing (1.927 ms) : 1910, 1945
.   : milestone, 1927,
section candidate
no_agent (1.25 ms) : 1237, 1262
.   : milestone, 1250,
iast (3.332 ms) : 3285, 3379
.   : milestone, 3332,
iast_FULL (6.038 ms) : 5977, 6100
.   : milestone, 6038,
iast_GLOBAL (3.626 ms) : 3568, 3685
.   : milestone, 3626,
profiling (2.268 ms) : 2248, 2289
.   : milestone, 2268,
tracing (1.85 ms) : 1834, 1865
.   : milestone, 1850,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.23 ms [1.218 ms, 1.241 ms]	-
iast	3.246 ms [3.199 ms, 3.293 ms]	2.016 ms (164.0%)
iast_FULL	6.143 ms [6.08 ms, 6.206 ms]	4.913 ms (399.6%)
iast_GLOBAL	3.726 ms [3.66 ms, 3.791 ms]	2.496 ms (203.0%)
profiling	2.555 ms [2.529 ms, 2.581 ms]	1.325 ms (107.8%)
tracing	1.927 ms [1.91 ms, 1.945 ms]	697.9 µs (56.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.25 ms [1.237 ms, 1.262 ms]	-
iast	3.332 ms [3.285 ms, 3.379 ms]	2.082 ms (166.6%)
iast_FULL	6.038 ms [5.977 ms, 6.1 ms]	4.788 ms (383.1%)
iast_GLOBAL	3.626 ms [3.568 ms, 3.685 ms]	2.377 ms (190.2%)
profiling	2.268 ms [2.248 ms, 2.289 ms]	1.018 ms (81.5%)
tracing	1.85 ms [1.834 ms, 1.865 ms]	599.735 µs (48.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.44 ms) : 19241, 19638
.   : milestone, 19440,
appsec (18.536 ms) : 18349, 18723
.   : milestone, 18536,
code_origins (18.744 ms) : 18558, 18931
.   : milestone, 18744,
iast (17.754 ms) : 17577, 17930
.   : milestone, 17754,
profiling (18.605 ms) : 18421, 18789
.   : milestone, 18605,
tracing (18.002 ms) : 17824, 18179
.   : milestone, 18002,
section candidate
no_agent (19.384 ms) : 19187, 19580
.   : milestone, 19384,
appsec (18.566 ms) : 18375, 18756
.   : milestone, 18566,
code_origins (17.734 ms) : 17560, 17909
.   : milestone, 17734,
iast (18.653 ms) : 18466, 18839
.   : milestone, 18653,
profiling (18.697 ms) : 18511, 18884
.   : milestone, 18697,
tracing (17.778 ms) : 17600, 17955
.   : milestone, 17778,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.44 ms [19.241 ms, 19.638 ms]	-
appsec	18.536 ms [18.349 ms, 18.723 ms]	-903.353 µs (-4.6%)
code_origins	18.744 ms [18.558 ms, 18.931 ms]	-695.099 µs (-3.6%)
iast	17.754 ms [17.577 ms, 17.93 ms]	-1.686 ms (-8.7%)
profiling	18.605 ms [18.421 ms, 18.789 ms]	-834.354 µs (-4.3%)
tracing	18.002 ms [17.824 ms, 18.179 ms]	-1.438 ms (-7.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.384 ms [19.187 ms, 19.58 ms]	-
appsec	18.566 ms [18.375 ms, 18.756 ms]	-818.3 µs (-4.2%)
code_origins	17.734 ms [17.56 ms, 17.909 ms]	-1.65 ms (-8.5%)
iast	18.653 ms [18.466 ms, 18.839 ms]	-731.369 µs (-3.8%)
profiling	18.697 ms [18.511 ms, 18.884 ms]	-686.788 µs (-3.5%)
tracing	17.778 ms [17.6 ms, 17.955 ms]	-1.606 ms (-8.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775497568	1775503219
git_commit_sha	`b664cc0`	`64af647`
release_version	1.61.0-SNAPSHOT~b664cc0f2a	1.61.0-SNAPSHOT~64af64756a

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1775505346	1775505346
ci_job_id	1570881601	1570881601
ci_pipeline_id	106256212	106256212
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-p8ynegt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-p8ynegt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.025 s) : 15025000, 15025000
.   : milestone, 15025000,
appsec (14.732 s) : 14732000, 14732000
.   : milestone, 14732000,
iast (18.687 s) : 18687000, 18687000
.   : milestone, 18687000,
iast_GLOBAL (18.101 s) : 18101000, 18101000
.   : milestone, 18101000,
profiling (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
tracing (14.772 s) : 14772000, 14772000
.   : milestone, 14772000,
section candidate
no_agent (15.394 s) : 15394000, 15394000
.   : milestone, 15394000,
appsec (14.795 s) : 14795000, 14795000
.   : milestone, 14795000,
iast (18.162 s) : 18162000, 18162000
.   : milestone, 18162000,
iast_GLOBAL (17.82 s) : 17820000, 17820000
.   : milestone, 17820000,
profiling (15.319 s) : 15319000, 15319000
.   : milestone, 15319000,
tracing (14.693 s) : 14693000, 14693000
.   : milestone, 14693000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.025 s [15.025 s, 15.025 s]	-
appsec	14.732 s [14.732 s, 14.732 s]	-293.0 ms (-2.0%)
iast	18.687 s [18.687 s, 18.687 s]	3.662 s (24.4%)
iast_GLOBAL	18.101 s [18.101 s, 18.101 s]	3.076 s (20.5%)
profiling	15.068 s [15.068 s, 15.068 s]	43.0 ms (0.3%)
tracing	14.772 s [14.772 s, 14.772 s]	-253.0 ms (-1.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.394 s [15.394 s, 15.394 s]	-
appsec	14.795 s [14.795 s, 14.795 s]	-599.0 ms (-3.9%)
iast	18.162 s [18.162 s, 18.162 s]	2.768 s (18.0%)
iast_GLOBAL	17.82 s [17.82 s, 17.82 s]	2.426 s (15.8%)
profiling	15.319 s [15.319 s, 15.319 s]	-75.0 ms (-0.5%)
tracing	14.693 s [14.693 s, 14.693 s]	-701.0 ms (-4.6%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.487 ms) : 1476, 1499
.   : milestone, 1487,
appsec (3.744 ms) : 3527, 3962
.   : milestone, 3744,
iast (2.269 ms) : 2200, 2338
.   : milestone, 2269,
iast_GLOBAL (2.313 ms) : 2244, 2382
.   : milestone, 2313,
profiling (2.528 ms) : 2356, 2701
.   : milestone, 2528,
tracing (2.082 ms) : 2028, 2135
.   : milestone, 2082,
section candidate
no_agent (1.498 ms) : 1486, 1509
.   : milestone, 1498,
appsec (3.841 ms) : 3618, 4064
.   : milestone, 3841,
iast (2.278 ms) : 2209, 2347
.   : milestone, 2278,
iast_GLOBAL (2.314 ms) : 2245, 2383
.   : milestone, 2314,
profiling (2.121 ms) : 2065, 2178
.   : milestone, 2121,
tracing (2.07 ms) : 2017, 2123
.   : milestone, 2070,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.487 ms [1.476 ms, 1.499 ms]	-
appsec	3.744 ms [3.527 ms, 3.962 ms]	2.257 ms (151.8%)
iast	2.269 ms [2.2 ms, 2.338 ms]	781.58 µs (52.6%)
iast_GLOBAL	2.313 ms [2.244 ms, 2.382 ms]	825.854 µs (55.5%)
profiling	2.528 ms [2.356 ms, 2.701 ms]	1.041 ms (70.0%)
tracing	2.082 ms [2.028 ms, 2.135 ms]	594.442 µs (40.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.498 ms [1.486 ms, 1.509 ms]	-
appsec	3.841 ms [3.618 ms, 4.064 ms]	2.344 ms (156.5%)
iast	2.278 ms [2.209 ms, 2.347 ms]	780.776 µs (52.1%)
iast_GLOBAL	2.314 ms [2.245 ms, 2.383 ms]	816.225 µs (54.5%)
profiling	2.121 ms [2.065 ms, 2.178 ms]	623.839 µs (41.7%)
tracing	2.07 ms [2.017 ms, 2.123 ms]	572.405 µs (38.2%)

mhlidd added 14 commits April 1, 2026 15:38

updating config helper to have descriptive exception in strict mode

4c8153c

enabling ConfigHelper strict mode by default for spock tests

f29cdf5

updating test-utils to include config-utils dependency at runtime

1a30830

overriding strictness for tests using fake env vars

da699c2

test adding real env vars

650e7d5

updating tests using fake env vars and adding more env vars

14e0405

overriding test mode for elasticsearch and opensearch

b998bec

adding final integration configs and overriding fake config tests

3e2ccfe

updating strictness in configurePreAgent

92e6d26

update config helper to guarantee throw an error that won't be caught

3582e00

configuring more fake config tests

e7b23e9

adding final configurations

b618154

Merge branch 'master' into mhlidd/run_spock_with_strict

84dc914

revert supported-configurations

9d52790

mhlidd changed the base branch from master to mhlidd/update_strictness_policy April 2, 2026 21:09

mhlidd changed the base branch from mhlidd/update_strictness_policy to master April 2, 2026 21:10

mhlidd mentioned this pull request Apr 5, 2026

Update supported-configurations.json w/ Missing Configs #11046

Merged

mhlidd added type: enhancement Enhancements and improvements comp: testing Testing comp: config Configuration labels Apr 6, 2026

cleanup

b0147c9

mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 5ba6f69 to b0147c9 Compare April 6, 2026 17:00

removing additional draining from InstrumentationSpecification

82a89e0

mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 95b7e3f to 82a89e0 Compare April 6, 2026 18:20

Merge branch 'master' into mhlidd/run_spock_with_strict

64af647

mhlidd marked this pull request as ready for review April 6, 2026 20:51

mhlidd requested review from a team as code owners April 6, 2026 20:51

mhlidd requested review from claponcet and manuel-alvarez-alvarez April 6, 2026 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Spock Tests to run with `STRICT_TEST` mode by default#11032

Update Spock Tests to run with `STRICT_TEST` mode by default#11032
mhlidd wants to merge 17 commits intomasterfrom
mhlidd/run_spock_with_strict

mhlidd commented Apr 1, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Apr 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mhlidd commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mhlidd commented Apr 1, 2026 •

edited

Loading

pr-commenter bot commented Apr 5, 2026 •

edited

Loading