Skip to content

Strictly check variant types in Flag Evaluations#11036

Open
typotter wants to merge 2 commits intomasterfrom
typo/evaluator-type-mismatch
Open

Strictly check variant types in Flag Evaluations#11036
typotter wants to merge 2 commits intomasterfrom
typo/evaluator-type-mismatch

Conversation

@typotter
Copy link
Copy Markdown
Contributor

@typotter typotter commented Apr 2, 2026 •
edited by atlassian bot
Loading

What Does This Do

Adds strict type checking to DDEvaluator so that requesting a flag evaluation with a type that doesn't match the flag's declared variationType returns TYPE_MISMATCH instead of silently coercing the value.

Also distinguishes variant value parse errors (broken config) from request type mismatches:

  • Request type vs flag variationType → TYPE_MISMATCH (e.g., STRING flag evaluated as Boolean)
  • Variant value vs declared type → PARSE_ERROR (e.g., INTEGER flag with string variant value)

Motivation

Aligns the Java SDK with Python (libdatadog) and Go (dd-trace-go#4590) type checking behavior. Previously, Java's mapValue() silently coerced types — Boolean.valueOf("on-value") returned false instead of erroring, and parseDouble("123").intValue() returned 123 instead of reporting a type mismatch.

Cross-tracer consistency is required for the FFE eval metrics system tests to pass (solves 2 of the 6 remaining test failures).

Type compatibility mapping:

variationType Valid Java type
BOOLEAN Boolean.class
STRING String.class
INTEGER Integer.class
NUMERIC Double.class
JSON Value.class

References:

Additional Notes

Contributor Checklist

  • Format the title according to the contribution guidelines
  • Assign the type: and (comp: or inst:) labels
  • Avoid using close, fix, or any linking keywords when referencing an issue

Jira ticket: FFL-1972

@typotter
fix(openfeature): strict type checking for flag evaluations
215d07a 
Check the requested evaluation type against the flag's declared
variationType before calling mapValue(). Return TYPE_MISMATCH when
they don't match (e.g., STRING flag evaluated as Boolean).

Separately, variant values that don't match their declared type
(e.g., INTEGER flag with string variant value) now return PARSE_ERROR
instead of TYPE_MISMATCH, aligning with Go SDK (dd-trace-go#4590)
and Python's libdatadog behavior.

Type compatibility mapping:
  BOOLEAN  → Boolean.class
  STRING   → String.class
  INTEGER  → Integer.class
  NUMERIC  → Double.class
  JSON     → Value.class
@typotter typotter added type: bug Bug report and fix tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Apr 2, 2026
@typotter typotter marked this pull request as ready for review April 2, 2026 05:18
@typotter typotter requested a review from a team as a code owner April 2, 2026 05:18
@typotter typotter requested review from greghuels and leoromanovsky and removed request for a team April 2, 2026 05:18
@PerfectSlayer
Copy link
Copy Markdown
Contributor

PerfectSlayer commented Apr 2, 2026

Hey @typotter 👋 Can you make sure the PR title follows the title format contribution guideline?
As the PR title goes straight into the release log for customers, please reword them accordingly :) Thanks!

@typotter typotter changed the title fix(openfeature): strict type checking for flag evaluations Strictly check variant types in Flag Evaluations Apr 2, 2026
@typotter
Copy link
Copy Markdown
Contributor Author

typotter commented Apr 2, 2026

Hey @typotter 👋 Can you make sure the PR title follows the title format contribution guideline? As the PR title goes straight into the release log for customers, please reword them accordingly :) Thanks!

done, thanks!

@typotter typotter mentioned this pull request Apr 2, 2026
Draft
3 tasks
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 8, 2026

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master typo/evaluator-type-mismatch
git_commit_date 1775589541 1775683574
git_commit_sha aa7c70f 4ceae78
release_version 1.61.0~aa7c70f2e7 1.61.0-SNAPSHOT~4ceae78daa
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775685607 1775685607
ci_job_id 1579524908 1579524908
ci_pipeline_id 106778762 106778762
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-cum7yh8d 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-cum7yh8d 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064580
Total [baseline] (8.846 s) : 0, 8846235
Agent [candidate] (1.057 s) : 0, 1056790
Total [candidate] (8.851 s) : 0, 8851098
section iast
Agent [baseline] (1.222 s) : 0, 1222281
Total [baseline] (9.567 s) : 0, 9566897
Agent [candidate] (1.221 s) : 0, 1221157
Total [candidate] (9.54 s) : 0, 9540033
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.065 s -
Agent iast 1.222 s 157.701 ms (14.8%)
Total tracing 8.846 s -
Total iast 9.567 s 720.662 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent iast 1.221 s 164.367 ms (15.6%)
Total tracing 8.851 s -
Total iast 9.54 s 688.935 ms (7.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.246 ms) : 0, 1246
crashtracking [candidate] (1.24 ms) : 0, 1240
BytebuddyAgent [baseline] (638.064 ms) : 0, 638064
BytebuddyAgent [candidate] (632.862 ms) : 0, 632862
AgentMeter [baseline] (29.58 ms) : 0, 29580
AgentMeter [candidate] (29.358 ms) : 0, 29358
GlobalTracer [baseline] (250.068 ms) : 0, 250068
GlobalTracer [candidate] (248.635 ms) : 0, 248635
AppSec [baseline] (32.283 ms) : 0, 32283
AppSec [candidate] (31.923 ms) : 0, 31923
Debugger [baseline] (59.508 ms) : 0, 59508
Debugger [candidate] (59.041 ms) : 0, 59041
Remote Config [baseline] (604.719 µs) : 0, 605
Remote Config [candidate] (614.898 µs) : 0, 615
Telemetry [baseline] (8.136 ms) : 0, 8136
Telemetry [candidate] (8.152 ms) : 0, 8152
Flare Poller [baseline] (8.899 ms) : 0, 8899
Flare Poller [candidate] (8.913 ms) : 0, 8913
section iast
crashtracking [baseline] (1.238 ms) : 0, 1238
crashtracking [candidate] (1.234 ms) : 0, 1234
BytebuddyAgent [baseline] (801.4 ms) : 0, 801400
BytebuddyAgent [candidate] (800.474 ms) : 0, 800474
AgentMeter [baseline] (11.358 ms) : 0, 11358
AgentMeter [candidate] (11.298 ms) : 0, 11298
GlobalTracer [baseline] (238.822 ms) : 0, 238822
GlobalTracer [candidate] (238.842 ms) : 0, 238842
AppSec [baseline] (30.083 ms) : 0, 30083
AppSec [candidate] (29.321 ms) : 0, 29321
Debugger [baseline] (61.294 ms) : 0, 61294
Debugger [candidate] (61.392 ms) : 0, 61392
Remote Config [baseline] (532.953 µs) : 0, 533
Remote Config [candidate] (534.957 µs) : 0, 535
Telemetry [baseline] (12.162 ms) : 0, 12162
Telemetry [candidate] (11.89 ms) : 0, 11890
Flare Poller [baseline] (3.449 ms) : 0, 3449
Flare Poller [candidate] (3.424 ms) : 0, 3424
IAST [baseline] (25.683 ms) : 0, 25683
IAST [candidate] (26.457 ms) : 0, 26457
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1057569
Total [baseline] (11.03 s) : 0, 11029530
Agent [candidate] (1.058 s) : 0, 1058341
Total [candidate] (11.11 s) : 0, 11110368
section appsec
Agent [baseline] (1.246 s) : 0, 1246137
Total [baseline] (11.154 s) : 0, 11154064
Agent [candidate] (1.25 s) : 0, 1249843
Total [candidate] (11.142 s) : 0, 11142269
section iast
Agent [baseline] (1.225 s) : 0, 1224666
Total [baseline] (11.238 s) : 0, 11237956
Agent [candidate] (1.244 s) : 0, 1244025
Total [candidate] (11.315 s) : 0, 11314640
section profiling
Agent [baseline] (1.184 s) : 0, 1183536
Total [baseline] (11.088 s) : 0, 11087821
Agent [candidate] (1.183 s) : 0, 1183259
Total [candidate] (11.031 s) : 0, 11030737
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent appsec 1.246 s 188.568 ms (17.8%)
Agent iast 1.225 s 167.097 ms (15.8%)
Agent profiling 1.184 s 125.967 ms (11.9%)
Total tracing 11.03 s -
Total appsec 11.154 s 124.534 ms (1.1%)
Total iast 11.238 s 208.426 ms (1.9%)
Total profiling 11.088 s 58.291 ms (0.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent appsec 1.25 s 191.502 ms (18.1%)
Agent iast 1.244 s 185.684 ms (17.5%)
Agent profiling 1.183 s 124.918 ms (11.8%)
Total tracing 11.11 s -
Total appsec 11.142 s 31.901 ms (0.3%)
Total iast 11.315 s 204.272 ms (1.8%)
Total profiling 11.031 s -79.631 ms (-0.7%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.249 ms) : 0, 1249
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (633.25 ms) : 0, 633250
BytebuddyAgent [candidate] (631.845 ms) : 0, 631845
AgentMeter [baseline] (29.335 ms) : 0, 29335
AgentMeter [candidate] (29.406 ms) : 0, 29406
GlobalTracer [baseline] (248.7 ms) : 0, 248700
GlobalTracer [candidate] (248.346 ms) : 0, 248346
AppSec [baseline] (32.028 ms) : 0, 32028
AppSec [candidate] (31.925 ms) : 0, 31925
Debugger [baseline] (59.853 ms) : 0, 59853
Debugger [candidate] (60.004 ms) : 0, 60004
Remote Config [baseline] (601.357 µs) : 0, 601
Remote Config [candidate] (665.686 µs) : 0, 666
Telemetry [baseline] (8.085 ms) : 0, 8085
Telemetry [candidate] (8.083 ms) : 0, 8083
Flare Poller [baseline] (8.336 ms) : 0, 8336
Flare Poller [candidate] (10.7 ms) : 0, 10700
section appsec
crashtracking [baseline] (1.243 ms) : 0, 1243
crashtracking [candidate] (1.246 ms) : 0, 1246
BytebuddyAgent [baseline] (661.016 ms) : 0, 661016
BytebuddyAgent [candidate] (662.875 ms) : 0, 662875
AgentMeter [baseline] (11.987 ms) : 0, 11987
AgentMeter [candidate] (12.024 ms) : 0, 12024
GlobalTracer [baseline] (248.103 ms) : 0, 248103
GlobalTracer [candidate] (249.091 ms) : 0, 249091
AppSec [baseline] (184.199 ms) : 0, 184199
AppSec [candidate] (184.768 ms) : 0, 184768
Debugger [baseline] (66.142 ms) : 0, 66142
Debugger [candidate] (66.28 ms) : 0, 66280
Remote Config [baseline] (610.114 µs) : 0, 610
Remote Config [candidate] (601.508 µs) : 0, 602
Telemetry [baseline] (8.467 ms) : 0, 8467
Telemetry [candidate] (8.463 ms) : 0, 8463
Flare Poller [baseline] (3.518 ms) : 0, 3518
Flare Poller [candidate] (3.558 ms) : 0, 3558
IAST [baseline] (24.586 ms) : 0, 24586
IAST [candidate] (24.547 ms) : 0, 24547
section iast
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.244 ms) : 0, 1244
BytebuddyAgent [baseline] (801.547 ms) : 0, 801547
BytebuddyAgent [candidate] (814.625 ms) : 0, 814625
AgentMeter [baseline] (11.417 ms) : 0, 11417
AgentMeter [candidate] (11.666 ms) : 0, 11666
GlobalTracer [baseline] (239.292 ms) : 0, 239292
GlobalTracer [candidate] (243.373 ms) : 0, 243373
AppSec [baseline] (30.229 ms) : 0, 30229
AppSec [candidate] (33.871 ms) : 0, 33871
Debugger [baseline] (61.337 ms) : 0, 61337
Debugger [candidate] (58.894 ms) : 0, 58894
Remote Config [baseline] (1.165 ms) : 0, 1165
Remote Config [candidate] (1.672 ms) : 0, 1672
Telemetry [baseline] (12.369 ms) : 0, 12369
Telemetry [candidate] (12.307 ms) : 0, 12307
Flare Poller [baseline] (3.375 ms) : 0, 3375
Flare Poller [candidate] (3.518 ms) : 0, 3518
IAST [baseline] (26.545 ms) : 0, 26545
IAST [candidate] (26.373 ms) : 0, 26373
section profiling
ProfilingAgent [baseline] (93.881 ms) : 0, 93881
ProfilingAgent [candidate] (93.829 ms) : 0, 93829
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.166 ms) : 0, 1166
BytebuddyAgent [baseline] (691.121 ms) : 0, 691121
BytebuddyAgent [candidate] (691.03 ms) : 0, 691030
AgentMeter [baseline] (9.154 ms) : 0, 9154
AgentMeter [candidate] (9.153 ms) : 0, 9153
GlobalTracer [baseline] (207.019 ms) : 0, 207019
GlobalTracer [candidate] (206.845 ms) : 0, 206845
AppSec [baseline] (32.334 ms) : 0, 32334
AppSec [candidate] (32.488 ms) : 0, 32488
Debugger [baseline] (65.498 ms) : 0, 65498
Debugger [candidate] (65.49 ms) : 0, 65490
Remote Config [baseline] (574.332 µs) : 0, 574
Remote Config [candidate] (592.679 µs) : 0, 593
Telemetry [baseline] (7.947 ms) : 0, 7947
Telemetry [candidate] (7.901 ms) : 0, 7901
Flare Poller [baseline] (3.589 ms) : 0, 3589
Flare Poller [candidate] (3.584 ms) : 0, 3584
Profiling [baseline] (94.446 ms) : 0, 94446
Profiling [candidate] (94.41 ms) : 0, 94410
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master typo/evaluator-type-mismatch
git_commit_date 1775589541 1775683574
git_commit_sha aa7c70f 4ceae78
release_version 1.61.0~aa7c70f2e7 1.61.0-SNAPSHOT~4ceae78daa
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775686073 1775686073
ci_job_id 1579524909 1579524909
ci_pipeline_id 106778762 106778762
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-ds8oqcma 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-ds8oqcma 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 18 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:petclinic:code_origins:high_load worse
[+1.661ms; +2.341ms] or [+9.489%; +13.375%]		 worse
[+1.456ms; +2.639ms] or [+5.047%; +9.146%]		 unstable
[-46.814op/s; +0.377op/s] or [-17.984%; +0.145%]		 19.502ms 30.898ms 237.094op/s 17.501ms 28.850ms 260.312op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.399 ms) : 19201, 19598
.   : milestone, 19399,
appsec (19.116 ms) : 18926, 19306
.   : milestone, 19116,
code_origins (17.924 ms) : 17745, 18103
.   : milestone, 17924,
iast (17.816 ms) : 17642, 17991
.   : milestone, 17816,
profiling (18.633 ms) : 18444, 18822
.   : milestone, 18633,
tracing (17.844 ms) : 17670, 18017
.   : milestone, 17844,
section candidate
no_agent (18.528 ms) : 18338, 18718
.   : milestone, 18528,
appsec (18.882 ms) : 18694, 19070
.   : milestone, 18882,
code_origins (19.689 ms) : 19495, 19882
.   : milestone, 19689,
iast (18.123 ms) : 17941, 18306
.   : milestone, 18123,
profiling (18.374 ms) : 18193, 18556
.   : milestone, 18374,
tracing (18.031 ms) : 17854, 18209
.   : milestone, 18031,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.399 ms [19.201 ms, 19.598 ms] -
appsec 19.116 ms [18.926 ms, 19.306 ms] -283.139 µs (-1.5%)
code_origins 17.924 ms [17.745 ms, 18.103 ms] -1.475 ms (-7.6%)
iast 17.816 ms [17.642 ms, 17.991 ms] -1.583 ms (-8.2%)
profiling 18.633 ms [18.444 ms, 18.822 ms] -766.435 µs (-4.0%)
tracing 17.844 ms [17.67 ms, 18.017 ms] -1.556 ms (-8.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.528 ms [18.338 ms, 18.718 ms] -
appsec 18.882 ms [18.694 ms, 19.07 ms] 353.925 µs (1.9%)
code_origins 19.689 ms [19.495 ms, 19.882 ms] 1.161 ms (6.3%)
iast 18.123 ms [17.941 ms, 18.306 ms] -404.679 µs (-2.2%)
profiling 18.374 ms [18.193 ms, 18.556 ms] -153.703 µs (-0.8%)
tracing 18.031 ms [17.854 ms, 18.209 ms] -496.721 µs (-2.7%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.268 ms) : 1255, 1281
.   : milestone, 1268,
iast (3.295 ms) : 3250, 3339
.   : milestone, 3295,
iast_FULL (6.17 ms) : 6107, 6233
.   : milestone, 6170,
iast_GLOBAL (3.772 ms) : 3708, 3837
.   : milestone, 3772,
profiling (2.372 ms) : 2349, 2395
.   : milestone, 2372,
tracing (1.872 ms) : 1856, 1888
.   : milestone, 1872,
section candidate
no_agent (1.254 ms) : 1242, 1266
.   : milestone, 1254,
iast (3.372 ms) : 3323, 3422
.   : milestone, 3372,
iast_FULL (6.133 ms) : 6070, 6196
.   : milestone, 6133,
iast_GLOBAL (3.753 ms) : 3693, 3813
.   : milestone, 3753,
profiling (2.221 ms) : 2201, 2242
.   : milestone, 2221,
tracing (1.941 ms) : 1924, 1958
.   : milestone, 1941,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.268 ms [1.255 ms, 1.281 ms] -
iast 3.295 ms [3.25 ms, 3.339 ms] 2.026 ms (159.8%)
iast_FULL 6.17 ms [6.107 ms, 6.233 ms] 4.902 ms (386.5%)
iast_GLOBAL 3.772 ms [3.708 ms, 3.837 ms] 2.504 ms (197.4%)
profiling 2.372 ms [2.349 ms, 2.395 ms] 1.104 ms (87.0%)
tracing 1.872 ms [1.856 ms, 1.888 ms] 603.732 µs (47.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.254 ms [1.242 ms, 1.266 ms] -
iast 3.372 ms [3.323 ms, 3.422 ms] 2.118 ms (168.9%)
iast_FULL 6.133 ms [6.07 ms, 6.196 ms] 4.879 ms (389.1%)
iast_GLOBAL 3.753 ms [3.693 ms, 3.813 ms] 2.499 ms (199.3%)
profiling 2.221 ms [2.201 ms, 2.242 ms] 967.314 µs (77.1%)
tracing 1.941 ms [1.924 ms, 1.958 ms] 686.814 µs (54.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master typo/evaluator-type-mismatch
git_commit_date 1775589541 1775683574
git_commit_sha aa7c70f 4ceae78
release_version 1.61.0~aa7c70f2e7 1.61.0-SNAPSHOT~4ceae78daa
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1775685818 1775685818
ci_job_id 1579524910 1579524910
ci_pipeline_id 106778762 106778762
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-v1ypu2c1 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-v1ypu2c1 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.465 s) : 15465000, 15465000
.   : milestone, 15465000,
appsec (14.918 s) : 14918000, 14918000
.   : milestone, 14918000,
iast (18.003 s) : 18003000, 18003000
.   : milestone, 18003000,
iast_GLOBAL (18.306 s) : 18306000, 18306000
.   : milestone, 18306000,
profiling (15.397 s) : 15397000, 15397000
.   : milestone, 15397000,
tracing (14.906 s) : 14906000, 14906000
.   : milestone, 14906000,
section candidate
no_agent (15.085 s) : 15085000, 15085000
.   : milestone, 15085000,
appsec (14.478 s) : 14478000, 14478000
.   : milestone, 14478000,
iast (18.485 s) : 18485000, 18485000
.   : milestone, 18485000,
iast_GLOBAL (17.916 s) : 17916000, 17916000
.   : milestone, 17916000,
profiling (15.06 s) : 15060000, 15060000
.   : milestone, 15060000,
tracing (14.904 s) : 14904000, 14904000
.   : milestone, 14904000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.465 s [15.465 s, 15.465 s] -
appsec 14.918 s [14.918 s, 14.918 s] -547.0 ms (-3.5%)
iast 18.003 s [18.003 s, 18.003 s] 2.538 s (16.4%)
iast_GLOBAL 18.306 s [18.306 s, 18.306 s] 2.841 s (18.4%)
profiling 15.397 s [15.397 s, 15.397 s] -68.0 ms (-0.4%)
tracing 14.906 s [14.906 s, 14.906 s] -559.0 ms (-3.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.085 s [15.085 s, 15.085 s] -
appsec 14.478 s [14.478 s, 14.478 s] -607.0 ms (-4.0%)
iast 18.485 s [18.485 s, 18.485 s] 3.4 s (22.5%)
iast_GLOBAL 17.916 s [17.916 s, 17.916 s] 2.831 s (18.8%)
profiling 15.06 s [15.06 s, 15.06 s] -25.0 ms (-0.2%)
tracing 14.904 s [14.904 s, 14.904 s] -181.0 ms (-1.2%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~4ceae78daa, baseline=1.61.0~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.485 ms) : 1474, 1497
.   : milestone, 1485,
appsec (2.526 ms) : 2472, 2581
.   : milestone, 2526,
iast (2.266 ms) : 2197, 2335
.   : milestone, 2266,
iast_GLOBAL (2.307 ms) : 2238, 2376
.   : milestone, 2307,
profiling (2.086 ms) : 2032, 2141
.   : milestone, 2086,
tracing (2.07 ms) : 2016, 2123
.   : milestone, 2070,
section candidate
no_agent (1.482 ms) : 1471, 1494
.   : milestone, 1482,
appsec (3.829 ms) : 3604, 4054
.   : milestone, 3829,
iast (2.264 ms) : 2195, 2333
.   : milestone, 2264,
iast_GLOBAL (2.318 ms) : 2249, 2388
.   : milestone, 2318,
profiling (2.531 ms) : 2363, 2698
.   : milestone, 2531,
tracing (2.069 ms) : 2016, 2123
.   : milestone, 2069,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.485 ms [1.474 ms, 1.497 ms] -
appsec 2.526 ms [2.472 ms, 2.581 ms] 1.041 ms (70.1%)
iast 2.266 ms [2.197 ms, 2.335 ms] 780.324 µs (52.5%)
iast_GLOBAL 2.307 ms [2.238 ms, 2.376 ms] 821.521 µs (55.3%)
profiling 2.086 ms [2.032 ms, 2.141 ms] 600.945 µs (40.5%)
tracing 2.07 ms [2.016 ms, 2.123 ms] 584.218 µs (39.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.482 ms [1.471 ms, 1.494 ms] -
appsec 3.829 ms [3.604 ms, 4.054 ms] 2.347 ms (158.4%)
iast 2.264 ms [2.195 ms, 2.333 ms] 782.227 µs (52.8%)
iast_GLOBAL 2.318 ms [2.249 ms, 2.388 ms] 836.075 µs (56.4%)
profiling 2.531 ms [2.363 ms, 2.698 ms] 1.049 ms (70.8%)
tracing 2.069 ms [2.016 ms, 2.123 ms] 587.408 µs (39.6%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greghuels greghuels greghuels approved these changes

@leoromanovsky leoromanovsky Awaiting requested review from leoromanovsky leoromanovsky is a code owner automatically assigned from DataDog/feature-flagging-and-experimentation-sdk

Assignees

No one assigned

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM type: bug Bug report and fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@typotter @PerfectSlayer @greghuels