FINERACT-2516: Upgrade MariaDB support to version 12

[airajena]

FINERACT-2516

Summary

This PR upgrades MariaDB support from 11.x to 12.x across runtime, development, and test container references, and updates the MariaDB JDBC driver version.

Scope

• No functional business-logic changes
• No API contract changes
• No schema/Liquibase changes
• Version/reference upgrade only

Risk

Low.
Changes are limited to version references for MariaDB image/driver and associated docs/test setup.

[meonkeys]

I can review this tomorrow.

I did a mariadb version bump a while ago, see b053c90

[meonkeys]

Oh never mind, looks like @IOhacker is re-running the ci jobs... are you reviewing, @IOhacker ? If not, let me know, I'm happy to help.

[meonkeys]

I see MariaDB >= 11.5.2 in several places (try git grep '11\.5\.2'). I think just make all those 12.2 as well.

[meonkeys]

the diff looks fine to me except:

1. It looks like CI is still failing so far.
2. I think we should change everywhere 11.5.2 shows up in the docs to 12.2.

I also haven't run any tests locally with mariadb 12.2 yet. I'll do that now.

@airajena , are you able to identify, repro, and fix the failing tests?

[IOhacker]

@meonkeys always happy to have a hand during the review.

[IOhacker]

@airajena please do a rebase

[airajena]

I see MariaDB >= 11.5.2 in several places (try git grep '11\.5\.2'). I think just make all those 12.2 as well.

Yes working on it

[airajena]

the diff looks fine to me except:

1. It looks like CI is still failing so far.
2. I think we should change everywhere 11.5.2 shows up in the docs to 12.2.

I also haven't run any tests locally with mariadb 12.2 yet. I'll do that now.

@airajena , are you able to identify, repro, and fix the failing tests?

I checked the failing ci checks, they are not related to version change, if it was then it should show concrete SQL/driver error.

[meonkeys]

@airajena wrote:

I checked the failing ci checks, they are not related to version change, if it was then it should show concrete SQL/driver error.

Thanks for checking. I understand and agree.

We still need a passing build before merging.

I skimmed the build logs, but I don't understand the test failures and there are many. I'll try re-running all the failed tests.

Otherwise this PR looks good to me.

Did you figure out what was mucking with line endings for the .adoc files?

was addressed

[airajena]

@airajena wrote:

I checked the failing ci checks, they are not related to version change, if it was then it should show concrete SQL/driver error.

Thanks for checking. I understand and agree.

We still need a passing build before merging.

Otherwise this PR looks good to me.

Did you figure out what was mucking with line endings for the .adoc files?

Yes, repo has no .gitattributes rule enforcing .adoc line endings. git check-attr for those .adoc files returns no EOL attributes. Those files are expected as LF (git ls-files --eol shows i/lf). I have windows, so if any Windows tool saves with CRLF (for example a full-file rewrite API/editor default), Git sees the whole file as changed. That’s exactly the churn pattern we hit.

[meonkeys]

@airajena integration tests are still failing and I don't know why. Can you learn anything else from the logs? Can you repro locally?

I'd say try rebasing onto the tip of develop, but maybe that's a bad idea: Looks like the build on develop has been broken for a dozen commits now. I'll ping @adamsaghy in chat (#fineract on Mifos Slack).

[nidhiii128]

hii @airajena just a suggestion try running grep -r "mariadb" . , i found more files where changes are needed like

kubernetes/fineractmysql-deployment.yml
docker-compose-development.yml
fineract-doc/src/docs/en/chapters/testing/integration.adoc
fineract-doc/src/docs/en/chapters/testing/cucumber.adoc
.github/workflows/build-docker.yml

i hope it helps you

[airajena]

hii @airajena just a suggestion try running grep -r "mariadb" . , i found more files where changes are needed like

kubernetes/fineractmysql-deployment.yml docker-compose-development.yml fineract-doc/src/docs/en/chapters/testing/integration.adoc fineract-doc/src/docs/en/chapters/testing/cucumber.adoc .github/workflows/build-docker.yml

i hope it helps you

@nidhiii128 Thanks for sharing
In my PR
I have these file changes already
kubernetes/fineractmysql-deployment.yml
fineract-doc/src/docs/en/chapters/testing/integration.adoc
fineract-doc/src/docs/en/chapters/testing/cucumber.adoc

Regarding
.github/workflows/build-docker.yml
docker-compose-development.yml
In these two files, I am not seeing any version mentioned. I might be missing something, please let me know if I have missed anythin.

[airajena]

I've reproduced the CI failures locally after running the full integration test suite (approx. 3-hour runtime). The failures appear to be systemic rather than isolated. The primary bottleneck is in the Spring Batch / COB (Close of Business) jobs. Most tests are failing with exitCode=UNKNOWN, specifically throwing org.springframework.batch.core.JobExecutionException.
This seems to be causing a "domino effect" on downstream tests:
SavingsAccountsExternalIdTest: Resulting in 400 and 404 errors because the accounts aren't reaching the expected state due to the failed batch jobs.
General Assertions: Multiple tests are failing simply because they expect a COMPLETED status, but receive FAILED or UNKNOWN.
It looks like the root cause is tied to the batch execution logic rather than the API endpoints themselves.

[Saifulhuq01]

@airajena Good effort on the version bump, but changing the version string broke the Testcontainers initialization.

If you look at the CI logs, the Integration Tests aren't failing due to schema or collation issues; they are timing out (hanging for 20-27 minutes). The Java MariaDBContainer class relies on parsing the container's startup logs to know when it's ready. MariaDB 12 changed its startup log format, causing Testcontainers' default LogMessageWaitStrategy to wait infinitely.

We need to override the wait strategy in CommandBaseTest.java to use a standard TCP port wait strategy instead of the default log string matcher:

@container
private static final MariaDBContainer<?> MARIADB_CONTAINER = new MariaDBContainer<>(DockerImageName.parse("mariadb:12.2"))
.withNetwork(network).withUsername("root").withPassword("mifos").withDatabaseName("fineract-test")
.waitingFor(Wait.forListeningPort());

Apply that override and the Integration Test shards will stop hanging and go green.

[airajena]

cool, thanks for the help, done the following changes,

[Saifulhuq01]

cool, thanks for the help, done the following changes,

@airajena did you check your forked fineract version for cicd pass, check that and take action if any error.

[meonkeys]

Rebased on 5bb0b3b with fingers crossed. I just restarted the failing postgresql check from .github/workflows/build-docker.yml (looks like it timed out pulling a Docker image)

[meonkeys]

rebased on develop again. 🤷

[airajena]

rebased on develop again. 🤷

I investigated the MariaDB 12 CI failures after rebasing to latest develop.

The Access denied for user 'runner' line is not the root cause. It comes from the readiness probe using mysqladmin ping without credentials, but the workflow continues immediately into DB initialization, so that warning is just noise.

The actual failures happen later inside the integration tests, after MariaDB and Fineract are already up. The common error is:
Record has changed since last read
I found this in the failing shards for:

• FeignTrialBalanceSummaryReportTest.testExternalAssetOwnerEntriesAppearInReport
• ExternalAssetOwnerTransferCancelTest
• InitiateExternalAssetOwnerTransferTest.saleTransferWithSameOwnerExternalIdInParallelShouldNotFail
• SavingsAccountTransactionTest.testDeadlockSavingsBatchTransactions
• SavingsAccountTransactionTest.testConcurrentSavingsBatchTransactions

The affected tables are:

• m_external_asset_owner_transfer
• m_savings_account

This points to a MariaDB 12 transaction-behavior change, not a JDBC auth/handshake problem and not a Liquibase/Hibernate startup issue. The failure is concentrated in concurrency-sensitive write paths, and I was able to reproduce and validate locally against mariadb:12.2.

The fix was to explicitly disable innodb_snapshot_isolation for MariaDB 12 startup paths. After setting innodb_snapshot_isolation=OFF, the previously failing targeted tests passed locally on MariaDB 12.2.

I applied that consistently in:

• GitHub Actions MariaDB workflow
• local Docker compose
• Kubernetes MariaDB deployment
• MariaDB Testcontainers setup in CommandBaseTest

I also updated the readiness probe to use root credentials so the logs no longer show the misleading runner access-denied message.

So the issue was not the healthcheck warning; the real regression was MariaDB 12 snapshot isolation causing Record has changed since last read failures in Fineract’s concurrent integration test scenarios.

[airajena]

rebased on develop again. 🤷

I investigated the MariaDB 12 CI failures after rebasing to latest develop.

The Access denied for user 'runner' line is not the root cause. It comes from the readiness probe using mysqladmin ping without credentials, but the workflow continues immediately into DB initialization, so that warning is just noise.

The actual failures happen later inside the integration tests, after MariaDB and Fineract are already up. The common error is: Record has changed since last read I found this in the failing shards for:

• FeignTrialBalanceSummaryReportTest.testExternalAssetOwnerEntriesAppearInReport
• ExternalAssetOwnerTransferCancelTest
• InitiateExternalAssetOwnerTransferTest.saleTransferWithSameOwnerExternalIdInParallelShouldNotFail
• SavingsAccountTransactionTest.testDeadlockSavingsBatchTransactions
• SavingsAccountTransactionTest.testConcurrentSavingsBatchTransactions

The affected tables are:

• m_external_asset_owner_transfer
• m_savings_account

This points to a MariaDB 12 transaction-behavior change, not a JDBC auth/handshake problem and not a Liquibase/Hibernate startup issue. The failure is concentrated in concurrency-sensitive write paths, and I was able to reproduce and validate locally against mariadb:12.2.

The fix was to explicitly disable innodb_snapshot_isolation for MariaDB 12 startup paths. After setting innodb_snapshot_isolation=OFF, the previously failing targeted tests passed locally on MariaDB 12.2.

I applied that consistently in:

• GitHub Actions MariaDB workflow
• local Docker compose
• Kubernetes MariaDB deployment
• MariaDB Testcontainers setup in CommandBaseTest

I also updated the readiness probe to use root credentials so the logs no longer show the misleading runner access-denied message.

So the issue was not the healthcheck warning; the real regression was MariaDB 12 snapshot isolation causing Record has changed since last read failures in Fineract’s concurrent integration test scenarios.

cool integration tests are green now, the failing check is not code related. It is :rat rejecting mariadb_compat.cnf file with no approved Apache license header, adding and pushing the fix.

[meonkeys]

Woah, fascinating. Would we then be using innodb_snapshot_isolation=OFF during tests and also recommending it for prod/deployments as well? If this is a change we'll be making everywhere, please write the dev list and see if anyone has input/feedback/advice. I don't have experience with this database setting.

[airajena]

Woah, fascinating. Would we then be using innodb_snapshot_isolation=OFF during tests and also recommending it for prod/deployments as well? If this is a change we'll be making everywhere, please write the dev list and see if anyone has input/feedback/advice. I don't have experience with this database setting.

Actually for this PR, I treated it as a compatibility setting for the MariaDB 12 upgrade path, not as a claim that Fineract should globally prefer. I agree this deserves wider input if we intend to keep applying it outside CI/tests, I will explain this in detail in mailing list, let's see if we get a better way. There are really two possible directions.

1. Keep innodb_snapshot_isolation=OFF for MariaDB 12 so runtime behavior stays aligned with what Fineract has already been validated against.
2. Keep MariaDB 12 defaults and revisit the affected Fineract transaction/locking flows so they are explicitly correct under the new upgrade.

[IOhacker]

This feature is Ok to be turned off for this upgrade. For production, this variable turned off (which is the default setting in previous MariaDb versions) will introduce data I consistency. In Apache Fineract it has optimistic locking code and it must be upgraded. I think that a follow up Jira Ticket should be raised about turning on this variable that will help to fix the existing locking in Apache Fineract. Regards Victor El vie., 6 de marzo de 2026 12:05 p. m., Aira Jena ***@***.***> escribió:

…

*airajena* left a comment (apache/fineract#5566) <#5566 (comment)> Woah, fascinating. Would we then be using innodb_snapshot_isolation=OFF during tests and also recommending it for prod/deployments as well? If this is a change we'll be making everywhere, please write the dev list and see if anyone has input/feedback/advice. I don't have experience with this database setting. Actually for this PR, I treated it as a compatibility setting for the MariaDB 12 upgrade path, not as a claim that Fineract should globally prefer. I agree this deserves wider input if we intend to keep applying it outside CI/tests, I will explain this in detail in mailing list, let's see if we get a better way. There are really two possible directions. 1. Keep innodb_snapshot_isolation=OFF for MariaDB 12 so runtime behavior stays aligned with what Fineract has already been validated against. 2. Keep MariaDB 12 defaults and revisit the affected Fineract transaction/locking flows so they are explicitly correct under the new upgrade. — Reply to this email directly, view it on GitHub <#5566 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD2ZAQDLKJ3H2KRIHWGAC34PMHQPAVCNFSM6AAAAACWDIBVR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJTGIYDCOJSGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[airajena]

This feature is Ok to be turned off for this upgrade. For production, this variable turned off (which is the default setting in previous MariaDb versions) will introduce data I consistency. In Apache Fineract it has optimistic locking code and it must be upgraded. I think that a follow up Jira Ticket should be raised about turning on this variable that will help to fix the existing locking in Apache Fineract. Regards Victor El vie., 6 de marzo de 2026 12:05 p. m., Aira Jena @.> escribió:
…
airajena left a comment (apache/fineract#5566) <#5566 (comment)> Woah, fascinating. Would we then be using innodb_snapshot_isolation=OFF during tests and also recommending it for prod/deployments as well? If this is a change we'll be making everywhere, please write the dev list and see if anyone has input/feedback/advice. I don't have experience with this database setting. Actually for this PR, I treated it as a compatibility setting for the MariaDB 12 upgrade path, not as a claim that Fineract should globally prefer. I agree this deserves wider input if we intend to keep applying it outside CI/tests, I will explain this in detail in mailing list, let's see if we get a better way. There are really two possible directions. 1. Keep innodb_snapshot_isolation=OFF for MariaDB 12 so runtime behavior stays aligned with what Fineract has already been validated against. 2. Keep MariaDB 12 defaults and revisit the affected Fineract transaction/locking flows so they are explicitly correct under the new upgrade. — Reply to this email directly, view it on GitHub <#5566 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALD2ZAQDLKJ3H2KRIHWGAC34PMHQPAVCNFSM6AAAAACWDIBVR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJTGIYDCOJSGE . You are receiving this because you were mentioned.Message ID: @.>

@meonkeys @IOhacker hii, so should I open a discussion on mailing list or leave, now that we already have a direction to proceed

[IOhacker]

Open the Discussion, because it is a warning about turning off a very important feature that is enabled by default in the MariaDb version 12. Community must be aware of this, either for understanding why it is turned off for the upgrade and how it will be solve with an improved locking mode. El vie., 6 de marzo de 2026 12:22 p. m., Aira Jena ***@***.***> escribió:

…

*airajena* left a comment (apache/fineract#5566) <#5566 (comment)> This feature is Ok to be turned off for this upgrade. For production, this variable turned off (which is the default setting in previous MariaDb versions) will introduce data I consistency. In Apache Fineract it has optimistic locking code and it must be upgraded. I think that a follow up Jira Ticket should be raised about turning on this variable that will help to fix the existing locking in Apache Fineract. Regards Victor El vie., 6 de marzo de 2026 12:05 p. m., Aira Jena *@*. *> escribió: … <#m_-9025985592548107596_> airajena left a comment (apache/fineract#5566 <#5566>) <#5566 (comment) <#5566 (comment)>> Woah, fascinating. Would we then be using innodb_snapshot_isolation=OFF during tests and also recommending it for prod/deployments as well? If this is a change we'll be making everywhere, please write the dev list and see if anyone has input/feedback/advice. I don't have experience with this database setting. Actually for this PR, I treated it as a compatibility setting for the MariaDB 12 upgrade path, not as a claim that Fineract should globally prefer. I agree this deserves wider input if we intend to keep applying it outside CI/tests, I will explain this in detail in mailing list, let's see if we get a better way. There are really two possible directions. 1. Keep innodb_snapshot_isolation=OFF for MariaDB 12 so runtime behavior stays aligned with what Fineract has already been validated against. 2. Keep MariaDB 12 defaults and revisit the affected Fineract transaction/locking flows so they are explicitly correct under the new upgrade. — Reply to this email directly, view it on GitHub <#5566 (comment) <#5566 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALD2ZAQDLKJ3H2KRIHWGAC34PMHQPAVCNFSM6AAAAACWDIBVR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJTGIYDCOJSGE <https://github.com/notifications/unsubscribe-auth/ALD2ZAQDLKJ3H2KRIHWGAC34PMHQPAVCNFSM6AAAAACWDIBVR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJTGIYDCOJSGE> . You are receiving this because you were mentioned.Message ID: @.*> @meonkeys <https://github.com/meonkeys> @IOhacker <https://github.com/IOhacker> hii, so should I open a discussion on mailing list or leave, now that we already have a direction to proceed — Reply to this email directly, view it on GitHub <#5566 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD2ZAQBTDVDOZQIU23ALLL4PMJM3AVCNFSM6AAAAACWDIBVR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJTGI4DOOBRHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[IOhacker]

LGTM