feat: proper warn & error for exceeding max dataclip size by doc-han · Pull Request #4555 · OpenFn/lightning

doc-han · 2026-03-23T17:17:30Z

Description

Show the user a warning when dataclip size exceeds our max limit(currently 10mb). This max limit is set in liveview and hence had to be drilled down to react.

huge JSON in dataclip input	Error message on submit of huge JSON

Closes #4524

Validation steps

Get a very huge JSON(greater than 10mb)
copy to clipboard
paste into custom dataclip input box
It should show you a warning that you've exceeded the size limit.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

I have used Claude Code
I have used another model
I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

I have performed an AI review of my code (we recommend using /review
with Claude Code)
I have implemented and tested all related authorization policies.
(e.g., :owner, :admin, :editor, :viewer)
I have updated the changelog.
I have ticked a box in "AI usage" in this PR

codecov · 2026-03-23T17:25:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.49%. Comparing base (7a47ef2) to head (67d036a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4555      +/-   ##
==========================================
- Coverage   89.51%   89.49%   -0.02%     
==========================================
  Files         441      441              
  Lines       21205    21207       +2     
==========================================
- Hits        18982    18980       -2     
- Misses       2223     2227       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

josephjclark · 2026-03-24T08:51:40Z

assets/js/collaborative-editor/api/dataclips.ts

-    throw new Error(error.error || 'Failed to submit manual run');
+    if (response.status === 413) {
+      throw new Error(
+        'Dataclip is too large. Please reduce the size and try again.'


We shouldn't say "please" in an error message like this. We're not asking the user for a favour - we're telling them they've hit a wall.

@brandonjackson do you have opinions on tone and content of error messages? Want to weigh in here?

I agree with you @josephjclark - the key info they need is the file size limit - would rather that than pleasantries

any suggestions on what the wording here should be?
Dataclip is too large. Reduce the size and try again.

Hi @doc-han I think that wording works well. Have you already made this change to the error message? I can do a design review if you have.

updated @theroinaochieng. you can review

josephjclark

I haven't tested but the solution looks good to me. Thanks @doc-han!

…igger-a-run

lmac-1

Hiya @doc-han I just made one change to move max_dataclip_size_bytes into config. Also I left a comment about the toast message but it shouldn't block the merge. Apart from that looks good to me.

lmac-1 · 2026-03-26T09:16:14Z

assets/js/collaborative-editor/api/dataclips.ts

-    const error = (await response.json()) as { error?: string };
-    throw new Error(error.error || 'Failed to submit manual run');
+    if (response.status === 413) {
+      throw new Error('Dataclip is too large. Reduce the size and try again.');


Is it very difficult to add the file limit to the toast message from app config?

So the error message would be more like:
"Dataclip exceeds the 10 MB limit. Reduce the size and try again"

In case they missed it in the validation message?

lmac-1 · 2026-03-26T09:27:49Z

lib/lightning_web/channels/workflow_channel.ex

-      kafka_triggers_enabled: Lightning.Config.kafka_triggers_enabled?()
+      kafka_triggers_enabled: Lightning.Config.kafka_triggers_enabled?(),
+      max_dataclip_size_bytes:
+        Application.get_env(:lightning, :max_dataclip_size_bytes, 10_000_000)


I think this would be best if it was in lib/lightning/config, then we could test it more easily. The other values here use Lightning.Config.* functions so that we can let tests stub via Lightning.MockConfig. I'll do a commit as I'm in the code.

@doc-han I haven't re-tested things since doing this so maybe if you could just give it another test? Thanks

Great catch @lmac-1

midigofrank

Nicely done @doc-han . Please fix the conflict in the changelog

…igger-a-run

feat: proper warn & error for exceeding max dataclip size

0272cfb

github-project-automation bot added this to Core Mar 23, 2026

doc-han linked an issue Mar 23, 2026 that may be closed by this pull request

Huge custom dataclips will refuse to trigger a run #4524

Closed

github-project-automation bot moved this to New Issues in Core Mar 23, 2026

doc-han requested review from elias-ba, josephjclark and lmac-1 March 23, 2026 17:39

doc-han added 2 commits March 23, 2026 17:41

chore: update changelog

f3843d9

feat: disable run button when max size exceeded

eb18f84

josephjclark reviewed Mar 24, 2026

View reviewed changes

josephjclark approved these changes Mar 24, 2026

View reviewed changes

doc-han added 2 commits March 26, 2026 08:52

feat: update wording

ab89568

Merge branch 'main' into 4524-huge-custom-dataclips-will-refuse-to-tr…

ad0a0fa

…igger-a-run

doc-han requested a review from theroinaochieng March 26, 2026 08:59

move max_dataclip_size_bytes into Lightning.Config

5a3f67c

lmac-1 reviewed Mar 26, 2026

View reviewed changes

midigofrank approved these changes Mar 27, 2026

View reviewed changes

Merge branch 'main' into 4524-huge-custom-dataclips-will-refuse-to-tr…

67d036a

…igger-a-run

elias-ba approved these changes Mar 27, 2026

View reviewed changes

doc-han merged commit ddd3d00 into main Mar 30, 2026
7 of 8 checks passed

doc-han deleted the 4524-huge-custom-dataclips-will-refuse-to-trigger-a-run branch March 30, 2026 08:21

github-project-automation bot moved this from New Issues to Done in Core Mar 30, 2026

Conversation

doc-han commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation steps

AI Usage

Pre-submission checklist

Uh oh!

codecov bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josephjclark left a comment

Choose a reason for hiding this comment

Uh oh!

lmac-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

midigofrank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

doc-han commented Mar 23, 2026 •

edited

Loading

codecov bot commented Mar 23, 2026 •

edited

Loading