Monitoring of Retrying Jobs?

[perryn]

Hello there,

I am a long time Sidekiq user who is considering using solid_queue instead.

One thing that I haven't seen much chat about is that solid_queue doesn't have the concept of 'the set of jobs that are currently retrying' ( ie: have failed at least once, have a retry scheduled (or currently running), but have not yet 'failed'.

This bothers me because in the past monitoring that set has been the most useful from an operational perspective.

Generally what I would do is the following

1. Gather metrics on the number of jobs in each state and generate alerts for spikes. So I was able to be alerted if there was a spike in the number of jobs retrying. (Alerting on a spike in the number of failed jobs would be too late)

2. Once I was alerted that there were more jobs retrying than usual, I would find the Retries Tab in Sidekiq very useful as I could see at a glance

   • Which Job types are failing
   • The retry count
   • what error was encountered to cause the retry

3. For further investigation I would look at our error tracking tool

However with solid_queue I can only monitor failed jobs (which is too late) or scheduled jobs ( which has actual scheduled jobs mixed in with retries, and no access to the last error)

I suspect that the reason behind this is that solid_queue relies on ActiveJob for retries whereas other backends have their own retry mechanism, but given that 'executions' and 'exception_executions' are available in the arguments it might be possible to add support for this.

Is anything like this planned?

In the meantime, I have been trying to do something for 1) above with a query like

SolidQueue::Job.where("scheduled_at > NOW() AND (arguments::jsonb->>'executions')::int > 1").count

But :

• It sometimes reports more than the number of retrying jobs that I can see in the Scheduled Jobs tab of Mission control and I'm not sure why
• I am unsure about the performance of such a query
• I am unsure about how 'future' proof such an approach might be

any advice?

[rosa]

Hey @perryn, thanks for trying out Solid Queue. You're completely right:

I suspect that the reason behind this is that solid_queue relies on ActiveJob for retries whereas other backends have their own retry mechanism

This is the reason, yes. This part is transparent to Solid Queue, as retries are handled by Active Job and Solid Queue simply sees them as jobs that get enqueued again. Solid Queue doesn't look into the serialized job arguments.

In the meantime, I have been trying to do something for 1) above with a query like

SolidQueue::Job.where("scheduled_at > NOW() AND (arguments::jsonb->>'executions')::int > 1").count

It sometimes reports more than the number of retrying jobs that I can see in the Scheduled Jobs tab of Mission control and > I'm not sure why

I think this might be just the time difference when you're looking at the query and at Mission Control, since usually the delay for retrying is very short.

I am unsure about the performance of such a query

Depending on how many jobs you have scheduled in the future at a given time, this query could be really slow, as there's no index in the job arguments.

I am unsure about how 'future' proof such an approach might be

I think it should be pretty much future-proof, as I don't think this part is going to change, either in Rails or in Solid Queue. Although you might want to look into using exception_executions as I'm not sure if Rails might finish deprecating executions, but that's a minor detail.

So, to sum up: we don't have plans to support this kind of monitoring in the future, as I hadn't even thought about it until today 😅 We'd need to record retries on the Solid Queue side as well as Active Job's serialised data, so it can be queried efficiently. I don't discard doing this in the future if more people show interest, though! But it won't be in the plans for version 2.0 (batch support and sharding are the two main features that I have in mind for that version).

I hope this answer your questions and thank you again!