How to reset Sidekiq failed tasks counter for Docker Mastodon

My Mastodon instance runs inside a Docker container, with its companion Sidekiq process. When I installed it, I messed up the network configuration, so the email sending tasks failed multiple times. After fixing the issue, I noticed that the Sidekiq control panel still read “Failed tasks: 38” and I really wanted to set this to zero (so I could more easily spot an issue in the future).

I found some info on StackOverflow, but it took some tinkering to make it work in my case (Mastodon-specific sidekiq instance running in a Docker container).

The first step was to open a shell in the context of the Docker container running Mastodon (note: I use podman, but you can substitute docker it that’s what you’re using):

$ podman exec -ti mastodon /bin/bash
root@c24be7da02be:/# irb
irb(main):001> require 'sidekiq/api'
<internal:/usr/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in
    `require': cannot load such file -- sidekiq/api (LoadError)

Dang. Apparently, the sidekiq API is not accessible from the main Ruby process. I don’t know much about Ruby, but I guess the modules paths are set by environment variables, so I could just takes the one set by the sidekiq process…

root@c24be7da02be:/# ps -ef | grep '[s]idekiq'
root        41     1  0 Nov07 ?        00:00:00 s6-supervise svc-sidekiq
abc        712    41  0 Nov08 ?        00:30:09 sidekiq 6.5.12 www [0 of 5 busy]

Okay, so sidekiq is PID 712. Since it’s running as user abc (UID 1000 according to the id command), let’s close this shell and open a new one using this user (and set HOME so bash does not complain):

$ podman exec -ti --user 1000 -e HOME=/tmp  mastodon /bin/bash
abc@c24be7da02be:/$

Now we can run irb with all the environment variables set by PID 712:
(Note: the following command will not work if any of the environment variables contain spaces; this is not the case here, fortunately)

abc@c24be7da02be:/$ env $(tr '\0' ' ' < /proc/712/environ) irb
/app/www/vendor/bundle/ruby/3.3.0/gems/irb-1.14.1/lib/irb/history.rb:51:in
    `stat': Permission denied @ rb_file_s_stat - /root/.irb_history
    (Errno::EACCES)

So close yet so far. After looking into it, it appears that despite running as user abc, process 712 has HOME=/root in its environment variables, so irb looks for its history file there. (I’m a bit surprised that “not being able to read the history file” is a fatal error for irb, by the way)

So let’s override HOME again:

abc@c24be7da02be:/$ env $(tr '\0' ' ' < /proc/712/environ) HOME=/tmp irb
irb(main):001> 

Finally! Let’s import the module and run the command now:

irb(main):001> require 'sidekiq/api'
=> true
irb(main):002> Sidekiq::Stats.new.reset('failed')
/app/www/vendor/bundle/ruby/3.3.0/gems/redis-4.8.1/lib/redis/client.rb:398:in
    `rescue in establish_connection': Error connecting to Redis on
    127.0.0.1:6379 (Errno::ECONNREFUSED) (Redis::CannotConnectError)

Okay, the Sidekiq API is trying to connect to the Redis databaselocalhost, but Redis actually runs on another Docker container (creatively named mast_redis in my case). I guess that when Mastodon starts sidekiq, it provides it with the Redis address in some way, and we need to emulate that.

After looking in lib/sidekiq.rb, I found a comment explaining how to configure the connection, so I tried it:

irb(main):001> require 'sidekiq/api'
=> true
irb(main):002* Sidekiq.configure_client do |config|
irb(main):003*   config.redis = { size: 1, url: 'redis://mast_redis:6379/0' }
irb(main):004> end
=> {:size=>1, :url=>"redis://mast_redis:6379/0"}
irb(main):005> Sidekiq::Stats.new.reset('failed')
=> "OK"

It worked! I now have zero failed sidekiq tasks. Let’s hope they add a reset button later…