Note that nowadays, you need not only a favicon.ico, but also a favicon.svg to have a favicon that renders on all devices.
And that Cassiopeia looks for pre-defined filenames to find them:

  • favicon.ico
  • joomla-favicon.svg
  • joomla-favicon-pinned.svg

To find the favicon, Cassiopeia looks first in templates/cassiopeia/images.
If it doesn't find a favicon there, it will default to media/system/images.

In the discussion about this, on the forum, there was a link to a Github issue, where DJGrammatiko said the following:
You shouldn't delete/rename files in the media/system folder, you're not the owner of those files!
If you want to override the favicons just copy the files [ favicon.ico, joomla-favicon.svg, joomla-favicon-pinned.svg ] to cassiopeia/images and edit them there. In this folder you are the owner of the files.

So I renamed my favicon.svg to joomla-favicon.svg and a copy of it to joomla-favicon-pinned.svg and moved them (and my favicon.ico) to template/cassiopeia/images.

The front-end favicon changed obligingly.