There are a number of transitive dependencies that are not explicitly listed in the go.mod file. Check Nakama’s vendor/modules.txt at the tag for your release for a full list of everything the server is built with.
You don’t need to match absolutely everything if it doesn’t appear at all in your plugin (or if your plugin uses dependencies Nakama does not) but where your plugin’s dependencies overlap with Nakama’s the versions must match exactly.