By itself, i.e. if you merely follow the specification, SemVer is quite useless. SemVer as it is practised is even more useless. Developers often break compatibility when they think they have a good enough reason, instead of incrementing the major version number, or they decide to increment the major version number with every release.
SemVer only becomes useful when done properly (according to the specification), while also following the spirit of SemVer.
The first thing that is missing from many projects that nominally implement semantic versioning is the specification of a public API. This is already part of the official SemVer spec, but people are not taking it seriously enough. Often it is not clear which parts of the API are public, and the API is under-specified. It can’t just be function signatures and types. If a function expects certain contracts to be upheld by the caller, that should be part of the API specification. If there is unspecified or undefined behaviour, this should be part of the official public API, otherwise pretty much any change to the implementation in a minor release could break existing code. All exceptions that might be thrown (even in future releases), error return codes that the caller must check, preconditions that must hold, or implementation details that are not guaranteed to stay this way (for example the ordered-ness of returned lists) have to be specified.
If you don‘t, people will complain that you changed the public API when you just fix a bug. It’s probably a good idea to make a policy about fixing bugs. Do you preserve bug-for-bug compatibility with the last release, or do you restore the behaviour promised by the spec?
Backward-compatible minor releases are useless if you increase the major version with every release. The whole point of SemVer is that minor releases are safe to upgrade to. An official roadmap or release schedule can tell downstream users what to expect from your project. Not every release is an LTS release, and not every bug fix will get backported, and maybe some major versions have no minor versions declared LTS releases at all. Still, if you don‘t release minor versions at all, or if you don‘t communicate when a new major version is around the corner, backward compatibility of minor versions is not all that useful.
In the same vein, your project should consider having release notes, changelogs, and migration guides. Even better than a compatible API for dependency resolution is an easy path to using the new API. Release notes can also point out changes in implementation or actual behaviour that are still conformant to the specified API.
Everything depends in one way or another on the CPU architecture, compiler version, and operating system. Platform support does not necessarily play nice with SemVer. Downstream users sometimes complain about dependencies that used to work no longer working, just because they switched from 32-bit OSX 10.5 on i386 to ARM64 mac OS 11.0. Unfortunately, there is not much you can do when Apple decides to stop shipping a sensible libGL. This is not conveyed by a version number, so you should probably mention it on the README.
You still need documentation for the internal API. You also need a separate section of your test suite for internal tests and a “public” test suite that only tests the specified behaviour of the public API and nothing more. Having your internals undocumented is worse in the long run than having no clearly defined public API.
If your upstream dependencies are not using SemVer, you can either “vendor in” code in your project’s source control, specify a known good range of versions, or make the build system download a tarball of a compatible release. If an upstream dependency uses SemVer correctly, you only have to specify the major version and a minimum minor version. Of course it can still make sense to re-write your code and bump requirements to a higher version. If a new release bumps the required versions of upstream libraries, well, that might affect downstream users too. Better document what you would do in this case!
You should probably give some more thought to versioning, build systems, and run-time concerns. Can different major versions of your library be installed on the same system? In different prefixes? Can the same program accidentally depend on two different major versions of your library?
If you do all this properly, downstream projects built against version 4.0 of your library can depend of version 4.X.X and still work. Well, that is until somebody upgrades their mac, or until somebody upgrades their debian to a version that runs wayland by default, but doesn’t have PyGame compiled with wayland support...
If you don’t, you can still say “We use SemVer“, and just bump the major version number with every release. It’s just not very useful.