"do I have to distribute my work under the terms of the GPL even if I don't distribute any GPL code directly"
That is an interesting question that have a handful court case in the US, however the result has gone both way. The primary example of that has been cheat software that injects into copyrighted games, which both goes again the copyright and EULA of the game. In the example of blizzard, there is cases when they won against the cheat software on copyright grounds, as you can read here: (https://massivelyop.com/2020/02/28/lawful-neutral-cheating-c...).
Basically, the art of work that get created when you run software is also a kind of copyrighted work, and a bit like how movie character is copyrighted as well as an actually movie itself. If a second company make a new movie with the same characters, story and so on, even if its all "original" code, it is still a derivative work. The article above describe those as "non-literal elements", the graphics, sounds, animations and characters produced by the literal code.
So to answer the question "when is my program a derived work of a GPL work", it would seem to be when the non-literal elements of the GPL is changed in ways that require copyright permissions by the GPL author.
It should be noted however that in practically all cases involving the linux kernel, companies that distribute non-gpl kernel modules are also distributing the kernel as part of products.
That is an interesting question that have a handful court case in the US, however the result has gone both way. The primary example of that has been cheat software that injects into copyrighted games, which both goes again the copyright and EULA of the game. In the example of blizzard, there is cases when they won against the cheat software on copyright grounds, as you can read here: (https://massivelyop.com/2020/02/28/lawful-neutral-cheating-c...).
Basically, the art of work that get created when you run software is also a kind of copyrighted work, and a bit like how movie character is copyrighted as well as an actually movie itself. If a second company make a new movie with the same characters, story and so on, even if its all "original" code, it is still a derivative work. The article above describe those as "non-literal elements", the graphics, sounds, animations and characters produced by the literal code.
So to answer the question "when is my program a derived work of a GPL work", it would seem to be when the non-literal elements of the GPL is changed in ways that require copyright permissions by the GPL author.
It should be noted however that in practically all cases involving the linux kernel, companies that distribute non-gpl kernel modules are also distributing the kernel as part of products.