It’s not like “even when you are not using” but when you are using app in the background. When you minimize app or turn off device screen then it is considered as “not being in (foreground) use” and GPS access is denied.
When you close/stop app then it won’t work/run and won’t access your GPS data.