ESPECIALLY with a .40, you need to be concerned with setback. It's a high-pressure round to start with...even a little setback can be dangerous.
If you're constantly loading/unloading, try this:
Box of carry ammo in the drawer. Load weapon. End of day, unload weapon, place ejected round in back of box. Next day, fresh round from box into magazine, load weapon...repeat. When you've repeated this process with all of your SD ammo (each round loaded once), tape the box and go buy more...start over...shoot the once loaded rounds at the range.
Now you know why I don't load/unload all that often with my .40s
Yes, it's wrong. Keeping mags loaded does not significantly wear the spring. 'Exercising' the spring as you do DOES wear the spring. Load them and leave them alone.