Fish do not care how much time you spent in decorating their tank. Small fish will constantly move gravel or small stones in their tanks while larger fish can move surprisingly large rocks.
Fish move gravel or rocks that will fit in their mouths in order to check for food or algae growing on the rocks, to create a spawning nest or by accidentally swimming into it.
Mouthing gravel or small rocks and spitting them back out is normal fish behaviour. Many fish species, like goldfish, also chase each other as part of spawning behaviour.
Gravel should be smooth and not have any paint or toxic dyes on it, just in case the fish swallows it. Aquarium rocks should not have sharp edges in case the fish knocks into it.
According to FirstTankGuide.net, large aggressive fish, like Oscars, will intentionally knock over rocks and move gravel into piles, perhaps to try and create their own hiding places.
According to "Freshwater Aquarium Problem Solver" (David Boruchowitz; 2006), if fish are constantly darting around the tank as if in a panic, knocking over rocks and are not spawning, this may be a sign that the tank water is too hot.