Do you move your plate?

As a server, and also an observant customer, I see people moving their plates. The server puts them down, and the customer moves them because they don't like where the server places them. Why don't you just take the plates from the server instead of letting the server put them on your table? There are tons of etiquette for servers, but what about customers? Why are they permitted to rudely move their plates?