Is it possible for something to be immoral if it harms no one?

I know that there are different opinions on what can be consider harmful but... should something be considered immoral if it does not directly harm anyone? Is harm a good measuring stick to determine how immoral an act is?