Update 2: After one day the system has moved 1.2 GB of memory into the swapper:

This is suspicious. Memory usage is still at 2.5GB and the swapper should still be empty. I can think of two explanation:

1. Unused parts of programs like initialisation routines which are used once and then never again. That's not a problem. The unused memory stays in the swapper till next reboot and causes not further harm.
2. A memory leak: one of the programs doesn't free unused memory. Leaking 1.2GB of memory in 24h, would be a serious problem. A reboot every few days to free memory would be needed.

I'll have to keep a close look at this.