Actually a lot of things can go into putting a network together. There are level 1, Level 2 and Level 3 switches and their are managed switches. They have many more options than the 4 port router.
One option is virtual LANS which can keep traffic a little more segregated.
Since your talking about a network server, then the backbone might be better off being gigabit, but it depends on the intended traffic.
10/100 with a Gigabit trunk port is a good working standard when people browse the internet and connect to the backbone.
If your going to relay a lot on the server, then Gigabit ethernet for everyone would be the way to go.
Only if the 20 locations exist on different floors then it could make sense to have a switch for each floor.
High end switches and routers have the ability to have their uplink port be fiber.
Now, you didn't really say what the speed of your internet connection will be, nor how many IP addresses will be assigned by the ISP. That's where things start getting messy.
As an example, suppose you get two IP addresses from your ISP. A run of the mill router isn't going to help. One possible reason for that is to use one IP address for say an Http/mail server and have everyone else share.
"Server" is a loose term. Is this a file server, back up server, http server for the internet, mail server etc. Will it be a a fixed IP address from your ISP?
A lot of questions need to be asked.
With 20 ports, this appears to be a business and not a home network.
You also have to ask whether people are going to connect to the inside network from home, etc.
So, I have to ask: What are your needs?