Dumbing down a panel is simple. Use R7,6,5 G7,6,5 B7,5,4 and ground the rest of the inputs. Panels with 12 bits are old and hard to find.
We are using a 100mhz ARM. That is not important. There is a built in video controller that moves data from RAM to LCD with out CPU overhead. That is big.
You need 640 x 480 x 3 bytes of RAM. That about 1M byte. If you use 8 bit video that helps by 3:1.
To draw a red line from 10,10 to 40,40 you need to:
Make red 10,10 11,11 12,12 13,13 ……39,39 40,40.
To move the red line up one pixel you need to:
Clear the red line at location 10,10 to 40,40 made of 31 red pixels.
Draw a new line 10,11 11,12……39,40 40,41 made of 31 pixels.
At the lowest level you need to write to every pixel you want to change. This may involve erasing the old line before making the new line. To make pixel 10,10 red you need to write FF to 10,10,red 00 to 10,10,blue and 00 to 10,10,green. Using 24 bit video and a 8 bit CPU moving the line up requires erasing 31 x 3 bytes and drawing 31 x 3.
At the same time you need to keep the LCD refreshed. 25mhz video 24 bits wide. That means 75 mega bytes/second.
Is this a hobby or a work project?