There are many factors leading to performance, or lack thereof. Number of rows is rarely the primary cause of poor performance.
Sorry, I need more than a sketch of the tables; please provide
* SHOW CREATE TABLE tbl\G -- engine, indexes
* SHOW TABLE STATUS LIKE 'tbl'\G -- sizes
* EXPLAIN SELECT ...\G -- clues of inefficiencies
* SHOW VARIABLES LIKE '%buffer%'; -- cache size
and ...



