Canvas has a great feature that lets you "favorite" or star your main Canvas courses. This will reduce the amount of courses that show up in the Dashboard/Courses tab. That way you can "clean up" your Dashboard and only see the courses you know you will need access to regularly.
All you need to do is click on the "Courses" button at the left side of the screen and scroll all the way down to the "All courses" option.
That will take you to a page that lists every single active Canvas course that you are assigned to. Whatever courses that are marked with a orange star (or "favorited") are the ONLY ones that will show up on the Dashboard / Courses tab.
If no courses are starred/favorited, Canvas will display ALL of your courses. Make sure to star all the courses you will need as once you star ONE course, only starred/favorited courses will be displayed.
You can always go back to this "All courses" page if you need to access your non-favorited courses.