For your visit one day should be sufficient.
Distances are all walkable.
Worthy attractions: Cathedrale Sainte-Cecile; Musee Toulouse-Lautrec; Les Jardins de la Berbie; Cloitre de la Collegiale Saint Salvy; Pont Vieux.
Left for another visit: Fashion museum; Musee Laperouse; Academie des Miniatures.
Last visit: August 2017.

Albi town center is a UNESCO World Heritage site. While commercialized and not raising to the level of exceptional, it boasts two can’t miss attractions and a number of other things to see and do.
Things to See
Cathedrale Sainte-Cecile ♥♥♥ is simply incredible. Monumentally towering above the town, it can be seen from afar from every perspective. It is one of the largest brick buildings in the world. Every inch of the cathedral’s interior is painted with vivid frescoes, which has never been in need of restoration according to the thorough audio-guide.
Musee Toulouse-Lautrec ♥♥ is a must for any aficionado of the Impressionist art. It boasts a well structured and not overwhelming collection, which centers on the local boy but also includes works of other painters.
The art museum is located in the former archbishop’s Palace Berbie, but there is not much of palatial interior to view. However, Les Jardins de la Berbie ♥♥ are very beautiful and offer a couple of high-bank perspectives over the opposite side of town.
Pont Vieux ♥ provides looking-up perspectives to both sides of town.
In the comparatively busy town core, Cloisters of St-Salvy ♥ offer a bit of serenity.
Our itinerary left no time for visiting a number of smaller museums in Albi – they all appear as pleasant diversions according to TripAdvisor reviews. I especially regret not being able to see Musee Laperouse.