Small ants that are nested in the ground near the house foundation will often find these breather holes and the entire colony will move into the house. Once they are behind the brick, they can move freely in any direction. Nests are often made where the main floor joists meet the outer wall. Bats of insulation are placed between these joists and the ants will nest behind this insulation. They will then start to appear around sliding glass doors and baseboards eventually spreading throughout the house even on to upper floors.
This problem is sometimes caused by the installation of patios at the back or sides of the house. Interlocking stones, patio slabs or flag stone are often place over sand. This is a natural area for small ants to build nests. These patios are usually raised and often cover up the breather holes. Ants nesting in the sand against the foundation can enter the holes without even being seen from above.
Prevention of this problem involves inspecting the exterior periodically over the summer. Watch for ant activity is noted near the foundation or in the patio. If you are planning to build a patio against part of the house, then seal those breather holes that would be covered by the patio. Limestone screenings, which are less attractive to ants, can be used as a base for the patio instead of sand.