Place the chai tea bag into a mason jar. In a small saucepan over medium-high heat, warm the whole milk, stirring constantly, until it begins to boil. Pour the boiling milk over the chai tea bag in the mason jar and let it steep for 20 minutes before removing the tea bag. Allow the mixture to cool completely to room temperature before moving on.
Preheat the oven to 350°F. Prepare a 9.5-inch (12 Cup) bundt pan by spraying the sides with baking spray (Baker's Joy is my favorite). Alternatively you can grease and flour the pan.
Add the sifted cake flour, baking powder, baking soda, 4 teaspoons of the chai spice mix, and salt into a bowl and whisk to combine. Set aside.
In the bowl of your stand mixer (a handheld mixer works fine too!), beat the butter on high for 2 minutes until it's creamy. Add in the sugar and continue to beat on medium-high for another 2 minutes, scraping down the bowl and paddle as needed. Add the eggs one at a time, then add the vanilla and sour cream and mix for 1 minute on high, scraping down the bowl and paddle once more.
With the mixer on low speed, add in the dry ingredients and mix until just combined. Add the (room temperature!) chai milk mixture slowly and mix until just incorporated. Scrape down the sides and bottom of the bowl and give it a few stirs to make sure there are no lumps (without over-mixing). The batter will be slightly thick.
Pour batter into prepared cake pan (no more than 2/3 full) and bake for 30-35 minutes. It's done when the cake springs back to the touch and a skewer inserted into the middle comes out clean. Let the bundt cake cool in the pan for 10 minutes before turning it out onto a wire rack for an additional few hours of cooling. Make sure it's entirely room temperature before applying the glaze.