Preheat oven to 350ºF and prepare three 6-inch cake pans by spraying the sides with cooking spray and placing a pre-cut wax or parchment paper circle in the bottom. In a medium bowl, whisk together the flour, baking powder, baking soda, nutmeg, cinnamon, and salt and set aside.
Mash the bananas by hand or use a stand mixer with a paddle attachment to whip them until they’re nice and mashed (about 30 seconds on medium). Set them aside.
Using a stand mixer with the paddle attachment, whip the butter until it’s fluffy and creamy, about 2 minutes, scraping down the bowl and paddle halfway through. Add the white granulated sugar and the brown sugar and cream on medium for another 2-3 minutes. Scrape down the bowl and paddle, turn the mixer to low, and add the eggs and vanilla before beating on medium for one minute.
Add the mashed banana and beat on medium for a full minute before scraping down the bowl and paddle again. Turn the mixer to low and alternate adding the dry ingredients in three additions and buttermilk in two additions, starting and ending with the flour mixture. Mix each addition until just incorporated before alternating. As soon as the last addition of the flour mixture is incorporated, turn the mixer off and give it a few stirs by hand to make sure there’s no flour lurking at the bottom of the mixing bowl.
Fill the cake pans with batter no more than 2/3 full and bake for 45-55 minutes (baking times will vary). The cakes are done when they spring back to the touch and a toothpick inserted comes out clean. Cool the cakes in the pan for five minutes before turning out to cool on a baking sheet or wire rack for several hours.