Preheat your oven to 350°F (175°C).
Prepare the pie shell by placing it in your pie dish.
Cream together the softened butter, white sugar, and light brown sugar until light and fluffy.
Add the egg and egg yolks one at a time, whisking well after each addition, then stir in the vanilla extract.
Whisk together the flour, salt, and nutmeg, then gradually add to the wet ingredients, alternating with the milk.
Pour the filling into the prepared pie shell and smooth the top if needed.
Bake for 45-50 minutes until the edges are set and the center has a slight jiggle.
Cool for at least an hour before slicing and serve warm or at room temperature.