Spray a 9x13-inch baking dish with nonstick cooking spray. Place the cubed challah bread into the prepared baking dish and spread it around into an even layer. Set aside.
In a very large mixing bowl, whisk together the milk, pumpkin puree, brown sugar, granulated sugar, eggs, vanilla, pumpkin pie spice, cinnamon, and nutmeg until fully combined.
Pour the mixture evenly over the bread in the prepared baking pan, making sure all of the bread is coated. Allow to sit and soak up the mixture for about 15 minutes.
Meanwhile, preheat the oven to 350°F (177°C). Once the 15 minutes is over, place the bread pudding in the oven and bake for 45 to 55 minutes or until the center is set.
Remove from the oven and transfer to a wire rack to cool slightly before serving.