Begin by adding the milk, sugar, cornstarch, and unsweetened cocoa powder to a medium saucepot. If your cocoa powder is really lumpy add it to a sifter with the cornstarch first and sift it into the pot with the milk and sugar. Use a whisk to whisk everything together.
Set the burner to medium heat and whisk and stir the mixture as it heats up. Stir occasionally until it gets nice and thick and resembles thick chocolate milk. It will take around 10 to 15 minutes.
Remove it from the heat and add in the butter and vanilla extract. Stir until the butter melts completely. Allow it to cool for just a few minutes.
Pour carefully or use a ladle or spoon, and add the fudgesicle mixture to the individual popsicle molds. They will expand when they freeze so don't fill it up to the very top leave about a quarter-inch from the top.
Place the fudge pops into the freezer and allow them to freeze for at least 4 hours. When you want to eat them, run the popsicle mold under some warm water which will help loosen up the fudgesicle. Enjoy!