Using a swivel motion with a peeler, as described for peeling tomatoes, peel the yellow zest of one of the lemons in long strips. Juice the lemon and set the juice aside in a small bowl. Put the lemon zest and milk in a medium saucepan and place over medium heat. When the milk releases steam when stirred, but before it comes to a boil, remove the pan from the heat.
Put the egg yolks with both sugars in the bowl of an electric mixer. Whisk at high speed until the mixture is smooth, pale yellow, and forms ribbons, 2 to 3 minutes. Add the flour and mix it in on low speed.
Remove the lemon zest from the milk and transfer the milk to a pitcher. With the mixer on low speed, slowly pour in the milk. Once all the milk has been added, return the mixture to the saucepan and place over medium-low heat. Cook, whisking constantly, until the custard thickens, about 10 minutes. Do not be discouraged if nothing seems to happen for a while. When the foam on the surface begins to subside, the custard will start to thicken. Once the custard is done, remove the pan from the heat.
Add the cherry brandy, rum, and water to the reserved lemon juice.
Line the bottom of a trifle dish, approximately 10x7inches (or the equivalent), with half of the ladyfingers. Using a pastry brush, dab the ladyfingers with half of the liqueur mixture. Pour half of the custard over them, then top with the remaining ladyfingers. Dab the ladyfingers with the remaining liqueur mixture, then pour the rest of the custard on top. Grate the zest of the remaining lemon and sprinkle it on top. Cover and refrigerate overnight before serving.
If desired, this dessert can be assembled in individual goblets instead.