Layer the ladyfingers across the bottom of a 3-quart (3-l) glass trifle bowl (or other large glass bowl). Drizzle the brandy, a spoonful at a time, evenly on the ladyfingers, and then spread a thin layer of the jelly (about ¼ cup/80 g) on top. Next, layer the fruit on top, followed by the macaroons. Drizzle the port, a spoonful at a time, evenly on the macaroons. Cover these first 4, booze-soaked layers with the rest of the jelly and set aside for an hour or two to steep.
Meanwhile, make a custard. Mix the powdered gelatin with 2 tablespoons water in a small bowl and let stand for 2 minutes to soften; if using the gelatin sheets, place them in a bowl, add cold water to cover, and let soak until floppy, 5–10 minutes. In a bowl, whisk the cornstarch into the milk until dissolved, then whisk in the egg yolks, 4 tablespoons (60 g) of the sugar, and the rose water until well mixed. Pour the mixture into a saucepan and heat gently over medium-low heat, stirring often, just until it starts to thicken. It should be the consistency of heavy cream. Add the softened gelatin. (If using powdered gelatin, first liquefy it by nesting the small bowl of gelatin in a larger bowl of hot water, or heating it in the microwave for 5 seconds.) Stir until the gelatin is dissolved. Do not allow the mixture to boil. Remove from the heat and stir in the ground almonds. Let cool completely before starting the next step.
You can prepare the recipe up to this point the day before you plan to serve the trifle and then finish the trifle the next day. Because you will refrigerate the custard overnight, you will need to melt it slightly before proceeding. You can do this by resting the bowl of custard in a bowl of hot water. It should be at room temperature and gloopy, rather than set, for the next step.
To finish assembling the trifle, spoon the custard on top of the booze-soaked layers, ensuring that it forms an even layer. In a bowl, using a whisk or a handheld mixer on medium speed, whip the cream with the remaining 2 tablespoons sugar until it is a good piping consistency. Transfer the whipped cream to a piping bag fitted with a star tip and pipe the cream attractively on top of the trifle. If not serving immediately, cover and chill thoroughly.
Just before serving, decorate the trifle with slivered almonds and crystallized fruit.