Put the saffron, garlic, egg yolks and potato, if using, into a blender or food processor. Blend until the mixture is thick and smooth. With the motor running, slowly trickle in the olive oil until fully incorporated. Season well with salt and pepper.
Transfer the aïoli to a bowl or jar, cover and refrigerate. Use within 3 days.