Blanch the shallots and the orange and lemon zests for 2 minutes in boiling water. Drain in a strainer. Combine with the jelly, port, orange juice, vinegar, mustard, and ginger in a small saucepan. Bring the sauce to a slow simmer and simmer long enough to cook the alcohol off the port and to dissolve the jelly. Serve cold.