Baltham Greyman

I finally got going on these quests a bit. Usually I get the first
summons when I am right in the middle of a Main Quest job and by the
time I get finished, it's too late to pick up Baltham Greyman. This
time I have managed to get as for as the Kingman Castle quest. I got
the map and the key and was supposed to meet Baltham in 18 days.
Well, I got the key and got back to the meeting place in 5 days and
Baltham wasn't there. I have the follower icon and have had it since
he asked me to get the key for him. Do I have to hang around for 13
more days, or can I just to to Kingman Castle, as I seem to have
Baltham with me already?

I do realize that the name of the dungeon is probably different
depending on which province you do this quest in. It is the quest
where you have to steal the map from the Mage's Guild, then find the
key (disguised as a Holy Dagger) in a dungeon, then meet up with
Baltham again.
--
Nyctolops