Agentic Learner with Grow-and-Refine Multimodal Semantic Memory