LLM not really read pictures in listmemory.

[meteorshowering]

Hello everyone.
I wrote a simple multimodal rag demo with listmemory. Although the agent seems to have the memory , it cannot actually read the images in memory.

And I'm wondering if it can memorize multimodal data for both images and text.
🙏 Would really appreciate your help.🙏

model_client = OpenAIChatCompletionClient(
    model="gpt-4o",  
)
memory = ListMemory(name="chat_history")
async def user_memory_add():
    local_image_path = "../allimages/Compass Towards Better Causal Analysis of Urban Time Series/figure_1.jpg"  # 本地图像路径
    img = AGImage.from_file(local_image_path)
    title = "figure_1:The interface of Compass. (a) The map view enables users to select a sensor to start analysis and reason causal graphs within a spatial context. (b) The time view presents the time series of the selected sensor and time partitioning results. (c) The graph view visualizes the dynamic causal graphs detected by a causal detection framework along the same timeline of the time view. (d) The relation view presents the causal relations involved in the causal graphs with a multi-dimensional visualization."
    content = MemoryContent(content=img, mime_type=MemoryMimeType.IMAGE, metadata={"usage":"picture","title": title})
    await memory.add(content)

    local_image_path = "../allimages/Compass Towards Better Causal Analysis of Urban Time Series/figure_2.jpg"  # 本地图像路径
    img = AGImage.from_file(local_image_path)
    title = "figure_2:Motivation illustration. (a) Real-world causal relations between sensors can change over time, shown by the different relations in three time windows. (b, c) Causal detection in large time windows may produce wrong or rough results. (d) Although the result is correct, further interpretation and verification are still required for informed policy making."
    content = MemoryContent(content=img, mime_type=MemoryMimeType.IMAGE, metadata={"usage":"picture","title": title})
    await memory.add(content)

picture_agent = AssistantAgent(
    name="picture_agent",
    model_client=model_client,
    description="Search and interpret images",
    system_message="You are a helpful assistant. You will receive many images, and you should view the images based on the user's questions and answer the user's questions.",
    reflect_on_tool_use=True,
    model_client_stream=True,  
    memory= [memory],
)

async def main() -> None:
    await user_memory_add()
    input1 = input("请输入您的问题：")
    await Console(picture_agent.run_stream(task=input1))
    await model_client.close()

asyncio.run(main())