Apple released an Open Source multimodal LLM called Ferret in October 2023 in collaboration with researchers at Columbia University, but it didn't attract much attention at the time, as reported by IT House on December 25. As a result, many members of the AI community missed the launch of Ferret. Bart de Witte, who runs a European nonprofit focused on open source AI in medicine, recently posted on X: "I somehow missed this, Apple joined the Open Source AI community in October. The launch of Ferret is a testament to Apple's commitment to far-reaching AI research, cementing its position as a leader in multimodal AI... ps: I'm looking forward to the day when Local Large Language Models (LLLMs) will run on my iPhone as an integrated service for the redesigned iOS. Ferret is said to be able to examine the areas drawn on the image, identify the elements in it, and box it up. It can then take the identified elements as part of the query and respond in a typical manner. Ferret's release is significant for researchers, as it shows that Apple is gradually opening up its AI research, in stark contrast to its previously mysteriously closed image.

According to a report by Webmaster House on October 12, Apple's AI/ML team and Columbia University's research team developed a multimodal large model "Ferret" that can accurately find traffic lights in images, which performs better than GPT-4V and improves the accuracy of large models in "looking, speaking, answering" tasks. Ferret's key innovation lies in the close combination of spatial understanding of both referring and grounding, enabling the model to understand the semantics of a given region and find the corresponding target at the same time.