AI-Powered 3D Asset Management with Neural Search

Imagine that as a video game designer,  you want to use a 3D dragon model in your game. Before, you could do this in two ways:

1. Make complex 3D models by yourself or take photos of dragon models from multiple angles.
2. Go to a 3D model repository to look for a dragon, but then you would have to search using keywords and hope the uploader had put it in their description.

This is fine if you're willing to look through hundreds, maybe thousands of 3D dragons. But if you want a specific kind of dragon? An Asian-style dragon instead of a Western? Something more snake-like? Or more dinosaur-like? You might spend hours hunting for good matches to what you have in mind.

Now, by using Asset Ovi, a 3D model repository based on Neural Search, you can just upload a 3D model to search for similar ones, or click on an existing model to find other models like it.

Despite the rapidly growing demand for 3D models from game designers, there is no mature solution for finding 3D assets stored in repositories. Most search engines are text-based: They can only match text queries with text descriptions and keyword lists. They say a picture is worth a thousand words, so a 3D model must be worth at least as many. Keyword lists and text descriptions are rarely more than a sentence or two. They could never capture all the information about a model that is relevant to a search.

Even the most experienced user can easily spend hours hunting in asset repositories, without ever finding what they need and without knowing if what they are looking for is or is not there.

To solve this problem, we have deployed Neural Search in Asset Ovi, using Jina AI's ground-breaking multimodal information retrieval framework to compare 3D models to other 3D models, without needing detailed descriptions or thousands of keywords.

Asset Ovi's Neural Search technology supports two search methods:

1. Upload model search. Users upload their own 3D models, from which we extract 3D mesh data and then use Jina's Neural Search technology to identify similar models based. The similarity is not based on gross shapes and colors but on AI-driven recognition of what kind of thing the mesh is modeling.
2. Search based on models already stored in Asset Ovi. This proceeds the same way that the user upload search does, except it receives a model ID from the user and works with the already stored model matching it.

Before integrating Jina's Neural Search solution, we spent a lot of time and labor manually tagging models and providing them with descriptions for our text-based search system. Jina cooperated with us in extending its already mature image search technology to 3D models.

The most critical challenge is to develop an efficient 3D model representation method and similarity matching algorithm to ensure the stable representation and classification of any 3D mode.

For this reason, Jina AI's team of engineers developed technology for encoding 3D models into vectors for high-level understanding.

The business process of model search can be roughly divided into the following processes.

1. Load 3D model: The 3D model is loaded (supported by GLB, GLTF, and other formats), and the point clouds are extracted and preprocessed.
2. Feature Extraction: To output a feature vector from 3D mesh data, we use the PointConv model.
3. Similarity retrieval: normalized feature vectors are indexed into the PostgreSQL database, and the HNSW algorithm is used to find high-dimensional vector similarity in a given dataset.

After building the whole process, we can use the tools provided by Jina to export the Kubernetes deployment file with one click and then deploy it directly on a cloud service or cluster.

YAHAHA currently has hundreds of thousands of creator users. To make it easier for users of our huge model repository to find exactly what they are looking for, we have made the following efforts.

1. 3D direct preview: Asset repositories like Unity Asset Store and CGTrader display objects in 2D, making it hard to see how they will render as 3D objects. Asset Ovi now gives users a real 3D preview, without compromising model creators’ intellectual property by transmitting the full mesh to the viewer’s browser.

1. Rendering effect: In order to ensure that what you see is what you get, we made sure that website rendering matched the client rendering as closely as possible, especially for Unity.
2. Neural Search: We have indexed millions of models. Thanks to Jina, we no longer have to manually tag all models. Our searches are performed in an AI-driven way that no longer needs keywords.

Neural Search empowers Asset Ovi users by giving them powerful search capabilities, greatly improving their efficiency and user experience in finding suitable models. In the future, we will also make Asset Ovi an affiliate program of the Unity Asset Store to help Unity developers efficiently find the relevant materials they need to create games on the Unity Asset Store. We will also cooperate with resource websites such as Sketchfab and CGTrader to help all users, developers, and artists alike, to find the resources they need more easily.