From one phone photo to a shareable 3D model
How I built ObjectShare3D: Next.js, FastAPI, YOLO11, SAM 2, TRELLIS.2 and Three.js, stitched into one pipeline.
Take a photo of an object on your phone. A few minutes later you can orbit it, zoom into it, spin it with your hand in front of the webcam, and send a link to someone else. That is ObjectShare3D.
The idea is simple to state. The implementation is mostly about deciding where one problem ends and the next begins. This post walks through the architecture, each pipeline stage, and the decisions I would defend or revisit.
Pipeline
- 01Upload
FastAPIsize + format check - 02Detect
YOLO11labelled boxes, under 1 s - 03Segment
SAM 2clean cut-out - 04Reconstruct
TRELLIS.2about 3 min per model - 05Clean up
Trimeshcentre, scale, write .glb - 06View + share
Three.jsorbit, gestures, share link
Each stage hands a smaller, cleaner problem to the next one.
The shape of the system
Three processes do the work:
- A Next.js 15 + React frontend for capture, object picking and the 3D viewer.
- A FastAPI backend that owns the pipeline and the REST API.
- A TRELLIS.2 subprocess in its own Python environment.
Architecture
- capture and object picking
- 3D viewer (Three.js)
- hand gestures (MediaPipe, on device)
- proxies API calls
- REST API
- orchestrator
- detect, segment
- reconstruct, texture
- mesh cleanup
- own Python environment
- own dependencies
- SQLite + local files
- Postgres / S3 supported
The proxy means a phone on the LAN needs no CORS setup. TRELLIS.2 is isolated because its dependencies differ from the backend's.
Two choices here are worth explaining.
FastAPI, because the ML code is Python. The detection, segmentation and mesh code all live in Python, so the API server sits right next to them. It is async, so uploads and status polling stay responsive while a long job runs.
Next.js proxies the API calls. The browser only ever talks to the frontend's origin, so a phone on the same LAN can open the app and upload a photo with no CORS configuration at all. It sounds minor, but "open it on my phone and it just works" was the demo I wanted, and this removed the most annoying setup step.
Storage is SQLite plus local files by default, with Postgres and S3 supported when it needs to grow. The backend is organised by pipeline stage (detect, segment, reconstruct, texture, mesh cleanup), with an orchestrator that runs them in order.
Stage 1: upload and validation
The server rejects bad input early: a maximum of 15 MB and a minimum of 256 px. It then stores the image and creates a database record. Failing here is cheap. Failing after three minutes of GPU time is not, so every cheap check goes first.
Stage 2: detect (YOLO11)
The photo might contain a desk, a mug, a keyboard and a plant, and the user has to say which one they want. YOLO11 (yolo11n, the nano model) returns labelled boxes in well under a second, and the user picks one.
I chose the smallest model on purpose. This step only needs a rough box, since the next stage does the precise work. If detection is unavailable, the pipeline falls back to treating the whole image as the object, so the app degrades instead of stopping.
Stage 3: segment (SAM 2)
SAM 2 (sam2-hiera-large) takes the chosen box as a prompt and returns a mask, which becomes a transparent cut-out of the object.
This is the stage I would call most underrated. Single-image 3D models reconstruct whatever they are shown, and leftover background pixels do not disappear: they become geometry. A clean cut-out matters more than almost any later tuning, which is why I use the large SAM 2 model here and the tiny model one step earlier. Spend the accuracy where it pays off.
Stage 4: reconstruct, and why there is a chain of engines
This is the heavy stage: a textured model takes about 3 minutes. The reconstruction code does not hard-code one model. It tries engines in order and moves on if one fails.
Fallback chain
- 1 · DefaultTRELLIS.2shape + colour togetherbaked UV texture
- 2 · FallbackHunyuan3D-2shape onlycolour added by my own texturing step
- 3 · FallbackTripoSRanother single-photo modellast-resort fallback
- 4 · Safety netFlat carda flat placeholderstill returns something
If an engine fails, the next one takes over. A slower, lower-quality answer beats an error screen.
A single configuration setting forces a specific engine, which is what I use when comparing outputs side by side.
Why a subprocess for TRELLIS.2? Its dependencies differ from the backend's. Rather than fight version conflicts in one environment, it runs in its own environment as a separate process. The cost is a process boundary to manage. The benefit is that upgrading one side cannot silently break the other.
The colour problem
The reason TRELLIS.2 became the default is texture, and the comparison explains it better than a screenshot would.
Why the colours look different
- Generate shape onlygeometry with no colour at all
- Project the photo from the frontthe visible side gets real pixels
- Fill hidden areasspread nearby colour across the gaps
- Result: can look blotchythe limit is the projection, not the shape
- Generate shape and colour jointlythe model sees both at once
- Bake a real UV texturestitching, tread and prints stay sharp
- Export a textured meshnot decimated, so the texture survives
- Result: crisp detailthe back of the object is still a guess
Hunyuan3D-2 only generates shape. Colour comes from my own texturing step, which projects the photo onto the mesh from the front and fills the hidden areas by spreading nearby colour. That is why its output can look blotchy: the limit is the projection step, not the shape model. A quality switch picks between the full model (the default) and a faster but rougher turbo mode.
TRELLIS.2 generates shape and colour together and bakes a real UV texture. Stitching, tyre tread and printed patterns come out sharp, because the model never had to guess colours onto a finished mesh.
My takeaway: if two stages are bolted together after the fact, you can see the join. Models that treat geometry and appearance as one problem avoid it.
Stage 5: clean up (Trimesh)
Raw generated meshes are not ready to show. The cleanup stage removes bad faces, centres the model, scales its longest side to 1 unit, and writes a .glb. GLB is the standard web 3D format, so the viewer needs no conversion.
One detail worth knowing: textured meshes are deliberately not decimated, because decimation would drop the texture.
Stage 6: the viewer
The viewer runs on Three.js, with orbit, zoom and fullscreen. Optional hand-gesture control uses MediaPipe hand tracking, which runs on the device. A public share link serves the model to anyone.
Lighting took the most care, because generated models look wrong in surprising ways:
- Studio environment lighting plus lights attached to the camera, so the side facing you is never dark while you orbit.
- ACES tone mapping at exposure 1.3, for brightness without blown highlights.
- Forced matte materials (non-metallic, roughness of at least 0.75). Generated PBR maps often claim a surface is metal, and a metal material mirrors the environment as white streaks. Overriding it trades a little realism for a result that looks right every time.
What it cannot do
Single-photo reconstruction has hard limits, and the app should not pretend otherwise:
- The back of the object is a guess. The model never saw it.
- Glare in the photo can show up in the colours.
- Input matters. A sharp, evenly lit, front-three-quarter photo on a plain background gives the best results.
Before deploying anywhere
Licences need checking. Hunyuan3D-2 is marked non-commercial and excludes the EU, UK and South Korea. I have not yet checked TRELLIS.2's licence. For a personal project that is fine; for a product it is the first thing to settle.
What I would do next
These are ideas, not shipped work:
- Multi-photo input, to replace the guessed back side with real observations.
- A licence-clean engine path for commercial use.
- Progress feedback during the 3-minute reconstruction, so the wait feels shorter than it is.
Takeaways
- Give each stage one job, and make every stage hand the next a smaller, cleaner problem.
- Put cheap checks first and expensive GPU work last.
- Plan the failure path. A rough answer beats an error screen.
- Isolate conflicting dependencies behind a process boundary.
- The viewer is part of the product. A good model with bad lighting still looks broken.
If you are working on image-to-3D, I would like to hear what has worked for you, especially on texture quality and reflective objects.