A framework for tangible user interfaces on handheld devices through the use of augmented reality, where virtual objects can be manipulated and arranged using physical objects in the real world. Visual feedback is provided in the high-resolution handheld display, where virtual objects are overlaid onto live video from the camera. The virtual objects are registered and tracked in real-time relative to the physical environment using the device's camera. Realistic lighting and rendering of high-resolution virtual models is achieved through hardware-accelerated graphics and shadow mapping. The user can interact with the virtual objects and system parameters both through an overlaid menu interface and through direct touch-screen interaction. We describe our framework, our adaptation of the ARToolKitPlus tracking library for a mobile platform, and a number of interaction techniques that we implemented for a prototype urban planning application.