We propose a projector-based interaction method and system with fusion of virtuality and reality, supporting participants to interact with real objects like cards physically on the desk or table. An infrared camera is mounted on the micro-projector to detect the hidden markers embedded on cards, while the projector is used to project the generated virtual objects on the surface of cards. When participants move or rotate the tangible cards, the velocity of markers is calculated from the pose change of cards. Especially both the pose and the velocity are used to control the virtual characters which enables participants to interact with virtual objects realistically with the help of physics simulation. Meanwhile, by tracking the multiple cards and orientation projecting, we design a configurable dynamic storyboard system. A variety of examples of interactive scenarios are provided, which verify our method’s validity and extensibility.