The purpose of this study was to construct a brain-computer-interface (BCI) based orthotic hand, including a prototype BCI system and a custom-made orthotic hand. The BCI can generate tri-state commands by processing a subject's mu wave while imaginary movement is performed, and the orthotic hand can be controlled to grasp, open and hold an object still. A BCI-based cursor control interface (visual feedback) with a simple classifier is designed to pre-train a naïve subject for performing imaginary movements. Then the classifier is adjusted to adapt to the subjects' EEG features for controlling the orthotic hand. The orthotic hand can be controlled under position and force modes, which are switched automatically by a logic controller depending on the commands and the current status of the orthotic hand. Three performance indices, namely, reaction time, information transfer rate and mean-squared error of force, were defined for evaluating the performance of the BCI-based orthosis under position and force control modes. The averaged success rates of three subjects were improved from 42% to 86% in 120 trials after pre-training. With the adaptive classifier, the averaged reaction time was reduced from 5 to 2 seconds under the position control mode, and the averaged information transfer rate was increased from 0.2 to 1.3 bit/trial. The mean-squared error of force was about 0.326 N while grasping an object under force control mode.