Colorectal cancer (CRC) is one of the most popular cancer in the world. Adenoma and sessile serrated polyp precursor lesions claim over 95% of CRC. The incidence of CRC is reduced 76-90% through the early diagnosis and removal of colorectal polyps. Colonoscopy is the golden standard for the detection of colorectal polyps but about 25% of polyps were missed during colonoscopy examinations. In this study, we proposed a novel method to recognize polyps from colonoscopy images based on bag-of-visual-words (BoW) with extracted regions of interest. The proposed method generates a histogram of visual word occurrences to represent an image, and uses support vector machine (SVM) with error correcting output codes (ECOC) for the detection of polyps. A dataset composed of 131 cases’ clinical data were used to train and test the proposed method. Validation demonstrates an average specificity of 97.8±1.5%, an average sensitivity of 97.2±1.7%, and an average accuracy of 97.5±1.0%.