Background: Health-related quality of life (HRQoL) in community-dwelling older adults increases as physical activity improves, and age-related frailty has a negative effect on HRQoL. Research on these associations is lacking. Purpose: The aims of this study were to (a) analyze the effect of demographic characteristics on HRQoL, (b) explore the correlation between physical activity and HRQoL, (c) analyze the effect of frailty on HRQoL, and (d) investigate the potential predictors of HRQoL in community-dwelling older adults. Methods: In this cross-sectional study, a convenience sample of 150 older adults was recruited from community care sites in Shilin and Beitou Districts in Taipei City, Taiwan. Data were collected at baseline using a demographic characteristics datasheet, the Center for Epidemiologic Studies Depression Scale, the Physical Activity Scale for the Elderly, and the 12-Item Short Form Health Survey. The Senior Fitness Test and hand-grip strength test were also performed. Student t test, chi-square test, analysis of variance, Pearson correlation coefficient, and hierarchical regression were applied to analyze the statistical results using IBM SPSS Statistics Version 22.0. Results: Being of older age, experiencing a higher number of falls, having more chronic diseases, and having a higher body mass index were identified as factors that significantly affect HRQoL. Moreover, HRQoL was found to be significantly affected by the performance of physical activity or status of frailty. Furthermore, the prefrail period was shown to be an important predictor of HRQoL after adjusting for demographic variables, history of chronic illness, history of falls, and physical activity. Conclusions/Implications for Practice: In this study, HRQoL was found to be significantly affected by upper limb dysfunction and the prefrail period. Community health promotion activities should focus greater attention on the physical functioning of older adults. Furthermore, providing information on age-related frailty and promoting active participation in community activities may increase the attention given by community-dwelling older adults to physical fitness and quality of life.