The effects of adding nano-fillers, including core shell rubber (CSR), carbon nanotube (CNT) and the simultaneous addition of CSR and CNT on the curing behavior and material properties of the benzoxazine-epoxy-phenolic (BEP) ternary system have been investigated. The addition of CSR and CNT both decreased the activation energy of the BEP system, thereby making its polymerization faster. There was no obvious change observed with the decomposition temperature of the BEP based nanocomposites indicating that the nano-fillers had little negative effect on the thermostability of the nanocomposites. The glass transition temperature (Tg) of the filled-systems increased slightly with the addition of nano-fillers to the BEP system. The coefficients of thermal expansion (CTEs) of the three filled-systems were higher than that of the BEP system both below and above the Tg temperature range except for that of the BEP-CNT nanocomposite below Tg. The addition of CNT and CSR nano-fillers increased the toughness of the BEP system by 14% and 61%, respectively. Moreover, the CSR-CNT nanocomposite further improved the toughness and flexural strength of the BEP system by 160 and 30%, respectively.