You may be required to attend an interview for obtaining the actual EB-5 visa (that is, after the I-526 petition is approved). Fluency is not required. When applying for citizenship, you will need to respond to the questions in English, and read and write a brief statement in English to prove proficiency (again, not fluency).
The ability to read, speak and understand English is not a requirement of the EB-5 program. Even at the time of the U.S. consulate interview for the conditional permanent residency you and your family members do not need to speak English. In fact at this interview you may bring in a translator.