Firefighters responded to a fast-moving brush fire near Castaic Lake on Saturday that had the potential for 500 acres, according to fire officials.