Massive black holes may have started as little red dots