How many people did Avraham take along for the first world war

In this weeks parsha, Lech Lecha, see the following:

14. And Abram heard that his kinsman had been taken captive, and he armed his trained men, those born in his house, three hundred and eighteen, and he pursued [them] until Dan.

יד. וַיִּשְׁמַע אַבְרָם כִּי נִשְׁבָּה אָחִיו וַיָּרֶק אֶת חֲנִיכָיו יְלִידֵי בֵיתוֹ שְׁמֹנָה עָשָׂר וּשְׁלֹשׁ מֵאוֹת וַיִּרְדֹּף עַד דָּן:

As you can see, the passuk states that Avraham took along 318 people to war.
Now lets see Rashi.

three hundred and eighteen: Our Sages said (Gen. Rabbah 43:2, Ned. 32a): It was Eliezer alone, and it [the number 318] is the numerical value of his name.

שמונה עשר ושלש מאות: רבותינו אמרו אליעזר לבדו היה, והוא מנין גימטריא של שמו:

Rashi says it was only Eliezer.

Sifsai Chachamim asks, that regardless, "ain mikrah yotzei midai pshuto". And he answers that indeed there were 318 people originally but they did not go along for the war since were excluded after the selection for war, as shown here:

5. And the officers shall speak to the people, saying, What man is there who has built a new house and has not [yet] inaugurated it? Let him go and return to his house, lest he die in the war, and another man inaugurate it.		ה. וְדִבְּרוּ הַשֹּׁטְרִים אֶל הָעָם לֵאמֹר מִי הָאִישׁ אֲשֶׁר בָּנָה בַיִת חָדָשׁ וְלֹא חֲנָכוֹ יֵלֵךְ וְיָשֹׁב לְבֵיתוֹ פֶּן יָמוּת בַּמִּלְחָמָה וְאִישׁ אַחֵר יַחְנְכֶנּוּ:
6. And what man is there who has planted a vineyard, and has not [yet] redeemed it? Let him go and return to his house, lest he die in the war, and another man redeem it.		ו. וּמִי הָאִישׁ אֲשֶׁר נָטַע כֶּרֶם וְלֹא חִלְּלוֹ יֵלֵךְ וְיָשֹׁב לְבֵיתוֹ פֶּן יָמוּת בַּמִּלְחָמָה וְאִישׁ אַחֵר יְחַלְּלֶנּוּ:
7. And what man is there who has betrothed a woman and has not [yet] taken her? Let him go and return to his house, lest he die in the war, and another man take her."		ז. וּמִי הָאִישׁ אֲשֶׁר אֵרַשׂ אִשָּׁה וְלֹא לְקָחָהּ יֵלֵךְ וְיָשֹׁב לְבֵיתוֹ פֶּן יָמוּת בַּמִּלְחָמָה וְאִישׁ אַחֵר יִקָּחֶנָּה:
8. And the officers shall continue to speak to the people and say, "What man is there who is fearful and fainthearted? Let him go and return to his house, that he should not cause the heart of his brothers to melt, as his heart."		ח. וְיָסְפוּ הַשֹּׁטְרִים לְדַבֵּר אֶל הָעָם וְאָמְרוּ מִי הָאִישׁ הַיָּרֵא וְרַךְ הַלֵּבָב יֵלֵךְ וְיָשֹׁב לְבֵיתוֹ וְלֹא יִמַּס אֶת לְבַב אֶחָיו כִּלְבָבוֹ:

Now, I have some issues with this pshat. One, as Josh toys with in this recent post, Rashi writes חֲנִיכָיו, without the yud. Which would make into singular form. According to this pshat, Rashi shouldn't have done this, since at that point there were still all 318 of the men ready for war.

Secondly, the next passuk clear states וַיֵּחָלֵק עֲלֵיהֶם לַיְלָה הוּא וַעֲבָדָיו, with וַעֲבָדָיו clearly meaning servants in the plural form:

15. And he divided himself against them at night, he and his servants, and smote them, and pursued them until Hobah, which is to the left of Damascus.

טו. וַיֵּחָלֵק עֲלֵיהֶם לַיְלָה הוּא וַעֲבָדָיו וַיַּכֵּם וַיִּרְדְּפֵם עַד חוֹבָה אֲשֶׁר מִשְּׂמֹאל לְדַמָּשֶׂק:

Rashi, citing a gemara, did not take issue with literal pshat here, it seem. But its this peirush I don't understand.
Thanks to Josh for the sources, and welcome to my new blog.