I found a bottle similar to this one at a local hardware store for about $6.00. I liked its flat shape because it could be carried in a haversack without too much bulk. Also, it doesn't use a screw top, so I'm already on third base, ready to score a homer!
I moved down to the Fasteners section of the store and found a suitable cork for $.50. So for less than $7.00, I had a usable canteen that could pass muster as a period correct accouterment. I suspect that a period correct bottle would have been tinted to protect its contents, but for $7.00, who cares?
When you get home, the first thing you need to do is remove the cage that provides the necessary seal. I did this with a pair of pliers, using less effort than I thought would be necessary. All that is left to betray its 21st century origin are the two little holes at the bottle's neck.
You can carry your new canteen in a haversack, but remember that if it leaks or the cork gets loose, the contents of the bag will be soaked. Store this bottle canteen away from your fire starting material and your foodstuffs. You can make a fitted leather bag from leather for next year. If I do make one, it will be a simple leather envelope with leather laces for closures. Paper pattern, cut leather, punch holes, lace to close. Easy Peasy.
Of course, you can always check the rubbish bin behind you local tavern to see what you can find.