It looks like a dump just for a certain bottling plant or company. And it looks heavily dug already to me. We have a SS Coke bottling dump here in Gainesville Fla; ....most of the Gville bottles are broken; ...thats why they were dumped. You can find some "rogue" city bottles that were whole. Bottling companies often just tossed other bottling plant bottles.
Your well looks like it is free stacked bricks (no mortar) so it would be very early.
I would give it a good shot, but be very careful.
loose bricks falling and it could be capped and who knows how deep it could be.
Good luck let us know what you find.
I think erosion will help your digging too. To me, it does look like a well and most certainly they didn't throw goodies down there on purpose and spoil the water. However, some bottles may have washed in there over time, but I'd probe it first. That my help answer your question of whether to dig it or not. No, you're not nuts, just a helpless bottle addict like many of us. GL&HH.