I think each option has its pros and cons...
Being the “only UXer in the village” has the potential to give you loads of hands on experience in using all the tools you can think of as your manager is unlikely to know anything about UX and give you quite a free range in getting your job done. You will develop a wide array of skills, anything from graphic design to business analysis, on top of your original skill set. The downside is that you may feel that there are a million tools and techniques you know nothing about and you may feel compelled to invest lots of your own time in “keeping up”. Plus everyone in the organisation will think their X input is as valid as yours...
Working in a big UX department is quite different: you are surrounded by people who know exactly what you are talking about, have strong feelings on the same subjects you do and can often be quite inspiring. You may even find a mentor or two. The clients you work with will give you more of a free reign (within budget) as they regard the UX team as experts. But, you may end up being quite niche, with a lot of experience with a small number of UX tools and a belief that a wide range of specialists are needed for any task, however small.
It’s the old wide or deep knowledge debate. And, of course, neither is better than the other. The answer for me is the same in both cases:
- one is not better than the other, so don’t let that influence your job choice;
- wherever you work, network like crazy with other UXers to find out about their world, the tools they use and what inspires them;
- go to conferences and talks, even if you have to pay for them yourself
- and lastly, consider doing the opposite to what you did before when you look for a new job.