You haven't really started fishing, until you've experienced fishing from a moving vessel. Whether that be a boat, a raft, or a kayak. Where I live, "yakangling" is huge, and everybody at least has some small boat to get out onto the local lake, "Pioneer". However, some people have preferences and different budgets, so you will see a wide range of kayaks and rafts out on the water. While you definitely need to do the leg work and read some fishing kayak reviews, here is my personal list of attributes to look for in a good fishing kayak.
This is numero uno when it comes to fishing from the water, as everything that you do is confined to a set space. Your also looking at significant comfort advantages by not being couped up, and trust me sitting in the same spot for hours is going to wreck havoc on you if your forced to lean.
The extra cockpit space is also important because you'll need to be able to cast, jig, and reel. If your doing this with fresh water fish, its mostly the jigging wher you need extra space. On ocean water however, when your fighting powerful fish, you'll need the extra room to reel. Check out the video attached at the bottom of this to see a kayaker tugged 10 MILES from shore by a Marlin.
Another important feature, you won't be able to do much without at least one hatch. The best angler kayaks have two hatches, but one will do as long as you have a dash to store items like hooks, cell phone, etc. Bungies to tie the boat up, and pull it on shore is also a must, and a spot for an anchor.
Go With a Sit on Top or a Sit In?
This is a major preference choice, and the design of your kayak and how you use it will entirely be dictated on what style of cockpit you go for. Sit on top kayaks have the advantage of stability in fresh water, comfort, and usually space. I've sat in some really nice SOT's in my life time, these are the vessels of choice if your just fishing in fresh water. Sit in kayaks, or SIK's, are designed with their seat inside the hollowed out cockpit(a sit on top has the seat above the water level). This gives you added leverage for maneuvering, speed, and in the case paddling in the ocean, stability. The problem is there is a lot less room and comfort, although the most expensive fishing kayaks have adjustable seating and space.
No matter what you decide on, these should be your three main concerns when you are looking around to purchase a kayak. Although I do think you need to find a good kayak fish finder, everything else is expensive fluff!