To claim a system you must build a structure in the system. You can only build in a system that is not already claimed and you can only build a structure when you are in command of a construction role ship.
The easiest structure to build is the buoy structure, as it only requires the buoy prefab structure component to be in your hold to build.