I know sea prism stones are extremely hard and can't remember any moment where I have seen anyone to cut or break them. If someone is cuffed with them and he or she happens to be extremely powerful and not a devil fruit eater can he or she break it? Can a swordsman cut it?
So far in the series, we have not had any instances where seastone was directly broken apart, whether with brute force or with a weapon like a sword. In cases where characters like Robin or Ace were restrained with seastone handcuffs, they were unlocked using the keys. Whether it can be done remains an open question, as we have seen both Zoro and Mihawk cut through other extremely solid objects in the past.
However, the mineral itself is not indestructible even though it is said to be as hard as a diamond. This is evident because the Marines are able to shape it into a variety of items like handcuffs, prison bars, weapons (like Smoker's sword) and even use it to coat the bottom of ships to escape Sea Kings. You can find the other limitations we know about seastone in the wikia.
In the latest manga chapters, we come to know that seastone may have originated
in Wano, and that there exist craftsmen with expertise in working with the mineral.