The Whispers is the name given to a ruined castle in the Crownlands, the region controlled directly by the Iron Throne in King's Landing. The castle is located north-east of King's Landing at the very tip of Crackclaw Point. The Whispers are named for the eerie sound the sea makes as it crashes into the holes in the cliffs below the castle.
In the books
In the A Song of Ice and Fire novels, the Whispers was the former stronghold of House Crabb. Ser Clarence Crabb is said to have beheaded every enemy he slew in combat and brought them back home, where his wife kissed them, restoring them to life. Crabb kept the living heads - including an alleged King of Duskendale - in the castle, seeking their counsel. The Whispers is named in legend for the sound of the heads' whispering their advice to the knight.
In reality, the castle is most likely named for the sound of the water pouring through holes in the cliff below the castle. Coastal erosion has seen the castle's beacon tower collapse into the sea within the last two generations.
The lords of Crackclaw Point - including those of the Whispers - were a quarrelsome lot, refusing to accept the authority of any of the Andal kings. During the War of Conquest, Visenya Targaryen flew to the area on her dragon Vhagar and won the allegiance of the people, promising they would hold fealty to the Iron Throne alone.
In A Feast for Crows, Brienne and Pod arrive at the Whispers while searching for Sansa. They encounter three former members of the Brave Companions - Timeon, Pyg and Shagwell - who kill their guide Dick Crabb. Brienne kills all three thugs.