I’m not expecting to see a ghost tonight, but I will go in search of one. I’ve always assumed that any self-respecting spectre would run (float?) a mile from the lurid carnival trappings of contemporary Halloween revelries. And yet I still entertain a faint hope that, after the sticky children have been and gone with their plastic pumpkins and benign extortion, and I settle into a favourite chair with a book of ghost stories, there might be some corresponding fluttering of the curtains, a tap at the window, a shadow in the doorway, a sudden unexpected chill.

I realise that in confessing my yearning for a haunting I risk my credentials as a good rationalist and proud patron of Humanists UK, but I suspect that this fascination with the idea of ghosts lies below the surface in all of us. I know several people – you probably do too – educated, sensible professionals, not given to histrionics, who quite calmly relate encounters with the unexplained that produce pleasing goosebumps. I want to believe them. As Dr Johnson observed about the existence of ghosts: “All argument is against it; but all belief is for it.”

In my case, ghosts retain the lure of the forbidden. Halloween was not acknowledged in our house when I was a child; I was raised in a style of evangelical Christianity that believed in the literal existence of demons and was consequently anxious to avoid anything that might encourage them. This included overt celebrations of the occult such as All Hallows’ Eve, which strikes me now as a failure to realise what the traditional church had understood for centuries: the importance of balancing light and dark, superstition and faith, in the imaginations of their congregation.

The result was that I grew up with a bizarre sense that the forces of darkness were real and alarming, ready to take any opportunity to slip a scaly foot in the door if you opened it even a crack by indulging in pop cultural representations.

However, ghost stories are hard to police. You can keep the likes of Stephen King out of the house all you want, but supernatural apparitions stalk the pages of Shakespeare, Dickens, Stevenson, the Romantics and the Victorian Gothic. By the time I arrived at university, I had loudly declared myself an atheist, which ought to have meant that I regarded superstition and faith with the same sceptical eye, but my love of ghostly literature survived this onslaught of reason.

Here, I discovered the stories of MR James, probably the best-known English writer in the genre after Dickens, whose enduring tales began life as fireside entertainments in his rooms in the college next door to mine and who evoked the uncanny melancholy of the East Anglian landscape in a way that appealed viscerally to my fascination with the past.

The best ghost stories are in large part an exercise in nostalgia – they recognise the ways in which places and buildings carry echoes of the lives that have played out there before us. “The English are… in many respects obsessed with the past, with ruins, with ancient volumes,” Peter Ackroyd writes in his book The English Ghost. “Ghosts therefore may be seen as a bridge of light between the past and the present, or between the living and the dead.” This phenomenon is not exclusive to the English tradition; Toni Morrison’s Beloved opens with one of the most disturbing depictions in literature of a house holding on to past trauma.

There’s a strong moral current running through the genre too. The restless spirit often heralds wrongs left unredressed, or is the direct result of the hapless protagonist’s meddling – either through greed or a desire for forbidden knowledge – in matters best left undisturbed. Many of James’s plots deal in the consequences of warnings that went unheeded, and in the stories of Susan Hill, the author of The Woman in Black and the most successful contemporary heir to the Victorian-Edwardian tradition, the ghosts are usually seeking a twisted form of reparation.

The “White Women”, whose legend appears across various cultures, are in many versions the revenant spirits of women abused, mistreated or violently killed, who return to exact revenge; it’s not hard to imagine that this interpretation was fashioned by women to reassure one another. My contribution to the genre, the suspense novel While You Sleep, is in part about the way women who choose to defy convention have always been regarded as unnatural and whether that has changed as much as we think since the 19th century.

Perhaps all this explains why, at a time when our relationship with our history grows increasingly fraught, and long-buried injustices are brought to light and re-examined, the ghost story is enjoying a revival. James remains an apparently inexhaustible source of inspiration, as does Shirley Jackson in the United States, but new voices are reinventing the genre’s scope. A new anthology, The Haunting Season, brings together some of best of these from the UK, including Laura Purcell, Andrew Michael Hurley and Kiran Millwood Hargrave, while writers such as Louise Doughty, Julie Myerson, Helen Oyeyemi and Sarah Waters have all used recent novels to reimagine the traditional ghost story in the light of broader social issues.

“The quintessential English ghost story is alarming but also oddly consoling,” Ackroyd writes. Perhaps this is because it allows us to confine our fears to the realm of the imagination. Ultimately, we read these books not for moral or political instruction but because we relish the pleasure of the shiver down the spine, whether or not we actually believe in anything beyond the grave. Obviously I don’t, not really. But I’ll keep an eye on the curtains tonight, just in case.