(Richard Norton-Taylor writes for the Guardian on defense and security).

At the turn of every year, hundreds of classified government documents are made available at the National Archives in Kew, south-west London. Many due for release are withheld or, as the Guardian has reported, have been lost. Whitehall departments do not have to explain why they have been retained, or where they are.

Documents held back this year include files relating to the Scott inquiry into the arms-to-Iraq affair, a file on allegations of sexual abuse at the Kincora boys’ home in Belfast which the former army information officer Colin Wallace said were covered up by MI5, and a file on the late Brian Nelson, a British army informer in Northern Ireland eventually jailed for conspiring to kill Catholics. Also withheld is a document on Anthony Blunt, surveyor of the Queen’s pictures, who was granted immunity from prosecution before confessing to having been a Soviet spy. Papers are retained, temporarily or indefinitely, while Whitehall weeders conduct what is called “sensitivity reviews.”

One file released now in an increasingly random process prompted a frisson of excitement in this writer. It refers to questions I asked many years ago about Margaret Thatcher’s suppression of two volumes in the official history of British intelligence in the second world war. One, called Strategic Deception, was written by the eminent historian Sir Michael Howard; the other, by the former deputy head of MI5, Anthony Simkins, was titled Security and Counter-Intelligence.

The file of highly classified papers opens in 1980 with Sir Robert Armstrong, Thatcher’s principal private secretary, abandoning his role as chief defender of official secrecy and arguing that publication of the official histories “would provide a valuable boost to the morale of present-day intelligence staffs.” Armstrong did warn that descriptions of what went on at Camp 020, MI5’s interrogation center in Lachmere House, west London, where captured German agents were taken, would have to be “sanitized out.” Its activities were “sometimes understandably grisly,” Armstrong noted. But with some judicious censoring, he was happy to see British intelligence wartime successes officially recognized.

The file shows the amount of time and energy ministers and senior Whitehall officials spend on suppressing information

Thatcher strongly disagreed. Publication of the volumes “would hand fresh material on a plate to skilled investigative journalists for further exploitation,” she told Armstrong. The dispute dragged on, and Howard became increasingly impatient. The file refers to an article I wrote in the Guardian in November 1983 in which I quoted the historian’s unhappiness about Thatcher’s continuing ban on the official histories.

The following year, Howard told Armstrong that those involved in what he called “one of the greatest success stories of the war” were “fast dying off.” He added: “It is sad that they should not be given the opportunity in their lifetime to enjoy public recognition for the amazing services which they rendered to their country.” Armstrong, too, was getting impatient. The argument that publication would risk current intelligence and security operations was “wearing thin,” he wrote.

Armstrong suggested in 1985 that publication of the official wartime histories would help divert attention away from MI5 at a time it was facing “a good deal of unfavourable comment” over the government’s attempt to ban Spycatcher, the memoirs of the former MI5 officer, Peter Wright, and the disclosure that Michael Bettaney, an MI5 officer who had suffered a breakdown, was arrested after handing over classified information to the Soviet embassy in London. Wright and Bettaney were both victims of appalling MI5 management.

Thatcher was unmoved. Armstrong was told she felt that “too much has been said and written about intelligence and that less should be said.” In April 1986, Thatcher finally relented. The file contains a copy of my report in the Guardian three months later under the heading: “Thatcher drops the ban on MI5 war books”. An official told Nigel Wicks, her principal private secretary: “It appears to be a coincidence that news of it should have reached the press only now though it is not surprising that having found out about it, Mr. Norton-Taylor should play up the alleged contradiction with the Government’s position over the Peter Wright case”.

The file also reveals that Thatcher and her successor, John Major, were concerned that a planned history of the army’s Intelligence Corps would reveal what the regiment had been up to in Northern Ireland. The Ministry of Defence told them not to worry as any reference to Northern Ireland would be “particularly anodyne”.

The file shows the amount of time and energy ministers and senior Whitehall officials spend on suppressing information. The Howard and Simkins histories were finally published in 1990 after a new Official Secrets Act imposed tighter controls over unauthorized disclosures. An account of Camp 020 was published by the Public Record Office (forerunner of the National Archives) in 2000. Ten years later the army published a five-page history of the Intelligence Corps with no reference to its postwar activities in Northern Ireland or anywhere else.