Like SIS, MI5 has its basis in the Secret Service Bureau[?], founded in 1909 as an organisation to control secret intelligence operations. The Bureau was originally split into a naval and military section. The naval section came to specialise in espionage activities in foreign countries, while the military section increasingly undertook counter-espionage activities within the UK. This new split was formalised. After a series of bureaucratic designation changes in which it was known as MO5 and gained various subdepartments denoted by letters of the alphabet, the domestic section came to be known as MI5, a name it retains today.
Its founding head was Vernon Kell[?], who remained head until the early part of the Second World War. Its role was originally quite restricted; it existed purely to ensure national security through counter-espionage. It originally worked in concert with the Special Branch of the Metropolitan Police[?]; MI5 was responsible for overall direction and the actual identification of foreign spies, while the Special Branch provided the manpower for the investigation of their affairs and their arrest and interrogation.
MI5 was very successful (against admittedly weak opposition) in the pre-war years. It was founded in a climate of hysteria over a supposedly huge network of German spies - numbers in the hundreds of thousands were quoted - who were apparently ready to perform espionage and sabotage activities in advance of a German invasion. In reality, no invasion was planned, and Germany had a mere handful of incompetent amateur spies active in Britain - just over 20. MI5 was quickly successful in identifying this group, and Kell took the intelligent decision not to arrest them but to keep them under surreptitious observation until the outbreak of war. He reasoned that if they were arrested Germany would simply send more in their place. Instead he waited until the eve of war - he was given twelve hours' notice of its outbreak - to arrest the entire network, thus depriving Germany completely of reliable intelligence from within Britain.
After this auspicious start, the history of MI5 becomes darker. It was consistently successful throughout the rest of the 1910s and the 1920s in its core counter-espionage role, however. Germany continued to attempt to infiltrate Britain throughout the war, but using a method that depended on strict control of entry and exit to the country and, crucially, large-scale inspection of mail, MI5 was easily able to identify all the agents that were dispatched. In post-war years attention turned to attempts by Russia and the Comintern to surreptitiously support revolutionary activities within Britain, and MI5's expertise combined with the early incompetence of the Russians meant the bureau was successful once more in correctly identifying and closely monitoring these activities.
However, in the meantime MI5's role had been substantially enlarged. Due to the spy hysteria, MI5 was formed with far more resources than it actually needed to track down German spies. As is common within governmental bureaucracies, this meant it expanded its role in order to use its spare resources. MI5 acquired many additional responsibilities during the war. Most significantly, its strict counter-espionage role was considerably blurred. It became a much more political role, involving the surveillance not merely of foreign agents but of pacifist and anti-conscription organisations, and organised labour. This was justified on the basis of the common (but mistaken) belief that foreign influence was at the root of these organisations. Thus by the end of the war MI5 was a fully-fledged secret police, in addition to being a counter-espionage agency. This expansion of its role has continued, after a brief post-war power struggle with the head of the Special Branch, Sir Basil Thompson. MI5 also managed to acquire responsibility for security operations not only in mainland Britain but throughout the Empire, which gave it a significant role in Ireland. MI5 now has a role similar to that of the American FBI, if not as extensive, which includes crime-prevention activities as well as political surveillance and counter-espionage. This expansion has happened almost entirely without supervision; MI5 had no responsibility to Parliament, and is often able to act with considerable independence even from the Cabinet and Prime Minister. Since the 1997? MI5 activities have been subject to scrutiny by Parliament's Intelligence and Security Committee.
MI5's Irish operations were an unmitigated disaster. Its operation was penetrated by the IRA, and even before Michael Collins ordered a ruthless purge of MI5's Irish agents - almost all of whom were assassinated - it was unable to provide useful intelligence on the Irish republican movement during the Home Rule and independence controversies.
MI5's decline in counter-espionage efficiency began in the 1930s. It was to some extent a victim of its own success; it was unable to break the ways of thinking it had evolved in the 1910s and 1920s. In particular, it was entirely unable to adjust to the new methods of the NKVD, the Russian secret intelligence organisation. It continued to think in terms of agents who would attempt to gather information simply through observation or bribery, or to agitate within labour organisations or the armed services, while posing as ordinary citizens. The NKVD, however, had evolved more sophisticated methods; it began to recruit agents from within the Establishment, most notably from Cambridge University, who were seen as a long-term investment. They succeeded in gaining positions within the Government (and, in Kim Philby's case, within British intelligence itself), from where they were much more easily able to provide the NKVD with sensitive information. The most successful of these agents - Harold 'Kim' Philby, Donald Maclean[?], Guy Burgess[?], Anthony Blunt[?] and John Cairncross[?] - went undetected until after the Second World War, and were known as the Cambridge Five.
MI5 experienced further failure during the Second World War. It was chronically unprepared, both organisationally and in terms of resources, for the outbreak of war, and utterly unequal to the task which it was assigned - the large-scale internment of enemy aliens in an attempt to uncover enemy agents. The operation was badly mishandled and contributed to the near-collapse of the agency by 1940. One of the earliest actions of Winston Churchill on coming to power in early 1940 was to sack the agency's long-term head, Vernon Kell. He was replaced initially by the ineffective Brigadier A.W.A. Harker. Harker in turn was quickly replaced by David Petrie, an SIS man, with Harker as his deputy. With the ending of the Battle of Britain and the abandonment of invasion plans (correctly reported by both SIS and the Bletchley Park ULTRA project), the spy scare eased, and the internment policy was gradually reversed. This eased pressure on MI5, and allowed it to concentrate on its major wartime success, the so-called "double-cross" system.
This was a system based on an internal memo drafted by an MI5 officer in 1936, which criticised the long-standing policy of arresting and sending to trial all enemy agents discovered by MI5. Several had offered to defect to Britain when captured; prior to 1939, such requests were invariably turned down. The memo advocated attempting to "turn" captured agents wherever possible, and use them to mislead enemy intelligence agencies. This suggestion was turned into a massive and well-tuned system of deception during the Second World War. Beginning with the capture of an agent called Owens, codenamed SNOW, MI5 began to offer enemy agents the chance to avoid prosecution (and thus the possibility of the death penalty) if they would work as British double-agents. Agents who agreed to this were supervised by MI5 in transmitting bogus "intelligence" back to the German secret service, the Abwehr. This necessitated a large-scale organisational effort, since the information had to appear valuable but in actual fact be misleading. A high-level committee, the Wireless Board, was formed to provide this information. The day-to-day operation was delegated to a subcommittee, the Twenty Committee (so called because the Roman numerals for twenty, XX, form a double cross). The system was extraordinarily successful; in fact, every important German agent in Britain during the war was in fact a double agent controlled by MI5. The system played a major part in the massive campaign of deception which preceded the D-Day landings, designed to give the Germans a false impression of the location and timings of the landings.
As with SIS, there was until recently an official pretence that MI5 did not in fact exist. This was abandoned in the mid-1990s, and MI5 has recruited openly through newspaper advertisements since 1997.
As well as the currently extant MI5 and MI6, there have been a number of British military intelligence groups designated as MI-(section number) existing at various times since WWI, which have now been abandoned or subsumed by MI5, MI6 or GCHQ. These included MI1 (codebreaking), MI2 (intelligence in Russia and Scandinavia), MI3 (Eastern Europe), MI4 (aerial photographic interpretation), MI8 (signals intelligence), MI9 (covert operations and PoW escape), MI10 (weapons analysis) and MI19 (PoW debriefing).