(Updated: Nov 23, 2016)
One of the most controversial NSA programs was the volume collection of domestic telepone records (metadata) nether authorization of Section 215 of the USA PATRIOT Act.
The Snowden revelations provided hardly whatever information close this program, but many details became available from documents that were declassified past times the US Director of National Intelligence (DNI).
Because inwards these declassified documents all codenames are redacted, it was largely a mystery which NSA systems were used to shop together with analyse these metadata.
By combining many split upward pieces from both the Snowden-documents, equally good equally those declassified past times the government, it instantly has dice clear that NSA seat the domestic telephone records inwards its fundamental contact chaining organization MAINWAY, which also contains all sorts of metadata collected overseas.
Reconstruction of the MAINWAY dataflow
(Click to enlarge)
MAINWAY versus MARINA
Initially it was thought that MAINWAY was a repository exactly for telephone metadata. This goes dorsum to a report past times USA Today from May 10, 2006, which revealed that the NSA created a database containing "the telephone telephone telephone records of tens of millions of Americans" obtained from AT&T, Verizon together with BellSouth (the latter merged with AT&T equally of 2007).
As such, MAINWAY was seen equally the equivalent of MARINA, which is NSA's storage for cyberspace metadata. But meanwhile, diverse documents from the Snowden revelations receive got made clear that the actual repositories for telephone metadata are ASSOCIATION (for metadata from mobile calls) together with BANYAN (for metadata from landline calls).
MAINWAY itself isn't exactly a database that stores raw metadata, but a organization that also "performs information quality, preparation together with sorting functions, together with and so summarizes contacts represented inwards the processed data". Afterwards, MAINWAY stores the "resulting contact chains together with provides analysts with access to these contact chains".
New documents receive got also shown that MAINWAY contains metadata from cyberspace communications too. For example, inwards the next diagram close the FAIRVIEW collection program, nosotros encounter that cyberspace metadata from the Upstream collection initiative of all flow into MAINWAY before ending upward inwards MARINA:
Dataflow for cyberspace metadata collected nether the
FAIRVIEW programme nether Transit Authority
(Click to enlarge)
It seems probable that inwards MAINWAY, metadata are stored to a greater extent than or less temporarily for the purpose of contact chaining together with analysing them. Metadata that NSA wants to proceed for a longer catamenia of time, or fifty-fifty indefinitely are together with so stored inwards repositories similar MARINA, ASSOCIATION together with BANYAN.
(However, a report past times The Guardian from September 30, 2013 says that MARINA "has the mightiness to hold off dorsum on the final 365 days' worth of DNI metadata seen past times the Sigint collection system")
While the domestic metadata collected inwards volume receive got to hold upward destroyed afterward 5 years, the calling records that are the consequence of a inquiry tin hold upward stored past times the analyst. According to the PCLOB-report (.pdf), they may together with so hold upward "subjected to other analytic methods or techniques likewise querying, or integrated with records obtained past times the NSA nether other authorities", equally good equally shared with others within together with exterior NSA.
MAINWAY, SIGINT Navigator (SIGNAV), ASSOCIATION together with BANYAN
mentioned inwards a presentation close DEMONSPIT, nether which call
records were obtained from major Islamic Republic of Pakistan telecom providers(!)
(Click to enlarge)
MAINWAY receiving domestic telephone records
Based upon Snowden documents, The New York Times reported on September 28, 2013, that MAINWAY is used for chaining both telephone numbers together with email addresses together with that it is fed with information from tapping "fiber-optic cables, corporate partners together with unusual figurer networks that receive got been hacked".
The study also says that equally of August 2011, MAINWAY was fed with "1.1 billion cellular records a twenty-four lx minutes catamenia inwards add-on to the 700M records delivered currently". However, The New York Times erroneously attributed these numbers to collection nether authorization of department 702 FAA together with was thus non able to seat that MAINWAY was also fed with the volume telephone records of Americans (which happens nether department 215 Patriot Act).
The latter exclusively became clear afterward The New York Times together with ProPublica published some NSA documents close the FAIRVIEW programme on August 15, 2015. One of these documents confirms that it was AT&T that provided the aforementioned number of records, together with also that this happened nether BR FISA (= Section 215) authority.
(A report past times the Washington Post from June 15, 2013 also identified MAINWAY equally the database inwards which the telephone records from the Section 215 programme were stored)So equally of 2011, at to the lowest degree 1,8 billion domestic telephone records a twenty-four lx minutes catamenia were coming in, which makes 54 billion a calendar month together with close 650 billion a year. Before they were handed over to NSA, AT&T stripped off the location information inwards gild to comply with the FISA Court orders, that don't allow those information to hold upward collected.
Apparently Verizon Wireless together with T-Mobile US saw no obligation to take these location data, so their prison theater cellular telephone telephone records couldn't hold upward collected past times NSA, which thus exclusively got less than 30% of the domestic telephone metadata.
According to NSA, 1 of the advantages of putting telephone records from multiple American telecommunication companies inwards 1 large repository, was that this allowed analysts "to seat chains of communications that cross dissimilar telecommunication networks".
Under the President's Surveillance Program (2001 - 2004/2006)
NSA started collecting telephone together with cyberspace metadata from US telecommunication providers before long afterward the attacks of September 11, 2001. This was component subdivision of the President's Surveillance Program (PSP, protected nether the STELLARWIND classification compartment), which was based upon what inwards the cease would hold upward 43 subsequent hole-and-corner authorizations past times president George W. Bush.
The goals of collecting these metadata were identifying unknown terrorist operatives through their contacts with known suspects, honour links betwixt known suspects, together with monitor the designing of communications alongside suspects.
At first, exclusively metadata were collected from communications inwards which at to the lowest degree 1 political party was exterior the US. AT&T (identified equally Company A, codenamed LITHIUM, with collection nether FAIRVIEW) started to render both telephone together with cyberspace metadata from international channels equally early on equally Nov 2001. For Verizon (Company B, with collection nether STORMBREW) the automated transfer of such information started inwards Feb 2002. Qwest refused to mitt over its records because the authorities couldn't acquaint a warrant.
Allegedly, raw metadata were selected for processing. These were flowing into the MAINWAY contact chaining database, which also contains metadata from collection abroad. The 2009 report close the STELLARWIND programme says:
"NSA's primary tool for conducting metadata analysis, for PSP together with traditional SIGINT collection, was MAINWAY. MAINWAY was used for storage, contact chaining, together with for analyzing large volumes of global communications metadata."
(interestingly, inwards some documents MAIN WAY seems to hold upward written equally 2 split upward words, which arrive resemble MAIN CORE, which is a fundamental database containing essential intelligence information on Americans produced past times the FBI together with other US intelligence agencies)
Under FISA Court orders (2004/2006 - 2011/2015)
In July 2004, the collection of domestic cyberspace metadata was moved from the President’s Surveillance Program to the FISA Court, which authorized this endeavour based upon department 402 FISA, or equally it is called past times NSA: PR/TT (short for Pen Register/Trap together with Trace).
In May 2006, the same happened with the volume telephone records, for which the FISA Court allowed continuation nether authorization of department 215 USA PATRIOT Act, or equally NSA calls it: BR FISA (short for Business Records FISA).
Under the FISA Court orders, volume telephone collection eventually became to include "all telephone telephone particular records or 'telephony metadata' created [...] for communications betwixt the U.S. of A. of America together with abroad" or "wholly within the United States, including local telephone calls". Only metadata of fully unusual communications were excluded, equally was the instance for most mobile telephone calls, due to technical reasons.
Because correct from the beginning, NSA stored these domestic telephone together with cyberspace metadata inwards the same database (MAINWAY) that contains metadata from traditional collection efforts abroad, queries could consequence inwards contacts chains made upward of identifiers from both unusual together with domestic sources. The inquiry tool exactly didn't seat the difference.
Also it was possible for analysts to start a inquiry with selectors that were non BR FISA-approved, together with inwards some cases this also provided results from both the unusual together with the domestic collection. This was non according to the FISA Court orders, together with afterward NSA informed the courtroom close this, they had to halt accessing the telephone metadata inwards 2009, until these issues had been solved.SIGAD of the intercept facility where that had happened.
H5N1 rare diagram close the BR FISA metadata collection:
the conclusion procedure equally it was from 2006 - 2009
(an explanation of this procedure tin hold upward read here)
(Source - Click to enlarge)
Other databases for domestic telephone telephone records
The domestic telephone telephone records were non exclusively stored inwards MAINWAY, but also inwards some other database, 1 that was plainly dedicated for US telephone metadata. An NSA training presentation (.pdf) from 2007 confirms that BR FISA information were stored inwards 2 NSA repositories, although both names had been redacted.
An NSA review from June 2009 describes this 2d database equally a "repository for private BR FISA metadata telephone telephone records for access past times authorized Homeland Security Analysis Center (HSAC) together with information integrity analysts to persuasion detailed information close specific telephony calling events".
This seems to refer to the consummate calling records, together with also the PCLOB-report (.pdf) close the BR FISA programme says there's analysis software that "provides the associated information close the telephone calls involved, such equally their date, fourth dimension of day, together with duration".
So likely the 2d database gave access to these additional details, whereas MAINWAY exclusively contains or provides "summaries of one-hop chains", i.e. selector #1 was inwards contact with selector #2 together with the number of times this happened within a specific timeframe.
In the glossary of the 2009 NSA Review, the 2d repository is listed with a remarkably long name, which, according to its position, has to start with together with M, northward or O:
This exceptionally long call of the 2d database could signal that it was some form of provisional repository, because on page 23 of the 2009 BR FISA review it is said:
"NSA is preparing to contain the [second database] into the NSA corporate architecture. This transition to the corporate applied scientific discipline framework volition maximize exercise of the latest technologies together with proven configuration administration to minimize whatever safety together with compliance risks"
And indeed, inwards appendix B of a report (.pdf) past times the NSA's Inspector General from August 1, 2012, nosotros encounter that the 2d database instantly has a shorter name, together with that it had replaced a "Transaction Database" with a much longer call inwards Jan 2011:
Transaction is some other term that NSA uses for metadata, so "transaction database" likely exactly way that it contains the (full) metadata records. This 2012 Inspector General study lists 3 additional storage systems for BR FISA data, making a total of 5 beingness involved here:
1. Contact chaining database that accepts metadata from multiple sources (= MAINWAY)
2. Database repository that stores detailed metadata information, which supports the contact chaining summaries inwards [MAINWAY]. Replaced an before database inwards Jan 2011.
3. Contingency database for the fourth dimension the aforementioned database was beingness rebuild
4. System backup that stores an exact re-create of the raw metadata from the providers
5. Backup tapes on which periodically the raw metadata were saved off-line
So when NSA needs large information centers, that's also because the same sets of information are stored multiple times. Besides backups, at that topographic point are oft split upward databases dedicated to a specific purpose or analysis method.
Bulk cyberspace metadata (PR/TT)
As mentioned before, MAINWAY was non exclusively fed with telephone metadata, but also with metadata from domestic cyberspace communications. These metadata include the "to", "from", together with "cc" lines of an e-mail, equally good equally the e-mail’s fourth dimension together with date. Its seems that for contact chaining, no metadata from other kinds of cyberspace communications, similar messengers, were used.
On August 11, 2014, an internal NSA Review (.pdf) close this PR/TT programme was declassified, which shows similar storage systems equally for the telephone records: total copies of the cyberspace metadata were also stored inwards the MAINWAY contact chaining database, equally good equally inwards a dedicated 2d repository:
The PR/TT volume cyberspace metadata programme was unopen downward inwards Dec 2011 for "operational together with resources reasons" together with all information were deleted. Based upon declassified NSA reports, The New York Times reported on Nov 19, 2015, that this "internet dragnet" was ended because, alongside other reasons, similar results could hold upward achieved nether other authorities:
- Section 702 FAA, which allows access to cyberspace communications betwixt foreigners together with Americans from the "PRISM-providers" together with "Upstream collection".
- The SPCMA regulation, which allows using US mortal identifiers for querying metadata that receive got been collected abroad.
With collection of cyberspace metadata both overseas (under EO 12333 authority) equally good equally at the physical together with virtual borders of the US (under 702 FAA), NSA likely didn't postulate the purely domestic ones anymore, to silent capture those that are of interest.
Also, querying the metadata collected overseas appeared wanted to create (or perhaps expand MAINWAY into) a metadata repository capable of taking inwards 20 billion metadata records a twenty-four lx minutes catamenia together with brand these available to analysts within lx minutes.
But afterward Snowden disclosed the Verizon volume telephone records gild inwards June 2013, the American populace became aware of the actual range of this programme together with it became the most controversial component subdivision of NSA's activities.
In Jan 2014, the Privacy together with Civil Liberties Oversight Board (PCLOB) judged that Section 215 collection was genuinely of "minimal value inwards safeguarding the state from terrorism" together with that at that topographic point was "no instance inwards which the programme straight contributed to the uncovering of a previously unknown terrorist plot or the disruption of a terrorist attack".
According to PCLOB, the volume telephone records did render some value "by offering additional leads regarding the contacts of terrorism suspects already known to investigators, together with past times demonstrating that unusual terrorist plots practise not receive got a U.S. nexus". This however, was non seen equally a sufficient justification for the large-scale collection of domestic telephone records.
In the class of 2015, US Congress eventually enacted the USA FREEDOM Act, which prohibits NSA to collect together with shop domestic telephone telephone records inwards volume equally of Nov 29, 2015. Instead, the agency instantly has to apply for a warrant from the FISA Court approbation specific selectors, which are together with so provided to telecommunication providers, who exercise them for querying their ain databases together with exclusively the results are handed over to NSA.
> See: Collection of domestic telephone records nether the USA FREEDOM Act
> Next: How NSA contact chaining combines domestic together with unusual telephone records
Links together with Sources
- EmptyWheel.net: At the 2d NSA unopen downward the PRTT metadata dragnet, FISC permitted it to inquiry Upstream metadata (2017)
- Lawfare Blog: Understanding Footnote 14: NSA Lawyering, Oversight, together with Compliance (2016)
- EmptyWheel.net: What We Know close the Section 215 Phone Dragnet together with Location Data (2016)
- PCLOB: Report on the Telephone Records Program Conducted nether Section 215 of the USA PATRIOT Act (pdf) (2014)
- Cryptome.org: NSA FISA Business Records Offer a Lot to Learn (2013)
- US Administration White Paper: Bulk Collection of Telephony Metadata nether Section 215 of the USA PATRIOT Ac(pdf) (2013)
- NSA: Business Records FISA NSA Review (.pdf) (2009)
- NSA: Pen Register/Trap together with Trace FISA NSA Review (.pdf) (2009)
- Andrew P. MacArthur: The NSA Phone Call Database: The Problematic Acquisition together with Mining of Call Records inwards the United States, Canada, the United Kingdom, together with Australia (2007)
Tidak ada komentar:
Posting Komentar