BLACKLIST ECOSYSTEM ANALYSIS: 2016 UPDATE

1 SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 1 Distribution Statement A: Approved for Public Release; Distribution is Unlimited BLACKLIST ECOSYSTEM analysis : 2016 U P D AT E Leigh Metcalf, Eric Hatleback, Jonathan M. Spring March 2016 Executive Summary This UPDATE , which is the latest in a series of regular updates, builds upon the analysis of blacklists presented in our 2013 and 2014 reports. In those reports, we established that the contents of blacklists generally fail to overlap substantially with each other [1, 2]. This report further corroborates that over-arching result. Our results suggest that available blacklists present an incomplete and fragmented pic-ture of the malicious infrastructure on the Internet, and practitioners should be aware of that insight.

2 This result also provides a starting point for further investigation to understand the dynamics of the BLACKLIST ECOSYSTEM . We have included 123 lists in our latest analysis . This includes 88 IP-address-based lists and 35 domain-name-based lists. The number of indicators included on any individual list varies from under 1,000 to over 50 million. Our analysis covers the 18-month period from July 1, 2014 to December 31, 2015. In this report, we revisit three of the metrics considered in the 2014 report to characterize overlaps: reverse counts, list counts, and pairwise intersection counts. We have omitted the following metric in order to give the issue of following a more complete treatment in a future report.

3 We have added two new metrics: a reverse lookup metric to capture counts of domains seen being resolved in passive DNS, and a persistence in blacklists metric that captures persistence of IPs on blacklists over long spans of time. Most indicators appear on a single list. Our analysis revealed that of IP address indicators appear on exactly one of the lists included in the study. For domain name indicators, appear on a single list. Additionally, in the case of domain-name-based lists, there are two distinct clusters of lists: 13 of the lists (out of 35) are populated in such a way that fewer than half of the domain names listed are active, while 18 of the 35 are populated such that 80% or more of their entries do resolve. SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 2 Distribution Statement A: Approved for Public Release; Distribution is Unlimited 1 Motivation This report is an UPDATE to a thread of investigation that started in 2012 that has so far resulted in two white papers [1, 2] and a conference publication [3].

4 Despite initial community skepticism, our result has been corroborated at least twice [4, 5]. Further, it has inspired some security tools to include lookups to nearly 100 blacklists to better cope with the observed fragmentation of the BLACKLIST ecosys-tem [6]. As a result of the interest surrounding the project, we plan to publish regular updates. By repeating this analysis , we also have the opportunity to develop new comparison techniques to characterize more precisely how the dynamics of the BLACKLIST ECOSYSTEM are evolving. This UPDATE includes two new comparison techniques, which the following sections describe in more detail. The real-world cause of the non-overlap of blacklists is not obvious.

5 We hypothesize that two factors contribute to the differences among blacklists . First, each BLACKLIST provider tailors its detection tech-niques to detect a particular behavior. Second, providers have different views of malicious behavior based on the vantage point afforded by their sensor networks. One empirical test of the sensor vantage hypothesis has been performed [7]. The results are consistent with the hypothesis that sensor vantage is part of the answer, but not the whole answer [8]. Other research into the specialization of the criminal ECOSYSTEM corroborates the conclusion that adversaries change their infrastructure based on their in-tended behavior [9, 10]. This would provide support for the other hypothesized cause for the fragmen-tation of the BLACKLIST ECOSYSTEM , namely that detection methods are tailored precisely to a particular behavior.

6 The themes of secrecy and pragmatism, as usual, generate conflicting incentives that work against se-curity practitioners. BLACKLIST curators do not disclose their list production method(s) or the sensor lo-cation(s), because doing so would generate a means for adversaries to keep themselves off blacklists . Vendors also have a strong economic incentive to keep the details of their technical approach secret from competitors. This reality causes difficulty for cybersecurity practitioners. As identified by Spring [11], in the game of internet security, the system architects and curators often have interests that do not align with those of the users of their service; the nature of the game prevents a solution from being calculated directly.

7 Security practitioners need to be aware of these dynamics when making decisions about acquiring blacklists . The study of quantifiable properties of the BLACKLIST ECOSYSTEM aims to assist in navigating these diffi-culties. Because the main thrust of this work is extended analysis within that existing project, interested readers are encouraged to visit (or revisit) the previous entries in the series for thorough clarification in instances where such depth may be omitted here [1, 2, 3]. We have also published a case study inves-tigating some possible causes of the fragmentation of the BLACKLIST ECOSYSTEM as a blog post [7]. SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 3 Distribution Statement A: Approved for Public Release; Distribution is Unlimited 2 Method The main comparison methods are unchanged from the previous installments, with the exception of the omission of the following metric and the addition of two new metrics.

8 For the basic metrics (reverse counts, list counts, and pairwise intersection counts), the method for obtaining the results is summarized in subsections These metrics are described in more detail in previous reports. Additionally, subsections and include the methods for two new analyses, which we call reverse lookup ( ) and persistence in blacklists ( ). This UPDATE surveys all of the unique indicators in each included list for the 18 months spanning July 1, 2014 to December 31, 2015. This extends the timespan of the project as a whole to 48 continuous months. We have added 38 new blacklists our data set. There are now 88 IP-address-based lists and 35 domain-name-based lists, increased from 67 and 18 (respectively) in the 2014 UPDATE .

9 We have also removed all lists that contain fewer than 100 indicators, since lists of such relatively small size contribute little beyond outliers to the results of the analysis . Every list receives an anonymized identification number prefaced with LI (for IP-address-based lists) or LD (for domain-name-based lists). The unique anonymized identification numbers denote the re-spective lists throughout the report. For each new UPDATE to the project, the anonymization labels are reassigned, partly because the addition of new lists and the removal of defunct lists would otherwise leave confusing gaps in the sequence of labels, and partly as a precautionary measure to maintain the anonymity of the lists over time. As time passes the likelihood that an observed IP address will correspond to a different host (or network owner) will increase.

10 This means that as we extend the time period covered by a particular intersection analysis , we can expect to overestimate overlap. Accounting for this overestimation strengthens our overall findings. This calculation was performed under the heading of following in our 2014 report, demonstrating that much overlap in extended time periods is in fact an artifact of the Internet architec-ture rather than genuine overlap between blacklists . Reverse Counts The reverse count refers to the number of lists on which a particular indicator (IP address or domain name) appears. The minimum number of lists on which an indicator can appear is one and the maximum number (in accordance with the lists surveyed for this UPDATE ) is either 88 (for IP address indicators) or 35 (for domain name indicators).