Columbia Off-Campus Residential Traffic

Data Description

Our pipeline (upgraded in May 2025) is built around an Intel Tofino programmable switch, which runs a scalable online network traffic anonymization system (ONTAS [Kim et al., SIGCOMM 2019 Workshop on Network Meets AI & ML]). This system efficiently anonymizes IP and MAC addresses and flexibly removes packet payload depending on protocol, while preserving key information such as DNS A records and TLS Server Name Indication (SNI).

IP Addresses:

External IP addresses (outside the university): keep as is
IP addresses (of Columbia University): hash in a prefix-preserving manner for the subnets of the residential buildings and Columbia's campus network

MAC Addresses: hash in a prefix-preserving manner to only preserve the Vendor ID
Payload after layer 4 is discarded
Domains (DNS / SNI): only keeps the last three suffixes (i.e., for domain a.b.c.d, we only keep b.c.d)

Packet Traces

We provide access to anonymized, unsampled packet traces (.pcap), allowing you to extract header fields as needed.

Flow associated with domains and services

To reduce the burden of large-scale processing, we provide flows (5-tuples) associated with DNS domains and TLS SNIs, as well as services identified via domain keywords. Metadata example:

Protocol	Remote IP	Anonymized residential IP	Remote-End Port	Residential-End Port	Flow Start Time	Flow End Time	Inbound Bytes	Outbound Bytes	DNS domain (if a mapping exists)	TLS Handshake Extensions Server Name (if a mapping exists)	Service

DNS traffic metadata

We also provide CSV files containing useful information extracted from DNS packets. Metadata example:

Frame Time	Protocol	Source IP	Destination IP	Source MAC Organization Unique Identifier	Destination MAC Organization Unique Identifier	DNS A	DNS CNAME	DNS Response TTL	DNS Query Name	DNS Query Type	DNS Flag Response

Access Request Terms

We are authorized by Columbia University’s IT and Institutional Review Board (IRB) to share our datasets with external researchers.

To ensure user privacy and prevent any risk of de-anonymization, we will share the dataset only with researchers who

have IRB approval or exemption from their own institution (or local equivalent), and
agree to our Acceptable Use Policy

Access Request

Please complete the Research Data Request Form to submit your IRB approval (or local equivalent) and request access.