Scalable network-based buffer overflow attack detection

Fu Hau Hsu, Fanglu Guo, Tzi Cker Chiueh

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

13 Scopus citations

Abstract

Buffer overflow attack is the main attack method that most if not all existing malicious worms use to propagate themselves from machine to machine. Although a great deal of research has been invested in defense mechanisms against buffer overflow attack, most of them require modifications to the network applications and/or the platforms that host them. Being an extension work of CTCP, this paper presents a network-based low performance overhead buffer overflow attack detection system called Nebula 1 NEtwork-based BUffer overfLow Attack detection, which can detect both known and zero-day buffer overflow attacks based solely on the packets observed without requiring any modifications to the end hosts. Moreover, instead of deriving a specific signature for each individual buffer overflow attack instance, Nebula uses a generalized signature that can capture all known variants of buffer overflow attacks while reducing the number of false positives to a negligible level. In addition, Nebula is built on a centralized TCP/IP architecture that effectively defeats all existing NIDS evasion techniques. Finally, Nebula incorporates a payload type identification mechanism that reduces further the false positive rate and scales the proposed buffer overflow attack detection scheme to gigabit network links.

Original languageEnglish
Title of host publicationANCS 2006 - Proceedings of the 2006 ACM/IEEE Symposium on Architectures for Networking and Communications Systems
Pages163-171
Number of pages9
DOIs
StatePublished - 2006
Event2nd ACM/IEEE Symposium on Architectures for Networking and Communications Systems, ANCS 2006 - San Jose, CA, United States
Duration: 3 Dec 20065 Dec 2006

Publication series

NameANCS 2006 - Proceedings of the 2006 ACM/IEEE Symposium on Architectures for Networking and Communications Systems

Conference

Conference2nd ACM/IEEE Symposium on Architectures for Networking and Communications Systems, ANCS 2006
Country/TerritoryUnited States
CitySan Jose, CA
Period3/12/065/12/06

Keywords

  • Buffer overflow attacks
  • CTCP
  • Generalized attack signatures
  • Network-based intrusion detection
  • Payload bypassing
  • Return-into-libc attacks

Cite this