Mining the Data Fields of the Internet, Part I.

by John Kutzschebauch (JohnK)
Lead Incident Response Engineer for Jawz, Inc.



I believe the saying goes something like: "Information is Power, and he who controls the Information controls the world." Now while this sounds very useful, most of us have problems keeping track of our email let alone trying to handle the formidable task of controlling the considerable amount of information available today via the Internet.

It pretty much seems that if it can be digitized, then someone has uploaded it somewhere onto the Net. Add into the whole seething pot of data overload the extremely prolific amount of "adult" related material masquerading under all forms and shapes of legitimate information. Now your searches for the latest information on Denial Of Service or Buffer Overflows can quickly flag your company's web monitoring software as inappropriate use of company resources.

What we need to do is take existing tools at our disposal and come up with the best methodologies to sift through the incredible amount of information that has little to no bearing on our research. We can then focus our attention on the nuggets of data that surface. Sadly this still can be an overwhelming amount of information but that is what makes the field of Information Assurance such an exciting and valuable career choice. Right?

For this issue I would like to focus on Mailing lists. Remember that reference to email earlier? What a better tie in for a tool set then something that allows you to filter your information through something that everyone uses everyday?

The concept behind mailing lists is very simple. Take an idea, add a program/service called a list server, than find a group of people that are interested in sharing information on that idea through the use of email being funneled through said list server. Mix them all together and you have a potentially very useful knowledge base. Of course add a few too many mailing lists with too many people interested in sharing information and you definitely have an overwhelmed email box.

There are two basic types of mailing lists: Moderated and Un-Moderated. Simply put, Moderated means that someone claims responsibility of the list and filters the mail to ensure that all posts meet a minimum set of requirements. Un-Moderated basically means free for all. Though most mailing lists that are "Un-Moderated" actually do have an administrator that will step in from time to time to ensure that proper form is kept. After all, allowing people an open avenue of expression can quickly turn into chaos and confusion without a little law and order enforced.



One of the largest and best known mailing lists for security is Bugtraq. Known for many years as one of the premier sources of exploits and vulnerabilities it has come under attack by numerous companies and "responsible" security experts as a conduit for hackers and script kiddies to take advantage of the unwary and slow to patch system administrators.

Luckily through the responsible nature of many of the exploit creators (an unwritten rule specifies to give the program creator several weeks to a month to attempt to respond or in some way resolve the problem before going public with the exploit) and the incredible reader support of the list, Bugtraq has managed to stay around and influence the creation of several offshoots.

Now mailing lists have been created that cover Vulnerabilities, Forensics, Firewalls, if there is a security subject, then most likely there is a mailing list that will cover it.

One of the best sites for mailing lists is Security Focus. They currently host Bugtraq and numerous other valuable mailing lists. They not only allow for you to subscribe to each mailing list and have it delivered directly to your email address for your perusal. But some have an alternate combined mailing list that will once a week send you one large email of all of that week's messages. Either way you have direct access to the posts and can respond with your own observations or even make requests for more information if needed.



A different approach offered is the web based archive option. Depending on the site, the mailing list is either archived by date or by thread and date. Security Focus archives by date so you can make sure you are keeping current with that week or even that day's news on the list. But other sites such as Insecure.org offer browsing by thread so you can see the whole discussion instead of having to wander through posts trying to see what everyone's view on the subject is. Of course the downside of using this approach is the fact it is a read only tool. You can see what others have posted but you are unable to provide any information or request any assistance.

My preferred method is to subscribe to all of the mailing lists I believe I may want to interact with. Then using an option provided for people who will be away from their mailbox for sometime, I send the list server a command telling it that I am away but not to unsubscribe me. This way I can browse through the archives yet still post to the list when I need to. No clutter in the mailbox, but still able to interact with an excellent source of information.



In an effort to create a better source of archives I am currently working on a project with Security Horizon to provide a larger source of archives that are searchable and easily accessed. So stay tuned, by the time this article reaches print that project should be up and running so you can use it as a valuable tool for your research.

Overall mailing lists are an extremely useful avenue of research. Unless your preferred method of learning about Denial Of Service is to watch Microsoft web sites as they get hit by the latest attacks, I would suggest you look into some of the above mentioned web sites and see what lists you can use to improve your information base.

Thank you and I look forward to discussing more ways to use the Internet for your research and benefit instead of for your bewilderment and confusion. We work in an environment that is literally changing everyday and the only way to stay current is to stay in the middle of the information flow.