Frequently Asked Questions
- What is LOBSTER?
- Why the name LOBSTER?
- Whats the deal with this SCAMPI project. Why does LOBSTER "succeed" it?
- What does it mean that LOBSTER is a "pilot" project?
Customers / Partners
- Who is behind this project?
- What is "Information Society Technologies"?
- Who are participating in the LOBSTER infrastructure so far?
- Who is it for/who can use it/why?
- Who are your partners?
- I want to become a participant and/or run a passive monitoring sensor. What do I do?
- What are the minimum required resources for participation (estimated minimum cost)?
- If I participate in the infrastructure what is the overhead in my network?
- If I participate in the infrastructure do I have unlimited access to all other participant resources?
- How secure is the infrastructure?
- How reliable is the anonymization?
- What are the objectives for this project?
- What are the milestones?
- What has been done until now, and what is yet to be done?
- Is the information provided by the LOBSTER project only for researchers?
- I need a short summary (1 A4 page) of the LOBSTER project to show to my colleagues and/or superiors.
- I need more information about the background of the LOBSTER project.
- What is your general contact information?
- Is there a public mailing list and/or announcement list I can subscribe to?
Calendar and Events
- Which are the software downloads you can provide?
- How do these projects fit into the LOBSTER project (software roadmap)?
- Can any of these packages be used (standalone) by individuals and/or companies without participating in the LOBSTER project?
- Which are the deployed monitoring sensors?
- What is the architecture of a single monitoring sensor?
- Can you explain how the passive monitoring sensors work?
- Who will be responsible for running the passive monitoring sensors, and where (in the Internet infrastructure) will they be set up?
- Where can I find more information about the equipment used for the passive monitoring sensors?
- Can I run my own passive monitoring sensor for our company network?
- What makes LOBSTER different from previous traditional monitoring systems?
- Why can't Netflow and IPFIX be used for passive monitoring?
- Which are the possible monitoring applications that I could run over the LOBSTER infrastructure?
- How do you secure the infrastructure?
- Do you have any support for privacy and confidentiality?
- What is your support for anonymization?
- What is SiSaL?
- I have a question not answered in this FAQ. Who do I contact?
- I have some great new ideas for the LOBSTER project I think is essential in order to achieve your objectives, how can I contribute?
Frequency Asked Questions - Answers
Q: What is LOBSTER?
A: LOBSTER is a pilot European Infrastructure for accurate Internet traffic monitoring. It is based on passive monitoring at speeds starting at 2.5Gbps, and possibly up to 10Gbps. It aims to enhance the European leadership in network monitoring technologies.
Q: Why the name LOBSTER?
A: LOBSTER is the acronym for the "Large-scale Monitoring of Broadband Internet Infrastructures", the choice was the obvious because LOBSTER is a successor of the recently successfully completed project SCAMPI (SCAlable Monitoring Platform for the Internet).
Q: Whats the deal with this SCAMPI project. Why does LOBSTER "succeed" it?
A: Within the FP5 IST SCAMPI project, we have successfully designed and developed an advanced Internet passive monitoring system at 10Gbps that combines novel hardware and software components. Having met the research challenges posed by passive monitoring at 10Gbps, we propose in LOBSTER to deploy a network of such passive sensors in several key nodes setting the foundations to create a European Passive Internet Traffic Monitoring Infrastructure.
Q: What does it mean that LOBSTER is a "pilot" project?
A: It means that it will be installed in key places in the network in Europe, in order to demonstrate the benefits of distributed passive network monitoring.
Customers / Partners
Q: Who is behind this project?
A: The organizations behind this project are FORTH-ICS (Greece), Vrije Univerity (The Netherlands), Uninett (Norway), CESNET (Czech Republic), FORTHnet (Greece), Alcatel (France), Symantec (UK), TNO Telecom (The Netherlands), and TERENA (The Netherlands). More information on the Partners page.
Q: What is "Information Society Technologies"?
A: IST (Information Society Technologies) is a European Commission Workprogramme, whose objective is "to ensure European leadership in the generic and applied technologies".
Q: Who are participating in the LOBSTER infrastructure so far?
A: So far the partners are participating in the LOBSTER infrastructure. During the first phase of the project monitoring sensors will be deployed in CESNET, UNINETT and ALCATEL. During the second phase, which starts in the third quarter of 2006, additional sensors will be installed in interested organizations around Europe.
Q: What is it for/who can use it/why?
A: LOBSTER will provide a pilot infrastructure for accurate traffic monitoring and measurements.
Accurate network monitoring systems enable the rise of a wide variety of new applications including (1) provision of early warning for the detection of Internet worms as soon as they start to spread, (2) detection of DoS (Denial of Service) attacks, and (3) accurate traffic characterization for applications that use dynamic ports (e.g., traffic from peer-to-peer applications). Upon completion of the project, all the research community interested in network monitoring issues will have the opportunity to use its software and join the LOBSTER infrastructure. So far, several NRNs (National Research Networks) and ISPs (Internet Service Providers) have expressed their willingness to participate in the LOBSTER infrastructure because they will be able to give a better service to their end users.
Q: Who are your partners?
A: The partners of this project are FORTH-ICS (Greece), Vrije Univerity (The Netherlands), UNINETT (Norway), CESNET (Czech Republic), ENDACE (UK), FORTHnet (Greece), ALCATEL (France), TNO Telecom (The Netherlands), and TERENA (The Netherlands). More information on the Partners page.
- Contact LOBSTER consortium (email@example.com) for information about the registration procedure
- Install one or more LOBSTER passive monitoring sensors in your network. Each sensor receives the traffic through a number of taps depending on the number of directional links you want to monitor.
(Network taps are used to create permanent access ports for passive monitoring. A tap can be set up between any two network devices, such as switches, routers and firewalls. A monitoring sensor connected to a tap receives the same traffic as it would if it were located directly on the wire.)
- Install the software nessesary to make use of LOBSTER infrastructure. This software includes Authentication and Anonymization Policy support, Communication protocol, Traffic Measurements daemons (AUTHd, MAPId, diMAPI) etc and will be provided to you by LOBSTER.
- In order to become a member of the LOBSTER infrastructure, you should carefully read the LOBSTER AUP document and comply with all terms and rules in it.
Q: What are the minimum required resources for participation?
1. Dedicated PC with decent CPU/RAM
2. 1 NIC (network interface card) per monitored link
3. 1 NIC for communication control
Some options for NICs:
a) commodity Gbps NIC
b) dedicated hardware:
- DAG card (http://www.endace.com/)
- COMBO card (http://www.liberouter.org/)
1. OS : Linux
2. Tools : LOBSTER protocols, libraries and daemons (will be provided)
Q: If I participate in the infrastructure what is the overhead in my network?
A: There is no direct overhead from the existence of a sensor in your network because there is no injection of additional packets into the network in order to perform the monitoring, unlike the case of active monitoring.
The extra overhead depends on the data that a client will request from that sensor. If requested information from the sensor include only statistics about the monitored data then the overhead is minimal. In the case that the sensor is willing to offer more data (packets) to the client then the overhead is getting larger. The kind of data that a sensor will offer (and the possible overhead) depends on the administrator/owner of the sensor and the policy that will apply to it.
Q: If I participate in the infrastructure do I have unlimited access to all other participant resources?
A: Access to resources depends on the policy that has been specified by the administrator of each node for each user. Policies define which users can access the sensor or not, the amount of computational resources committed to the user, and the form of data seen by the user. For example, a policy can specify that "monitoring requests from the user cannot exceed 50% of the CPU usage and all packets received by his applications have their payload removed".
Q: How secure is the infrastructure?
A: Security was a major issue at the design phase of the infrastructure and covers multiple aspects. Monitoring sensors are protected against DoS attacks as well as eavesdropping adversaries. Moreover, each sensor performs access control, thus allowing access only to registered users. Finally, packet anonymization is supported in order to remove any sensitive information so that such data can be safely distributed. For more information see "How do you secure the infrastructure?"
Q: How reliable is the anonymization?
A: In short, the anonymization is as reliable as you want it to be. As users have full control over which anonymization methods should be used, anonymization policies may range from sending unmodified packets to stripping headers and payload from the packet, and any policy in between these two extremes. Anonymization procedures go beyond the IP packet level as there is also support for anonymization of higher-level protocols. The LOBSTER framework provides anonymization mechanisms rather than predefined policies. The network administrator decides which form of anonymization is most appropriate for which class of users.
Q: What are the objectives for this project?
A: The objectives are:
- Deploy an advanced pilot Internet Traffic Monitoring Infrastructure across Europe
- Organize stakeholders in the area of advanced Internet traffic monitoring
- Realize the appropriate data anonymizing tools that will prohibit unauthorized tampering with the original traffic data
- Develop novel applications enabled by the availability of the passive network traffic monitoring infrastructure
Q: What are the milestones?
A: The milestones are:
0. Requirements Analysis
- The result of this work package is expected to provide essential input to the architecture design and system implementation work packages - Expected date: 6th month of the project
- System definition - Expected date: 12th month of the project
- Release of LOBSTER core - Expected date: 15th month of the project
- Core infrastructure deployed and later - Expected date: 21st month of the project
- The extended infrastructure deployed - Expected date: 27th month of the project
- First LOBSTER workshop - Expected date: 9th month of the project
- Second LOBSTER workshop - Expected date: 21st month of the project
- Operational archive of anonymized traces - Expected date: 27th month of the project
- First year review - Expected date: 15th month of the project
- Second year review - Expected date: 27th month of the project
1. Monitoring Infrastructure Design
2. Monitoring Infrastructure Realization
3. Monitoring Infrastructure Deployment
5. Project Management
Q: What has been done until now and what is yet to be done?
A: At the time of this writing (30/9/2005) the first year of the project has just been completed. Briefly, the first workshop was held last June and many NRNs have been informed about the project and are aware about the current status and the progress. Also a tutorial about passive monitoring and LOBSTER was held in June. From the technical point of view, distributed monitoring API (DiMAPI) has been developed and executed in test-beds. Now we are in the phase of stress testing the developed software. Network monitoring applications are being developed in order to be used with the DiMAPI. Anonymization of traces with IP prefix preserving techniques has been achieved and performance results are examined in order for the application to be improved. Traffic characterization application is on progress. The overall architecture of the system is already designed. For more updated information visit our web pages: http://www.ist-lobster.org/
Q: Is the information provided by the LOBSTER project only for researchers?
A: No. Firstly, as a LOBSTER participant you can monitor data from your own sensors for purposes that you find desirable. Secondly, as a participant you can decide and manage which (anonymized) information you are willing to provide to other LOBSTER participants and third parties.
Q: I need a short summary (1 A4 page) of the Lobster project to show to my colleagues and/or superiors.
A: The LOBSTER project has prepared a project presentation: http://www.ist-lobster.org/deliverables/D4.1b-presentation.pdf
It contains general information about the project, objectives, challenges, technical approach and expected impact.
Q: I need more information about the background of the LOBSTER project.
A: You can find more information about the background of the LOBSTER project from the SCAMPI project website (LOBSTER is a successor of the SCAMPI project):
Other Related Sites also contains a lot of useful information.
Q: Is there a public mailing list and/or announcement list I can subscribe to?
A: There is a public mailing list firstname.lastname@example.org. You can subscribe to that list to receive regular updates on the LOBSTER activities. Note that you will not be able to post to this list. You can subscribe from: http://www.ist-lobster.org/announcements/
Calendar and Events
Q: How can I be notified about the events in the Lobster project?
A: If you are interested to receive information about the LOBSTER events, you can subscribe to the LOBSTER News and Announcement mailing list:
Also you can visit time to time LOBSTER event's page where we have information about all upcoming and past LOBSTER events:
Q: Which are the software downloads you can provide?
A: The software available for download through the Lobster home page is software that is at least partially developed or extended as part of the LOBSTER project. For full description of the available software look at http://www.ist-lobster.org/downloads/
Q: How do these projects fit into the LOBSTER project?
A: All of the software is used for passive monitoring of some kind and has been taken into the LOBSTER project for further development so that they can be used with the LOBSTER infrastructure.
Q: Can any of these packages be used (standalone) by individuals and/or companies without participating in the LOBSTER project?
A: Yes, all the software can be used as standalone applications. They are all released under some type of open source license like GPL or Apache License. See each software package for more details.
Q: Which are the deployed monitoring sensors?
A: So far all the deployed monitoring sensors are used for internal testing within the Lobster project. At the end of the project when all the software needed to run and operate the Lobster infrastructure is ready and stable, the infrastructure will be opened up for external partners and details about deployed monitoring sensors will be provided.
Q: What is the architecture of a single monitoring sensor?
A: A LOBSTER monitoring sensor consists of three main modules: the MAPI daemon, the communication agent, and the authorization daemon. The monitoring daemon (mapid) is the most sophisticated part of the software architecture, as this is where all the processing of the monitoring requirements of remote user applications is performed. Mapid is a user-level process with exclusive access to the captured packets, and is optimized to perform intensive monitoring tasks at high speeds, exploiting any specialized features of the underlying packet capture hardware. The communication agent (commd) handles all communication between remote applications and the sensor, forwarding their monitoring requests to mapid, and sending back to them the computed results. The authorization daemon (authd) is responsible for user authentication and access control. A detailed description of the architecture is provided in D1.2 "Common Access Platform Definition"
Q: Can you explain how the passive monitoring sensors work?
A: At each sensor, the MAPI daemon (mapid) is responsible for capturing and processing the monitored traffic on behalf of user applications. Depending on the monitoring requirements of each user application, mapid processes the captured packets, computes the required metrics, and returns the results to the user application through the communication agent (commd). Mapid supports several packet capturing hardware such as commodity NICs, DAG cards, or COMBO cards. A detailed description of the sensors is provided in D1.2 "Common Access Platform Definition"
Q: Who will be responsible for running the passive monitoring sensors, and where (in the Internet infrastructure) will they be set up?
A: The owner of each sensor is responsible for setting up and running it. There will however be detailed documentation on how a passive monitoring sensor should be installed and there will be a mailing list that can be used for getting help.
The exact location is decided by the owner but will typically be backbone links in NREN's or ISP's.
Q: Where can I find more information about the equipment used for the passive monitoring sensors?
A passive monitoring sensor will typically be a PC with Linux and some hardware adapter for capturing packets. This adapter can be a normal NIC or specialized hardware like Combo6 or DAG cards. Information about these cards can be found here:
Q: Can I run my own passive monitoring sensor for our company network?
A: Yes, you can run your own passive network monitoring sensor in your own company network. Even if you do not participate in the LOBSTER infrastructure you can use the sensor to monitor your own company's traffic.
Q: What makes LOBSTER different from previous traditional monitoring systems?
A: The first difference is that anonymization is a core component of the LOBSTER framework, since it is essential to address privacy concerns that exist when network data is provided to different parties. The second difference is that LOBSTER explicitly addresses the domain of distributed network monitoring. For instance, it supports primitives to apply the same filters or functions to many different remote sensors at once, to aggregate results, etc. The third difference is that LOBSTER is a passive monitoring system, rather than an active monitoring system which is what most other systems deliver. Translated into UNIX terminology, LOBSTER is a member of the tcpdump family (although much more advanced), while most other systems are related to ping or traceroute. Passive monitoring is hard, as it often involves processing huge amounts of traffic at high link rates. On the other hand, compared to active monitoring it provides very different information. For instance, applications may be programs like (i) a network scanner that detects the presence of worms in the network, (ii) a traffic classifier that informs us of the different kinds of traffic streams present in the network, or (iii) an application that estimates the amount of traffic between different sites.
Q: Why can't Netflow and IPFIX be used for passive monitoring?
A: Netflow and IPFIX can and is being used for passive monitoring. However, Netflow and IPFIX only gives details about network flows and not individual packets. It is also not possible to look deeper into packets using Netflow, something that will be possible with the Lobster infrastructure.
Q: Which are the possible monitoring applications that I could run over the LOBSTER infrastructure?
A: LOBSTER by design is a monitoring infrastructure that can be used by any monitoring application that uses the open source DiMAPI interface. This makes a large amount of (custom) applications possible, such as:
- Accurate traffic characterization for programs using dynamic ports.
This application will provide accurate distribution of traffic to applications and will work even for applications that use dynamic ports to communicate, such as peer-to-peer systems. In contrast to our application, traditional traffic characterization methods, such as netflow and IPFIX, are based on static ports and thus are not able to categorize packets belonging to applications that use dynamically generated ports.
- Spread of zero-day worms
Based on traffic captures at several different sensors, this security-related application will focus on finding worms as soon as they start to spread on the Internet, so as to provide an early-warning system.
- Applications performance measurement
Passive measurement can be used for measuring the performance of some applications or services on the Internet. One example of this is DNS measurements. The Lobster infrastructure can give a distributed view of the performance for such services.
- Detect and Trace DOS attacks
DOS attacks often have spoofed source IP addresses. The Lobster infrastructure can help trace the origin such attacks.
- Test platform for new IPFIX attributes
The IPFIX protocol allows for new attributes to be added to a flow record. One problem is that getting new attributes implemented in routers will be difficult and time consuming. The Lobster platform will be a good platform for large scale testing and proof of concept for new attributes.
Q: How do you secure the infrastructure?
A: In order to counter DoS attacks, each monitoring sensor should be equipped at least with the commonly-used security-related software including a firewall, configured using a conservative policy that selectively allows inbound traffic only from the predefined IP addresses of legitimate users. Inbound traffic from any other source is dropped. For protection against eavesdropping, any communication between the monitoring application and a remote sensor is encrypted using the Secure Sockets Layer protocol (SSL). Moreover, each monitoring sensor performs access control based on the user's request and credentials. Credentials are created by the administrator of each monitoring sensor and specify the usage policy applicable to that user. Finally, the sensor architecture supports an advanced framework for creating and enforcing anonymization policies, so that network packets and statistics can be shared between different parties without privacy concerns.
Q: Do you have any support for privacy and confidentiality?
A: Privacy is our key concern in the deployment of the LOBSTER advanced monitoring infrastructure. That is the reason why privacy and confidentiality management are an integral part of LOBSTER by design. At the hardware level encryption, hashing, and stripping of data can be used to make sure that no sensitive data is measured at your node in the LOBSTER infrastructure. Each of the participating organizations can manage the privacy and confidentiality of the measurements that are performed on their measurement points (sensors).
Q: What is your support for anonymization?
A: Anonymization is a core part of the LOBSTER infrastructure. It provides a large set of anonymization primitives that can be applied up to the application layer. Primitives include hashing (MD5, SHA, CRC32, AES and DES algorithms), mapping to sequential values, replace with constant, mapping based on distribution functions (uniform and Gaussian), prefix preserving (for IP addresses), regular expression substitution, checksum adjust (for all protocols) and removal of fields (for application level protocols), thus providing adequate functionality for every user needs. Functions can be applied to any field of most common protocols such as IP, TCP, UDP, ICMP, HTTP or FTP. Anonymization can also be transparently applied to streams rather than raw packets. The administrator is able to define practically any anonymization policy that will be forced to network packets. The anonymization function is currently part of the LOBSTER software. Additionally, we provide a graphical anonymization policy creation tool. (D1.1a - Anonymization Framework Definition)
Q: What is SiSaL?
A: Since anonymization is a fundamental aspect of the LOBSTER framework, it is important to make the specification of anonymization policies as safe and convenient as possible. We do this by providing a specification language SiSaL (Scripting Sanitization Language). SiSaL allows an anonymization policy to be specified in a flexible and concise manner. This clarity of specification avoids errors, and allows a network administrator to confidently specify complex anonymization policies.
Q: I have a question not answered in this FAQ. Who do I contact?
A: All your questions about the LOBSTER project please send to the general email: email@example.com and we will reply as soon as possible. We will update this FAQ with questions received and answers provided.
Q: I have some great new ideas for the LOBSTER project I think is essential in order to achieve your objectives, how can I contribute?
A: Please contact the LOBSTER project at firstname.lastname@example.org describing your ideas and we will contact you as soon as possible to discuss them further.