Performance analysis of jNetPcap library

We are working on creation of VoIP sniffer application which must to capture RTP packets and SIP messages. We had to make the performance analysis to find the quickest way to free internal jNetPcap ring buffer and avoid network packets loss. Our application is used on multi core PCs, e.g. Xeon E5630.

For analysis of the performance jVisualVM profiler 1.3.3. was used. For the test I used trace pcap file with 262000 network packets including 1500 RTP packets and some small number of SIP messages. So useful traffic was just 1%.

Our first implementation of a capture thread was like following:

public void nextPacket(final PcapPacket packet, final String unused) {
   
    NetworkPacket outputPacket = null;
    boolean tcp = false;

    try {
        if ((packet.hasHeader(updHeader) || (tcp=packet.hasHeader(tcpHeader))) && !packet.hasHeader(icmpHeader)) {
           
            if (packet.hasHeader(rtpHeader)) {
                if ( packet.hasHeader(ipHeader) ) {
                    outputPacket = new RtpPacket(ipHeader, updHeader, rtpHeader);
                }
            } else if (packet.hasHeader(sipHeader)) {
                if ( packet.hasHeader(ipHeader) ) {
                    if (tcp) {
                        outputPacket = new SipPacket(ipHeader, tcpHeader, sipHeader);
                    } else {
                        outputPacket = new SipPacket(ipHeader, updHeader, sipHeader);
                    }
                }
            }

        if (outputPacket != null) {
            if (!packetProcessingQueue.offer(outputPacket)) {
                logger.error("packetProcessingQueue is full when packet [{}] is received", packet.getFrameNumber());
            }
        }

    } catch (ExceptionBadPacket e) {
        logger.info("Incorrect packet has been received from JNetPacket", packet.getFrameNumber());
    }
}

The RtpPacket and SipPacket constructors make copy of the network packets payload.
The profiler showed following results.

After that I tried to replace hasHeader() by getHeader() function call. It is less convenient because an user have to compare with null. However this function enabled us to increase the speed twice.

I was sure that the filtering enabled us to get higher speed because we skipped unnecessary packets and copied only useful RTP and SIP packets. However the results was absolutely unexpected for me. jNetPcap library copies network packets faster than checks its type. So most efficient way is to capture ALL network packets into the second thread and make analysis there.

The source code for the capture thread looks like:

public void nextPacket(PcapHeader header, JBuffer buffer, String any) {
   
    PcapPacket packet = new PcapPacket(header,buffer);
   
    if (packet != null) {
        if (!packetProcessingQueue.offer(packet)) {
            logger.error("packetProcessingQueue is full when packet [{}] is received", packet.getFrameNumber());
        }
    }
}

Thread states in VisualVM profiler

VisualVM uses following states to indicate thread activity status:

• Running; 
• Sleeping; 
• Wait; 
• Monitor. 

 

What do these states mean?

Running.

Thread.State = RUNNABLE.

"Thread state for a runnable thread. A thread in the runnable state is executing in the Java virtual machine but it may be waiting for other resources from the operating system such as processor."

Sleeping

Thread.State = TIMED_WAITING.

 Thread state for a waiting thread with a specified waiting time. A thread is in the timed waiting state due to calling one of the following methods with a specified positive waiting time:

• Thread.sleep
• Object.wait with timeout
• Thread.join with timeout
• LockSupport.parkNanos
• LockSupport.parkUntil

Wait


Thread.State = WAITING.

Thread state for a waiting thread. A thread is in the waiting state due to calling one of the following methods:

• Object.wait with no timeout
• Thread.join with no timeout
• LockSupport.park

A thread in the waiting state is waiting for another thread to perform a particular action. For example, a thread that has called Object.wait() on an object is waiting for another thread to call Object.notify() or Object.notifyAll() on that object. A thread that has called Thread.join() is waiting for a specified thread to terminate.

Monitor

Thread.State = BLOCKED.

Thread state for a thread blocked waiting for a monitor lock. A thread in the blocked state is waiting for a monitor lock to enter a synchronized block/method or reenter a synchronized block/method after calling Object.wait.

SIP protocol parsing: How to match call legs.

One of the main tasks for any protocol parser is how to find parts of the same call (call legs).

Any call is established through PBX and looks like it is showed in the picture 1 or 2.

Subscriber A calls to subscriber B. In the beginning a connection from Party A to PBX is established. Then PBX establishes the connection to Party B. From SIP protocol point of view two different calls are established with different Call-Id 12 and 23. After that RTP streams are transmitted from Party A to Party B and vice versa. There are two possible routes. The RTP streams could be passed through the PBX or directly between the participants.
 
You can see that one conversation consists of two SIP calls with different Call-Id. One call is established to PBX and another call is established from PBX. These calls are usually called "call legs".

Lets look into one real trace of a basic call scenario.

In the picture above PBX has 192.168.2.102 address.

Two calls with different call ids were established. They depicted by wireshark tool in different colors. Lets look into details of the calls.

The first call has Call-Id="***b3b1b". 
The calling party has extension "49602150014015" and media IP address 192.168.2.63.
The called party has extension  "4100" and media IP address 192.168.2.59.

 

The second call has Call-Id="***JPwcK6".
The calling party has extension "49602150014015" and media IP address 192.168.2.63.
The called party has extension  "49602150014100" and media IP address 192.168.2.59.

You can see that the PBX substitutes short phone extension for long extension and vice-versa. That is why it is not possible to use phone extensions to match call legs. On the contrary media IP addresses are constant.

So SIP call legs can be matched only by using media IP addresses.

Pitfalls of the JNetPcap library

During some software development we have found the following problems with using the JNetPcap library (1.3.0). Points 1 and 4 are still valid for the version 1.4.r1300 (I didn't check the other points). I would like to share our experience.


1. SIP messages support.

In order to check if some packet contains SIP message, we use the following code:

public void nextPacket(PcapPacket packet, String arg1) {
    Sip sipHeader = new Sip();
    if (packet.hasHeader(sipHeader)) {
        doSomething();
    }
}

However JNetPcap doesn't recognize all SIP messages. For example, the messages NOTIFY (RFC3265), SUBSCRIBE (RFC3265) and  REFER (RFC3515) will not be parsed as a SIP message and hasHeader(sipHeader) call will return false. Probably the messages INFO (RFC2976), UNSUBSCRIBE (RFC3265) and some other methods are not also supported by the library.

Update: Version 1.4.r1300 doesn't still support these messages.

2. SIP messages payload

To get a SIP message we used method getPayload() of Sip class. However, sometimes the method returned null for good SIP message. We realised that it is necessary to use Udp or Tcp class to have reliable access to the payload. Following code shows the work around solution:

Udp udpHeader = new Udp();
Tcp tcpHeader = new Tcp();

if ( packet.hasHeader(udpHeader) ) {
    sipMessage = new String(udpHeader.getPayload());
} else if ( packet.hasHeader(tcpHeader) ) {
    sipMessage = new String(tcpHeader.getPayload());
}

3. RTP packets with G.729 payload are not supported

The following code caused the exception "ArrayIndexOutOfBoundsException" when we tried to execute it against RTP packet with G.729 payload inside:

if ( packet.hasHeader(rtpHeader)) {
    payloadType = rtpHeader.typeEnum();
}

Complete exception trace:

Exception in thread "pool-1-thread-2" java.lang.ArrayIndexOutOfBoundsException: 18
    at org.jnetpcap.protocol.voip.Rtp$PayloadType.valueOf(Unknown Source)
    at org.jnetpcap.protocol.voip.Rtp.typeEnum(Unknown Source)
    ...
    at org.jnetpcap.Pcap.loop(Native Method)

The author of JNetPcap promised us to fix this issue in the next release 1.4.0.

4. Incorrect validation of RTP packets

Some time ago I wrote about how to determine if some packet is RTP or not. In that article I showed example from JNetPcap library. It considers RTP packets with zero sequence number and zero timestamp as invalid RTP packets. In practice we received RTP packet with sequence number = 0 and this packet was rejected by JNetPcap. According to the specification RFC3550 it is correct situation. I couldn't find restriction to have zero sequence number and timestamp.

Update: It is still valid for the version 1.4.r1300.

5. Pcap file reopen issue

When some file is opened, parsed, closed and opened again, the frame numbers are not initialized from the zero. The method getFrameNumber() of PcapPacket class returns the next number after the last one. For example, my pcap file contains 20 packets. After reopening it the method getFrameNumber() will return 21 for the first packet. From my point of view the frame numbers should be taken from the file. However this shows something different.

6. Performance issues

Let's consider the code to read a SIP message:

Tcp tcpHeader = new Tcp();
Udp udpHeader = new Udp();
Sip sipHeader = new Sip();

if ( packet.hasHeader(udpHeader) && packet.hasHeader(sipHeader) ) {

    sipMessage = new String(udpHeader.getPayload());

} else if ( packet.hasHeader(tcpHeader) && packet.hasHeader(sipHeader) ) {

    sipMessage = new String(tcpHeader.getPayload());

}

The methods getPayload() don't return a reference to a payload. They copy data from packet to a byte array. After that String constructor will copy the byte array into itself.

The method getByteArray() is also a copying method. I could not find a way to get the reference to the payload.

I suppose that the methods hasHeader() are also copying methods. They copy header data into newly created object.

To sum up, in order to read SIP message we have to allocate 3 objects in the memory, we have to make complete copy of different headers 3 times just to check if they are TCP, UDP or SIP, and at last we need to make double copy of the payload.

It doesn't look efficient.

PS. Since the JNetPcap doesn't support number of SIP messages, this example is not complete solution. UDP or TCP payload is necessary. It will require more efforts.

7. TCP getPayload() problem.

The method getPayload() of Tcp class works incorrectly. Please look the following code example:

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID,ethernetPacketWithRtpInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();
Rtp rtpHeader = new Rtp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);
packet.getHeader(rtpHeader);

RTP header is not parsed and exception "com.sun.jdi.InvocationException occurred invoking method" happens.
We found the workaround:

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID,ethernetPacketWithRtpInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();
Rtp rtpHeader = new Rtp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);

byte[] payload = tcpHeader.getPayload();

JPacket rtpJPacket = new JMemoryPacket(JProtocol.RTP_ID, payload);
rtpJPacket.getHeader(rtpHeader);

The similar problem happens when you try to get TCP payload with SIP message inside. However it looks a little bit different.

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID, ethernetPacketWithSipInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);

String msg = new String(tcpHeader.getPayload());

msg will contain only last 20 bytes of the payload.

The workaround hasn't been found.

8. Big trace files processing.

I found that big files (in my test I used 200 MBytes file) are not processed by jNetPcap correctly. We used parameters -Dnio.mx=1024mb -Dnio.ms=512mb -Dnio.blocksize=2mb.

The following exception happens:

Exception in thread "pool-3-thread-3" java.nio.BufferUnderflowException
        at org.jnetpcap.nio.JBuffer.check(Unknown Source)
        at org.jnetpcap.nio.JBuffer.getUByte(Unknown Source)
        at org.jnetpcap.protocol.tcpip.Tcp.hlen(Unknown Source)
        at org.jnetpcap.protocol.tcpip.Tcp.decodeHeader(Unknown Source)
        at org.jnetpcap.packet.JHeader.decode(Unknown Source)
        at org.jnetpcap.packet.JPacket.getHeaderByIndex(Unknown Source)
        at org.jnetpcap.packet.JPacket.hasHeader(Unknown Source)
        at org.jnetpcap.packet.JPacket.hasHeader(Unknown Source)


9. DTMF payload type

In the chapter 4 I wrote about incorrect RTP packet identification. There is one more issue with that code. It rejects all payload types greater than 25. There are 127 standard payload types totally. So 102 types are not supported. DTMF uses some of dynamic payload types which are 95-127.

Jain SIP: How to read a SDP content

import javax.sdp.*;
import javax.sip.*;

ContentTypeHeader contentType = (ContentTypeHeader) msg.getHeader(ContentTypeHeader.NAME);
ContentLengthHeader contentLen = (ContentLengthHeader) msg.getHeader(ContentLengthHeader.NAME);

if ( contentLen.getContentLength() > 0 && contentType.getContentSubType().equals("sdp") ){
    String charset = null;
   
    if (contentType != null)
        charset = contentType.getParameter("charset");
    if (charset == null)
        charset = "UTF-8"; // RFC 3261

    //Save the SDP content in a String
    byte[] rawContent = msg.getRawContent();
    String sdpContent = new String(rawContent, charset);

    //Use the static method of SdpFactory to parse the content
    SdpFactory sdpFactory = SdpFactory.getInstance();
    SessionDescription sessionDescription = sdpFactory.createSessionDescription(sdpContent);
    Origin origin = sessionDescription.getOrigin();
   
    System.out.println("A Session ID is " + origin.getSessionId());
} else {
    System.out.println("It is not a SDP content");
}

How to create initialized enum in Java.

One day I needed to create enum initialized to values other than its default values in Java. It can be done easily in C++. For example, look at the code below:

typedef enum ResponseType{
     OK = 200,
     TRYING = 100,
     RINGING = 180
} TResponseType;

However it is not so obvious in Java.

public enum ResponseType {
     OK(200),
     TRYING(100),
     RINGING(180),
     SESSION_PROGRESS(183),
     BAD_REQUEST(400),
     NOT_FOUND(404);

     private int value;

     ResponseType(int value){
           this.value = value;
     }

     public int getValue() {
          return value;
     }
};

Heuristic decode of RTP packets in practice

It is not possible to a distinguish RTP packet reliably without additional analysis of signaling.

Different ways are used to understand if some packet is RTP or not.
I've made investigation of the X company's IP sniffer module and the JNetPcap library.

You can find here methods used in these software. All these methods don't give a 100% exact decision.
You can find RTP header description in specification of RTP protocol.

X company's IP sniffer module

Analysis is done inside method IsValid() of RtpPacket class.
A decision is taken on base check of the three parameters: RTP version, payload type and header length.

• The payload type is checked if it belongs to a valid range of values from 34 (H263) to 96 (DynMin).
• The header length check is a little more complex.
  IPP calculates the RTP header length on base value the CSRC count (field CC) and padding (field P). During this step IPP supposes that a current packet is a RTP packet.
  Then it calculates the data length as a difference between the total packet length and the calculated header length - DataLen = TotalPacketLen - HeaderLen.
  After that it compares calculated data length with total packet length.
  If data length is bigger than total length then it is not RTP.
• RTP version must be 2.

Source code:

bool RtpPacket::IsValid()
{
    //check for correct version
   if (version != 2) return false;
  
   //check for a known payload type
   if (payloadType > rtpPayloadH263 && payloadType < rtpPayloadDynMin) return false;

   if (dataLen >= totalLength) return false;

   return true;
}

 JNetPcap

Analysis is done inside function validate_rtp() of packet_protocol.cpp file (line 143 for version 1.3.b3).
The decision is taken by using more parameters. However I am not sure that probability of the recognition is higher. Let's consider the algorithm.

• It checks the RTP version. The version must be 2 also.
• The CSRC count check is empty. The JNetPcap check only low boundary. It must be greater than 15. But CSRC count is 4 bits field and it can't be more 15.
• The payload check is much simpler. It also checks only low boundary. It must be greater than 25.
• Sequence number is checked for zero.
• SSRC is checked for zero.
• Timestamp is checked for zero.
• All other fields if any are also checked for zero.
• If a packet has extension then extension length is also checked for out of bounds.

JNetPcap source code:

int validate_rtp(scan_t *scan) {

if ((scan->buf_len - scan->offset) < sizeof(rtp_t)) {

return INVALID;

}

register rtp_t *rtp = (rtp_t *)(scan->buf + scan->offset);

if (rtp->rtp_ver != 2 ||

rtp->rtp_cc > 15 ||

rtp->rtp_type > 25 ||

rtp->rtp_seq == 0 ||

rtp->rtp_ts == 0 ||

rtp->rtp_ssrc == 0

) {

TRACE("INVALID header flad");

CALL(debug_rtp(rtp));

EXIT();

return INVALID;

}

uint32_t *c = (uint32_t *)(rtp + 1);

for (int i = 0; i < rtp->rtp_cc; i ++) {

TRACE("CSRC[%d]=0x%x", i, BIG_ENDIAN32(c[i]));

if (BIG_ENDIAN32(c[i]) == 0) {

TRACE("INVALID CSRC entry is 0");

EXIT();

return INVALID;

}

/*

* Check for any duplicates CSRC ids within the table. Normally there

* can't be any duplicates.

*/

for (int j = i + 1; j < rtp->rtp_cc; j ++) {

if (BIG_ENDIAN32(c[i]) == BIG_ENDIAN32(c[j])) {

TRACE("INVALID duplicates CSRC entries");

EXIT();

return INVALID;

}

}

}

if (rtp->rtp_ext) {

rtpx_t * rtpx = (rtpx_t *)(

scan->buf +

scan->offset +

RTP_LENGTH +

(rtp->rtp_cc * 4));

register int xlen = BIG_ENDIAN16(rtpx->rtpx_len) * 4;

if ((!SCAN_IS_FRAGMENT(scan) &&

(scan->offset + xlen > scan->wire_len))    ||

(xlen > 1500) ) {

TRACE("INVALID rtpx_len > %d bytes (wire_len) in extension header",

scan->wire_len);

EXIT();

return INVALID;

}

}

TRACE("OK");

EXIT();

CALL(debug_rtp(rtp));

return RTP_ID;

}

C++/Qt vs Java

From my point of view, this article is useful for to realize possible risks.

Please find extracts from it below. The original article is attached at the end of the page.

Programmer efficiency

"Programmer-efficiency describes how efficiently (i.e. how quickly and accurately) a programmer with a given degree of experience and knowledge can implement a certain set of requirements in a particular programming language, including debugging and project setup time."

"Java has several features designed to make it more programmer-efficient. For example, unlike C++ (or C), the programmer does not have to explicitly "free" (give back) allocated memory resources to the operating system."

"Research shows that in practice, garbage collection and other Java features, do not have a major influence on the programmer-efficiency."

"In conclusion: both research and practice contradict the claim that Java programmers achieve a higher programmer-efficiency than C++ programmers."

Runtime efficiency

"Prechelt provides useful data. The amount of data he provides is huge, but he arrives at the conclusion that "a Java program must be expected to run at least 1.22 times as long as a C/C++ program".

"Note that he says at least; the average runtime of Java programs is even longer. Our own experience shows that Java programs tend to run about 2-3 times as long than their equivalent C/C++ programs for the same task."

Cross platform approaches

"It is the semi-interpreted nature of Java programs that makes the "compile once, run anywhere" approach of Java possible in the first place. Once a Java program is compiled into bytecode, it can be executed on any platform which has a JVM. In practice, this is not always the case, because of implementation differences in different JVMs, and because of the necessity to sometimes use native, non-Java code, usually written in C or C++, together with Java programs."

"But is the use of platform-independent bytecode the right approach for crossplatform applications? With a good cross-platform toolkit like Qt and good compilers on the various platforms, programmers can achieve almost the same by compiling their source code once for each platform: "write once, compile everywhere"."

"The question arises why it should be necessary to run the Java Virtual Machine in software; if a program can be implemented in software, it should also be possible to have hardware implement the same functionality."

Memory efficiency

"Java and C++ take completely different approaches to memory management. In C++, all memory management must be done explicitly by the programmer …. Java automatically de-allocates (frees) unused memory"

"Garbage collection is very convenient, but the trade offs are greater memory consumption and slower runtime speed. …. In addition to the higher memory requirements, the garbage collection process itself requires processing power … Since the garbage collector runs periodically, it can occasionally lead to Java programs "freezing" for a few seconds…..".

"The Qt toolkit takes a more efficient approach to easing the memory management task for its programmers: when an object is deleted, all dependant objects are automatically deleted too. Qt’s approach does not interfere with the programmer’s freedom to delete manually when they wish to."

"To sum up this discussion, we have found C++ to provide much better runtime- and memory-efficiency than Java, while having comparable programmer-efficiency."

Available libraries and tools

"The Java platform includes an impressive number of packages that provide hundreds of classes for all kinds of purposes, including graphical user interfaces, security, networking and other tasks. This is certainly an advantage of the Java platform. For each package available on the Java platform, there is at least one corresponding library for C++."

"However, this strength of Java is also one of its weaknesses. It becomes increasingly difficult for the individual programmer to find their way through the huge APIs."

"If Java has an advantage on the side of available libraries, C++ clearly has an advantage when it comes to available tools"

Conclusion

Both independent academic research and industrial experience demonstrate that the hype favouring Java is mostly unjustified, and that the C++/Qt combination is superior. This is mainly due to the runtime and memory efficiency problems in Java programs … and Java’s failure to deliver increased programmer efficiency. In various programming projects we have been involved in, junior-level programmers learnt Java faster, but more experienced and senior-level programmers (which are usually in charge of the application design and the implementation of the critical parts of an application) achieved better and faster results using C+.

References


Qt vs. Java: A Comparison of Qt and Java for Large- Scale, Industrial-Strength GUI Development

Jain SIP introduction

IM application (IP active) implementation.

There is a good article "An Introduction to the JAIN SIP API"
which can be used as first step guide for development of "IP active" component.

SIP parser application (IP passive) implementation.

However we need to use JainSIP for parsing SIP as well.
This topic is not covered in the article. But it is enough easy.

First, we need to create and initialize SipFactory and MessageFactory instances:

sipFactory = SipFactory.getInstance();
sipFactory.setPathName("gov.nist");
messageFactory = sipFactory.createMessageFactory();

Second, we need to get instance of SIPRequest or SIPResponse and initialize them by input buffer with a real SIP message.

SIPRequest sipRequest = (SIPRequest)messageFactory.createRequest(sipMessage);

How to understand if the income message request or response:

if ( sipMessage.startsWith( SIPConstants.SIP_VERSION_STRING ) ){
processResponse( sipMessage );
}
else {
processRequest( sipMessage );
}

Now we can get any information from the SIP message.

ListIterator<String> itHeaders = sipRequest.getHeaderNames();
String requestMethod = sipRequest.getMethod();

Detailed description of JainSIP API can be found here.

Basic connections and identifiers of SIP protocol

1. Transaction

"A SIP transaction occurs between a client and a server and comprises all messages from the first request sent from the client to the server up to a final (non-1xx) response sent from the server to the client." (RFC3261)

In other words the SIP transaction includes a request and all responses on it.

The  SIP transaction is identified by a branch parameter in the top of  "Via" header field and "CSeq" header field value. These identifiers are showed in the picture below in cyan color.


2. Dialog

"A dialog is a peer-to-peer SIP relationship between two UAs that persists for some time." (RFC3261)

A dialog is identified by a call identifier in "Call-ID"header field, local tag and a remote tag in "To" and "From" header fields. The identifiers are depicted in green color.


3. Session

"A multimedia session is a set of multimedia senders and receivers and the data streams flowing from senders to receivers." (RFC2327)

The session is defined by the concatenation of the SDP user name, session id, network type, address type,  and address elements in the origin field "o=". The session identifiers are showed in yellow color.

One of the best illustrated SIP introductions - Tech-invite.

In the picture below you can see example of a SIP response message.

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.2.102:5060;branch=z9hG4bKSEC-a01a8c0-a01a8c0-1-oW11d3T02H
From: "Test 4100" <sip:4100@192.168.2.102>;tag=snl_951dhI11AG
To: <sip:49602150014101@192.168.2.70>;tag=4138389295

Call-ID: SEC11-a01a8c0-a01a8c0-1-D9KQeE4Yc8gu

CSeq: 1235 INVITE

Allow:  INVITE,ACK,CANCEL,BYE,REFER,NOTIFY,MESSAGE,UPDATE
Allow-Events: talk, hold
Contact: 49602150014101 <sip:49602150014101@192.168.2.70:5060;transport=udp>
Server: OpenStage_20_V2 R0.30.2      SIP  100120
Supported: replaces, timer
X-Siemens-Call-Type: ST-insecure
Content-Type: application/sdp
Content-Length: 265

v=0

o=MxSIP 0 814456601 IN IP4 192.168.2.70

s=SIP Call
c=IN IP4 192.168.2.70
t=0 0
m=audio 5062 RTP/AVP 8 0 101
a=rtpmap:8 PCMA/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=silenceSupp:off - - - -
a=fmtp:101 0-15
a=ptime:20
a=sendrecv

Simple arrays in Lua language

Arrays in Lua are based on Lua's tables.

Declaration of an array:

sample_array = {}

Addition of a new element into the sample_array:

table.insert( sample_array, some_var )

How to get array size:

array_size = #sample_array

Access to the array elements ( the first element has index 1 ):

for i=1, #sample_array do
   message( sample_array[i] )
end -- for

Lua 5.1 Reference Manual

Plugin development for Wireshark

There are two types of the wireshark plugins: dissectors and taps.

The dissectors are used for specific protocol packets parsing. It is called each time when packet data should be displayed in a packet list window or in a packet details window. Documentation about dissectors development.

The tap plugins are used to collect any statistic about some protocol. It is called once for each packet. Documentation about tap plugins development.

The best guide is located on wireshark website.

Both types of the plugin can be written in C or Lua languages. All information about the Lua plugin development with examples is here.

To make and build the wireshark plugin on Windows platform it is necessary to perform some procedure. The best description is given here.

I found enough interesting article about wireshark plugin development. It gives some additional information. There is a plugin example. Although there are many good examples in wireshark/plugins directory.

I used the articles mentioned above for own plugin development. However I met some issues and I would like to describe them (Windows platform).

Tip 1. None of the sample plugins work with the official wireshark

After successful build, plugin is not loaded by wireshark. It reports about plugin loading problem. See a picture below. But it is possible to use this plugin with the wireshark built by yourself.

To solve the issue it's necessary:

1. Open the file "config.nmake" in your developer wireshark folder, e.g. C:\wireshark\config.nmake
2. Search for "DLL_LDFLAGS = /MANIFEST:no" and comment this line out (using "#")
3. Compile your plugin again, a manifest file will be created in your plugin folder named "pluginName.dll.manifest"
4. This manifest has to be included by your dll. You can either do this by typing the following command to the shell or add it somewhere in the file "makefile.nmake" (i.e. under "all:", so it is always included when "nmake... all" is called):
   mt.exe -manifest pluginName.dll.manifest -outputresource:pluginName.dll;2

Tip 2. How to debug.

Of course it is possible to use WinDbg from MS SDK. But the most convenient way is using MS Visual Studio. I just opened a source file and attached to the wireshark process. Don't forget to specify a path to debug symbol information (pdb-file). You can do this in Tools/Options/Debugging/Symbols menu.


Tip 3. Setup environment vars

I used following batch file:

@echo off
echo Adding things to the path...
set PATH=%PATH%;.
set PATH=%PATH%;d:\cygwin\bin
set HTTP_PROXY=http://proxy.yourcompany.ru:1080//

echo Setting up Visual Studio environment...
call "d:\Program Files\Microsoft Visual Studio 8\VC\bin\vcvars32.bat"

How to open and setup a run configuration for an Equinox plugin project

This note is valid for Eclipse Helios.


There is no "Open project" menu in Eclipse. To open an existing project use command File/Import.

In order to setup the Run configuration you need:

1. Press the "Run configuration" drop down list button on a tool bar (see mark 1 in a picture below).
2. Select the "Run configurations..." command.
3. Deselect all items in a bundle list by pressing the "Deselect All" button (see mark 2).
4. Select your own bundles.
5. Press the "All Required Bundles" button to add into selection only necessary bundles (see mark 3).
6. Finally check the "Only show selected" checkbox to filter out only selected items in the bundle list. (see mark 4)

Equinox tips and tricks

How to start Equinox from command line.

Equinox framework can be started from Windows command line by following command:

java -jar org.eclipse.osgi_3.5.2xxx.jar -console -clean

The framework file can be found in eclipse/plugins directory. A specific version will be placed instead "xxx" characters.

How to clean Equinox cache.

The Equinox framework keeps first started bundles in its cache. That is why your updates are not applied. To clean the cache you need to use a "-clean" parameter. For Windows command line please see the example in a previous clause. In a picture below you can see how to use this parameter inside Eclipse IDE. After this action you need to restart the Equinox framework.

How to get complete set of tools for plugin development

Plugin development is integrated into Eclipse JEE IDE. Don't try to use Eclipse SE.

Newest version of Eclipse

It is highly recommended to use latest version of Eclipse Helios due to enormous number of issues with OSGi.

What is a bundle URL?

The bundle URL is a path which identifies the bundle. It can look like

file:\Full_Path_To_The_Bundle

For example:

file:\C:\Java\bundle_1.0.0.jar

How to resolve a native library call in bundles?

All used native libraries should be listed in a manifest file. See the example below.

Bundle-NativeCode: lib/jnetpcap.dll;
osname = WindowsXP;
osversion = 5.1;
processor = x86

Please read the OSGi core specification (chapter 3.9) for more information.

OSGi reference framework comparision

Brief information

Felix:
Developer:                         Apache Software Foundation
License:                             Apache license 2 (FLOSS)
OSGi version:                    R4
IDE integration:                  Eclipse, NetBeans
Framework jar-file size:     400Kb

Knopflerfish / KnopflerfishPro:
Developer:                         Makewave
License:                             BSD style license (version Pro is commercial)
OSGi version:                    R4
IDE integration:                  Eclipse
Framework jar-file size:     400Kb

Eqiunox:
Developer:                         Eclipse
License:                             Eclipse Public License
OSGi version:                    R4
IDE integration:                  Eclipse
Framework jar-file size:     1200Kb

FUSE ESB 4:
It is built on Felix.
Developer:                         Progress Software Corporation
License:                             Apache license 2 (FLOSS)
OSGi version:                    R4
IDE integration:                  Eclipse
Framework jar-file size:    

It is based on Apache ServiceMix4 which uses Apache Felix.

"Enterprise IT organizations have stringent requirements for infrastructure software, and the FuseSource team addresses these needs to bridge the gap between open source projects and enterprise-class, production-ready software" 

Karaf:
It uses Felix or Equinox.
Developer:                         Apache Software Foundation
License:                             Apache license 2 (FLOSS)
OSGi version:                    R4
IDE integration:                 ?
Framework jar-file size:

Karaf can use either Apache Felix Framework or Eclipse Equinox.

OSGi services support

Service Specification	Felix	Knopflerfish/pro	Equinox
Core Specification
Framework Specification (all layers)	ok	ok	ok
Framework Launching	ok	ok	ok
7 Package Admin Service	ok	ok	ok
8 Start Level Service	ok	ok	ok
9 Conditional Permission Admin	ok	ok	ok
10 Permission Admin Service	ok	ok	ok
11 URL Handlers Service	ok	ok	ok
12 Service Hooks	ok	ok	ok
Compendium Specification
101 Log Service	ok	ok	ok
102 Http Service	ok	ok	ok
103 Device Access	no	ok	ok
104 Configuration Admin Service	ok	ok	ok
105 Metatype Service	ok	ok	ok
106 Preferences Service	ok	ok	ok
107 User Admin Service	no	ok	ok
108 Wire Admin Service	no	ok	ok
109 IO Connector Service	no	ok	ok
110 Initial Provisioning	no	ok	ok
111 UPnP™ Device Service	ok	ok	no
112 Declarative Services	ok	ok	ok
113 Event Admin Service	ok	ok	ok
114 Deployment Admin	no	no/ ok	no
115 Auto Configuration	no	no/ ok	no
116 Application Admin	no	no/ ok	no
117 DMT Admin Service	no	no/ ok	no
119 Monitor Admin Service	no	no/ ok	no
120 Foreign Application Access	no	no/ ok	no
121 Blueprint Container	no	no	no
701 Tracker Specification	no	ok	no
702 XML Parser Service Specification	no	ok	no
703 Position Specification	no	ok	no
704 Measurement and State	no	ok	no
999 Execution Environment
Enterprise Specification
13 Remote Services	no	no	no
122 Remote Service Admin Service	no	no	no
123 JTA Transaction Services	no	no	no
124 JMX™ Management Model	no	no	no
125 JDBC™ Service	no	no	no
126 JNDI Services	no	no	no
127 JPA Service	no	no	no
128 Web Applications	no	no	no
129 SCA Configuration Type	no	no	no

I couldn't find application benchmark tests for the OSGi frameworks. The OSGi Alliance publishes test cases to test compliance, not performance. The Knopflerfish has a regression test suite. It is located here