Perl long paths issue in Windows environment

Windows allows to create file paths greater than Windows MAX_PATH value (260 characters). Perl native functions cannot work correctly with such files. For example, function rmtree() is not able to delete such files.

There is a module which solves the issue in Perl - Win32-LongPath. It provides functions to access long paths and Unicode in the Windows environment.

It could be easily installed for Perl 5.18. You need just search module with the Win32-LongPath name inside Perl package manager. If it is necessary to install the module for older versions of Perl, e.g. 5.8, you need to use the following command:

ppm install http://www.bribes.org/perl/ppm/Win32-LongPath.ppd

Below example of  implementation rmtree using Win32-LongPath functions. Function rmtreeWin() receives full file name as a parameter, identify whether it directory of file and tries to delete it. Note that Wind32-LongPath functions rmdirL and unlinkL cannot delete read-only files/dirs, so we need to change its attributes first.

use Win32::LongPath;

sub rmtreeWin() {
    chmod 0777, $_[0];

    if (-d) {
        rmdirL $_[0] or die "unable to remove directory $_[0] ($^E)";
    }
    else {
        unlinkL $_[0] or die "unable to delete file $_[0] ($^E)";
    }
}

How to read byte buffer in Java

In telecommunications we need to read and analyze raw byte buffers frequently because this is usual representation of a network packet. However there is an issue with the Java language which does not have unsigned integer types. All integer types in Java are signed. So it is not possible just to read a byte and then analyze it.

For example, if we have byte with value 0x96 we will get -106 instead 150. Why does this happen and what to do?

Negative numbers are stored in the memory in two's complement system. According to this approach we need to make the following conversation to write negative number into the memory:
Step 1. invert all bits
Step 2. add 1

For example, let's try to convert a number -6 to the two's complement system representation.
The binary representation of the number 6 is 00000110.
Step 1. After invert conversion we will have 11111001.
Step 2. 11111001 + 1 = 11111010
So the negative number -6 will look in the two's complement system as 11111010. This number corresponds to unsigned byte 250.

Now let's see what Java makes with our number 0x96 or 150. Since there is no unsigned types in Java, this number is considered as number with a sign. Maximum value for signed byte is 127. So Java makes reverse conversion.
The binary representation of the number 0x96 is 10010110.
Step 1. subtract 1:  10010110-1=10010101
Step 2. invert all bits: ~10010101=01101010
01101010 is -106.
That is why Java considers the byte 0x96 as the negative number -106.

How to fix this Java feature? We need to make two things:
1. Convert a byte into int because the maximum value of a signed byte is only 127. At this step Java will fill the high-order bits with 1 to keep the number negative.
2. Then we need to zero high-order bits to get the original value.

The following code does this:

byte b = (byte) 0x96;
int i = (int) b & 0xFF; 

How to build jNetPcap library

Access to the source code of jNetPcap
 
jNetPcap source code is located under SVN control.
To access the repository you can use TortoiseSVN client.
Then you can get the source code using following link:https://jnetpcap.svn.sourceforge.net/svnroot/jnetpcap/jnetpcap/trunk

Preparation

The library should be linked statically with the libraries LIBGCC_S_DW2-1.DLL and LIBSTDC++-6.DLL which is located in bin folder of MinGW. The default build.xml doesn't contain necessary options. So before to start build procedure you need to update the build.xml file.

Both sections with compilation options for 32 and 64 bits platforms should be updated.

Find section "comp-jni-win32" and "comp-jni-win64" and update the section <exec executable...> inside both of them. The following lines should be added there "<arg value="-static" />"
So in result section <exec executable..."> should look like this code:

 <exec executable="${compiler.ld.cmd}">
   <arg value="-shared" />
   <arg value="-g" />
   <arg value="-Wl,--add-stdcall-alias" />
   <arg value="-static" />
   <arg line="-o ${jni.build.filepath}" />
   <arg line="${jni.obj.files}" />
   <arg line="-L ${pcap.lib.dir}" />
   <arg line="-l ${pcap.lib.name}" />
   <arg line="-l ${ms.lib.name}" />
</exec>

How to build jNetPcap library for 32-bits platform
Download and install Apache Ant building software.
It could be taken from the official site. To install it you need just open zip archive and extract the files into any directory. Then you need to create Environment variable ANT_HOME with the path to the ant directory. It is also necessary to add the path to ant bins into the PATH environment variable.

Download and install compiler MinGW.
It could be taken from the official site. During the setup you need to select "Download latest repository catalogues". It is important at least for Windows XP. Then you need to update the PATH environment variable again and add the path to MinGW bins there.

Build the jNetPcap library
Go into the jnetpcap directory with the source code and run the following command to get the complete list of possible targets:

ant help

You need to compile jnetpcap.jar and jnetpcap.dll files. Execute the following command one by one.

ant comp-java

ant build-jar

ant comp-jni

 
How to build jNetPcap library for 64-bits platform

Download and install Apache Ant building software.
It could be taken from the official site. To install it you need just open zip archive and extract the files into any directory. Then you need to create Environment variable ANT_HOME with the path to the ant directory. It is also necessary to add the path to ant bins into the PATH environment variable.

Download and install compiler MinGW64.
It could be taken from the official site. But there is no binaries ready to execute. You can find there only source code and you will need to compile the compiler yourself. The compilation procedure is enough complicated.
So it would be much faster to get already build binaries from another site. You can use Windows XP/2003 x64 binaries for Windows 7. It is ok.

Build the jNetPcap library
Go into the jnetpcap directory with the source code and run the following command to get the complete list of possible targets:
ant help

You need to compile jnetpcap.jar and jnetpcap.dll files. Execute the following command one by one.
ant comp-java
ant build-jar
ant comp-jni

Performance analysis of jNetPcap library

We are working on creation of VoIP sniffer application which must to capture RTP packets and SIP messages. We had to make the performance analysis to find the quickest way to free internal jNetPcap ring buffer and avoid network packets loss. Our application is used on multi core PCs, e.g. Xeon E5630.

For analysis of the performance jVisualVM profiler 1.3.3. was used. For the test I used trace pcap file with 262000 network packets including 1500 RTP packets and some small number of SIP messages. So useful traffic was just 1%.

Our first implementation of a capture thread was like following:

public void nextPacket(final PcapPacket packet, final String unused) {
   
    NetworkPacket outputPacket = null;
    boolean tcp = false;

    try {
        if ((packet.hasHeader(updHeader) || (tcp=packet.hasHeader(tcpHeader))) && !packet.hasHeader(icmpHeader)) {
           
            if (packet.hasHeader(rtpHeader)) {
                if ( packet.hasHeader(ipHeader) ) {
                    outputPacket = new RtpPacket(ipHeader, updHeader, rtpHeader);
                }
            } else if (packet.hasHeader(sipHeader)) {
                if ( packet.hasHeader(ipHeader) ) {
                    if (tcp) {
                        outputPacket = new SipPacket(ipHeader, tcpHeader, sipHeader);
                    } else {
                        outputPacket = new SipPacket(ipHeader, updHeader, sipHeader);
                    }
                }
            }

        if (outputPacket != null) {
            if (!packetProcessingQueue.offer(outputPacket)) {
                logger.error("packetProcessingQueue is full when packet [{}] is received", packet.getFrameNumber());
            }
        }

    } catch (ExceptionBadPacket e) {
        logger.info("Incorrect packet has been received from JNetPacket", packet.getFrameNumber());
    }
}

The RtpPacket and SipPacket constructors make copy of the network packets payload.
The profiler showed following results.

After that I tried to replace hasHeader() by getHeader() function call. It is less convenient because an user have to compare with null. However this function enabled us to increase the speed twice.

I was sure that the filtering enabled us to get higher speed because we skipped unnecessary packets and copied only useful RTP and SIP packets. However the results was absolutely unexpected for me. jNetPcap library copies network packets faster than checks its type. So most efficient way is to capture ALL network packets into the second thread and make analysis there.

The source code for the capture thread looks like:

public void nextPacket(PcapHeader header, JBuffer buffer, String any) {
   
    PcapPacket packet = new PcapPacket(header,buffer);
   
    if (packet != null) {
        if (!packetProcessingQueue.offer(packet)) {
            logger.error("packetProcessingQueue is full when packet [{}] is received", packet.getFrameNumber());
        }
    }
}

Thread states in VisualVM profiler

VisualVM uses following states to indicate thread activity status:

• Running; 
• Sleeping; 
• Wait; 
• Monitor. 

 

What do these states mean?

Running.

Thread.State = RUNNABLE.

"Thread state for a runnable thread. A thread in the runnable state is executing in the Java virtual machine but it may be waiting for other resources from the operating system such as processor."

Sleeping

Thread.State = TIMED_WAITING.

 Thread state for a waiting thread with a specified waiting time. A thread is in the timed waiting state due to calling one of the following methods with a specified positive waiting time:

• Thread.sleep
• Object.wait with timeout
• Thread.join with timeout
• LockSupport.parkNanos
• LockSupport.parkUntil

Wait


Thread.State = WAITING.

Thread state for a waiting thread. A thread is in the waiting state due to calling one of the following methods:

• Object.wait with no timeout
• Thread.join with no timeout
• LockSupport.park

A thread in the waiting state is waiting for another thread to perform a particular action. For example, a thread that has called Object.wait() on an object is waiting for another thread to call Object.notify() or Object.notifyAll() on that object. A thread that has called Thread.join() is waiting for a specified thread to terminate.

Monitor

Thread.State = BLOCKED.

Thread state for a thread blocked waiting for a monitor lock. A thread in the blocked state is waiting for a monitor lock to enter a synchronized block/method or reenter a synchronized block/method after calling Object.wait.

SIP protocol parsing: How to match call legs.

One of the main tasks for any protocol parser is how to find parts of the same call (call legs).

Any call is established through PBX and looks like it is showed in the picture 1 or 2.

Subscriber A calls to subscriber B. In the beginning a connection from Party A to PBX is established. Then PBX establishes the connection to Party B. From SIP protocol point of view two different calls are established with different Call-Id 12 and 23. After that RTP streams are transmitted from Party A to Party B and vice versa. There are two possible routes. The RTP streams could be passed through the PBX or directly between the participants.
 
You can see that one conversation consists of two SIP calls with different Call-Id. One call is established to PBX and another call is established from PBX. These calls are usually called "call legs".

Lets look into one real trace of a basic call scenario.

In the picture above PBX has 192.168.2.102 address.

Two calls with different call ids were established. They depicted by wireshark tool in different colors. Lets look into details of the calls.

The first call has Call-Id="***b3b1b". 
The calling party has extension "49602150014015" and media IP address 192.168.2.63.
The called party has extension  "4100" and media IP address 192.168.2.59.

 

The second call has Call-Id="***JPwcK6".
The calling party has extension "49602150014015" and media IP address 192.168.2.63.
The called party has extension  "49602150014100" and media IP address 192.168.2.59.

You can see that the PBX substitutes short phone extension for long extension and vice-versa. That is why it is not possible to use phone extensions to match call legs. On the contrary media IP addresses are constant.

So SIP call legs can be matched only by using media IP addresses.

Pitfalls of the JNetPcap library

During some software development we have found the following problems with using the JNetPcap library (1.3.0). Points 1 and 4 are still valid for the version 1.4.r1300 (I didn't check the other points). I would like to share our experience.


1. SIP messages support.

In order to check if some packet contains SIP message, we use the following code:

public void nextPacket(PcapPacket packet, String arg1) {
    Sip sipHeader = new Sip();
    if (packet.hasHeader(sipHeader)) {
        doSomething();
    }
}

However JNetPcap doesn't recognize all SIP messages. For example, the messages NOTIFY (RFC3265), SUBSCRIBE (RFC3265) and  REFER (RFC3515) will not be parsed as a SIP message and hasHeader(sipHeader) call will return false. Probably the messages INFO (RFC2976), UNSUBSCRIBE (RFC3265) and some other methods are not also supported by the library.

Update: Version 1.4.r1300 doesn't still support these messages.

2. SIP messages payload

To get a SIP message we used method getPayload() of Sip class. However, sometimes the method returned null for good SIP message. We realised that it is necessary to use Udp or Tcp class to have reliable access to the payload. Following code shows the work around solution:

Udp udpHeader = new Udp();
Tcp tcpHeader = new Tcp();

if ( packet.hasHeader(udpHeader) ) {
    sipMessage = new String(udpHeader.getPayload());
} else if ( packet.hasHeader(tcpHeader) ) {
    sipMessage = new String(tcpHeader.getPayload());
}

3. RTP packets with G.729 payload are not supported

The following code caused the exception "ArrayIndexOutOfBoundsException" when we tried to execute it against RTP packet with G.729 payload inside:

if ( packet.hasHeader(rtpHeader)) {
    payloadType = rtpHeader.typeEnum();
}

Complete exception trace:

Exception in thread "pool-1-thread-2" java.lang.ArrayIndexOutOfBoundsException: 18
    at org.jnetpcap.protocol.voip.Rtp$PayloadType.valueOf(Unknown Source)
    at org.jnetpcap.protocol.voip.Rtp.typeEnum(Unknown Source)
    ...
    at org.jnetpcap.Pcap.loop(Native Method)

The author of JNetPcap promised us to fix this issue in the next release 1.4.0.

4. Incorrect validation of RTP packets

Some time ago I wrote about how to determine if some packet is RTP or not. In that article I showed example from JNetPcap library. It considers RTP packets with zero sequence number and zero timestamp as invalid RTP packets. In practice we received RTP packet with sequence number = 0 and this packet was rejected by JNetPcap. According to the specification RFC3550 it is correct situation. I couldn't find restriction to have zero sequence number and timestamp.

Update: It is still valid for the version 1.4.r1300.

5. Pcap file reopen issue

When some file is opened, parsed, closed and opened again, the frame numbers are not initialized from the zero. The method getFrameNumber() of PcapPacket class returns the next number after the last one. For example, my pcap file contains 20 packets. After reopening it the method getFrameNumber() will return 21 for the first packet. From my point of view the frame numbers should be taken from the file. However this shows something different.

6. Performance issues

Let's consider the code to read a SIP message:

Tcp tcpHeader = new Tcp();
Udp udpHeader = new Udp();
Sip sipHeader = new Sip();

if ( packet.hasHeader(udpHeader) && packet.hasHeader(sipHeader) ) {

    sipMessage = new String(udpHeader.getPayload());

} else if ( packet.hasHeader(tcpHeader) && packet.hasHeader(sipHeader) ) {

    sipMessage = new String(tcpHeader.getPayload());

}

The methods getPayload() don't return a reference to a payload. They copy data from packet to a byte array. After that String constructor will copy the byte array into itself.

The method getByteArray() is also a copying method. I could not find a way to get the reference to the payload.

I suppose that the methods hasHeader() are also copying methods. They copy header data into newly created object.

To sum up, in order to read SIP message we have to allocate 3 objects in the memory, we have to make complete copy of different headers 3 times just to check if they are TCP, UDP or SIP, and at last we need to make double copy of the payload.

It doesn't look efficient.

PS. Since the JNetPcap doesn't support number of SIP messages, this example is not complete solution. UDP or TCP payload is necessary. It will require more efforts.

7. TCP getPayload() problem.

The method getPayload() of Tcp class works incorrectly. Please look the following code example:

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID,ethernetPacketWithRtpInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();
Rtp rtpHeader = new Rtp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);
packet.getHeader(rtpHeader);

RTP header is not parsed and exception "com.sun.jdi.InvocationException occurred invoking method" happens.
We found the workaround:

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID,ethernetPacketWithRtpInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();
Rtp rtpHeader = new Rtp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);

byte[] payload = tcpHeader.getPayload();

JPacket rtpJPacket = new JMemoryPacket(JProtocol.RTP_ID, payload);
rtpJPacket.getHeader(rtpHeader);

The similar problem happens when you try to get TCP payload with SIP message inside. However it looks a little bit different.

JPacket packet = new JMemoryPacket(JProtocol.ETHERNET_ID, ethernetPacketWithSipInTcp);

Ip4 ipHeader = new Ip4();
Tcp tcpHeader = new Tcp();

packet.getHeader(ipHeader);
packet.getHeader(tcpHeader);

String msg = new String(tcpHeader.getPayload());

msg will contain only last 20 bytes of the payload.

The workaround hasn't been found.

8. Big trace files processing.

I found that big files (in my test I used 200 MBytes file) are not processed by jNetPcap correctly. We used parameters -Dnio.mx=1024mb -Dnio.ms=512mb -Dnio.blocksize=2mb.

The following exception happens:

Exception in thread "pool-3-thread-3" java.nio.BufferUnderflowException
        at org.jnetpcap.nio.JBuffer.check(Unknown Source)
        at org.jnetpcap.nio.JBuffer.getUByte(Unknown Source)
        at org.jnetpcap.protocol.tcpip.Tcp.hlen(Unknown Source)
        at org.jnetpcap.protocol.tcpip.Tcp.decodeHeader(Unknown Source)
        at org.jnetpcap.packet.JHeader.decode(Unknown Source)
        at org.jnetpcap.packet.JPacket.getHeaderByIndex(Unknown Source)
        at org.jnetpcap.packet.JPacket.hasHeader(Unknown Source)
        at org.jnetpcap.packet.JPacket.hasHeader(Unknown Source)


9. DTMF payload type

In the chapter 4 I wrote about incorrect RTP packet identification. There is one more issue with that code. It rejects all payload types greater than 25. There are 127 standard payload types totally. So 102 types are not supported. DTMF uses some of dynamic payload types which are 95-127.