Archive for the ‘linux’ Category

Both ffmpeg and vlc are two of the important applications in free software media processing. I wanted to compile both to incorporate latest codecs, x264 and libvpx for video along with aac and libopus for audio. This is what I have done on Debian unstable.

For my purpose, I was compiling these codecs from source. To use the development libraries and headers given by Debian, one has to install the following packages, sudo is used to denote usage of root

sudo apt-get install libfaad-dev libfaac-dev libx264-dev libvpx-dev libopus-dev

Instead of using the development pacakges provides by Debian, I was compiling and installing these libraries to /opt. To use proper package config paths for both ffmpeg and vlc, one has to do export PKG_CONFIG_PATH or alternatively supply then to the configure script.

export PKG_CONFIG_PATH=/opt/lib/pkgconfig

Downloading the corresponding source code

# faac
wget -c http://downloads.sourceforge.net/faac/faac-1.28.tar.gz
# faad
wget -c http://downloads.sourceforge.net/faac/faad2-2.7.tar.gz
# x264
git clone git://git.videolan.org/x264.git
# libvpx
git clone https://chromium.googlesource.com/webm/libvpx
# opus
git clone http://git.xiph.org/opus.git
# ffmpeg
git clone git://source.ffmpeg.org/ffmpeg.git
# vlc
git clone git://git.videolan.org/vlc.git

As I mentioned above, I was using /opt as the prefix. These are the compilation commands used. Replace the number of CPU cores with “-jN” option of make.

# faac
./configure --prefix=/opt
# comment out line 126 of faac-1.28/common/mp4v2/mpeg4ip.h
# strcasestr in case of any compilation error
make -jN
sudo make install
# faad
./configure --prefix=/opt
make -jN
sudo make install
# x264
sudo apt-get install yasm # install dependencies
./configure --prefix=/opt --enable-shared
make -jN
sudo make install
# libvpx
sudo apt-get install yasm #install dependencies
./configure --prefix=/opt --enable-vp8 --enable-vp9 --enable-shared
make -jN
sudo make install
# libopus
./configure --prefix=/opt
make -jN
sudo make install


# install these extra package dependencies:
sudo apt-get install libopenjpeg-dev libvorbis-dev libschroedinger-dev libvpx-dev libopus-dev
./configure --prefix=/opt --extra-cflags='-Wall -g -I/opt/include/' --extra-ldflags='-L/opt/lib/' --enable-avfilter --enable-libschroedinger --enable-libvpx --enable-libspeex --enable-libtheora --enable-libopus --enable-libvorbis --enable-pthreads --enable-zlib --disable-stripping --enable-runtime-cpudetect --enable-libfaac --enable-libx264 --enable-postproc --enable-swscale --enable-x11grab --enable-libopenjpeg --enable-shared --enable-gpl --enable-nonfree --disable-static --disable-stripping --disable-altivec --disable-armv5te --disable-armv6 --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages
make -jN
sudo make install


# package dependencies:
apt-get install libtool gettext cvs libfaad-dev libfaac-dev libswscale-dev libpostproc-dev libqt4-dev libgcrypt20-dev libdbus-1-dev lua5.2 liblua5.2-dev libproxy-dev libdvbpsi-dev liblivemedia-dev libnotify-dev libshout3-dev libdca-dev libflac-dev libmpeg2-4-dev libmpeg3-dev libsdl2-dev libgnutls28-dev libavc1394-dev libsmbclient-dev libtwolame-dev liba52-0.7.4-dev libfribidi-dev libjack-dev libmad0-dev
# a few more extra dependencies:
apt-get install libcdio-dev libvcdinfo-dev libpulse-dev libxml2-dev libmodplug-dev libnotify-dev libcaca-dev libncursesw5-dev libxpm-dev libxcb-shm0-dev libxcb-xv0-dev libxcb-keysyms1-dev libxcb-randr0-dev libx11-xcb-dev libxcb-composite0-dev
./configure --prefix=/opt --disable-update-check --disable-gnome --disable-gtk --disable-familiar --disable-fb --enable-sdl --enable-esd --enable-mad --enable-arts --disable-jack --enable-pulse --enable-lirc --enable-a52 --enable-aa --enable-dvbpsi --without-dv-raw1394 --disable-dc1394 --disable-kde --enable-mp4 --enable-dvb --disable-satellite --enable-ogg --enable-vorbis --enable-shout --enable-qt4 --disable-slp --enable-flac --disable-skins --disable-basic-skins --enable-skins2 --enable-freetype --enable-mkv --enable-speex --enable-caca --enable-live555 --enable-libmpeg2 --enable-fribidi --enable-cdio --enable-mod --enable-theora --enable-modplug --enable-gnutls --enable-ffmpeg --enable-ncurses --enable-smb --disable-gnomevfs --enable-bonjour --enable-mpc --enable-vcd --enable-vcdx --enable-twolame --enable-x264 --enable-faad -enable-faac --enable-vpx --enable-opus --enable-xcb --disable-zvbi --enable-telx --enable-mediacontrol-bindings --disable-atmo --enable-taglib --enable-libass --enable-libdca --enable-alsa --disable-dv --disable-dvdnav --disable-notify --enable-v4l --enable-v4l2 --enable-pvr --enable-dvd --without-dvdcss CPPFLAGS='-I/opt/include' CFLAGS='-g -O2 -I/opt/include' LDFLAGS='-L/opt/lib -Wl,--as-needed' CXXFLAGS='-g -O2' PKG_CONFIG_PATH='/opt/lib/pkgconfig:/usr/lib/pkgconfig'
make -jN
sudo make install

Read Full Post »

Recently I bought an Airtel 3G connection using Huawei E1731Bu-1 USB modem. As usual, I wanted to get it working in Debian GNU/Linux amd64 version. As a first step, installed the following packages using apt-get.

sudo apt-get install ppp pppconfig wvdial

Normally this modem is detected properly, but I was curious to know the extra modules loaded when it is connected. So, I tried lsmod before and after connecting the device and found the following extra modules.

cdc_ether, mii, option, usbnet, usbserial, usb_storage, usb_wwan


First, I tried to configure wvdial since that seems to be the easiest thing to try. Google gave me reference to PJP’s website.


He was kind enough to answer my questions also.

Pon and Poff

WvDial is great. But, I wanted to try Pon and Poff also. They offer a few advantages over wvdial. As an “original” command, many distributions support them out of the box where as wvdial has to be installed separately. There used be a lot of problems in using WvDial on ARM based systems because of getcontext/setcontext support on ARM.

Already I experienced that while using EC1261 Tata Photon, posted the following on Debian forum.



For my Debian desktop, I used pppconfig with the following configuration. But, it did not create a proper /etc/chatscripts/airtel. So, I had to update it to the one given at bottom.

Change 98xxxxxxxx to the mobile number associated with the modem.

pppconfig options

Create Create a connection

Provider Name: airtel
Configure Nameservers (DNS): None
Authentication Method for airtel: Peer Authentication Protocol
User Name: 98xxxxxxxx
Password: 98xxxxxxxx
Speed: 460800
Pulse or Tone: Tone
Phone Number: *99#
Choose Modem Config Method: No
Manually Select Modem Port: /dev/ttyUSB0

Finished Write files and return to main menu.

For the Debian laptop, instead of using pppconfig, I just copied the following generated files from desktop, set the ownership and permissions properly.

cp /path/to/chatscripts/airtel /etc/chatscripts/airtel
chown root:dip /etc/chatscripts/airtel
chmod 640 /etc/chatscripts/airtel

cp /path/to/ppp/pap-secrets /etc/ppp
chown root:root /etc/ppp/pap-secrets
chmod 600 /etc/ppp/pap-secrets

cp /path/to/ppp/peers/airtel /etc/ppp/peers/airtel
chown root:dip /etc/ppp/peers/airtel
chmod 640 /etc/ppp/peers/airtel

That is all. You could use pon/poff to connect and disconnect to the network.

pon airtel
poff airtel

MobilePartner Application

Surprisingly, I could get Airtel application to work. Note that we absolutely do not require this to connect to the network. But, it is good to have it, since it is easy to check 3G Balance using *123*11#. It is also helpful while sending and receiving SMS.

There is a CDROM device associated with the modem. It could be found using the “dmesg” command.

sr1: scsi-1 drive
sr 12:0:0:0: Attached scsi CD-ROM sr1

That means the device is /dev/sr1. Just mount it and install it.

sudo mount /dev/sr1 /media/cdrom
cp -fr /media/cdrom/Linux /tmp
cd /tmp/Linux
sudo ./install

You may find a few errors because it is trying to install “NDIS” driver, but it could be ignored since we are not planning to use NDIS driver to connect to the network. By default it is installed in /usr/local/airtel directory. Start the “/usr/local/airtel/MobilePartner” application.


These are the Configurations I used.


cat /etc/wvdial.conf
[Dialer Defaults]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT+CGDCONT=1,"IP","airtelgprs.com"
Modem Type = Analog Modem
Phone = *99#
Username = 98xxxxxxxx
Password = 98xxxxxxxx
New PPPD = yes
Modem = /dev/ttyUSB0
Baud = 460800
Stupid Mode = 1


cat /etc/chatscripts/airtel
# This chatfile was generated by pppconfig 2.3.18.
# Please do not delete any of the comments. Pppconfig needs them.
# ispauth PAP
# abortstring


# modeminit
'' ATZ
OK 'ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0'
OK 'AT+CGDCONT=1,"IP","airtelgprs.com"'

# ispnumber
# ispconnect
# prelogin

# ispname
# isppassword
# postlogin

# end of pppconfig stuff


cat /etc/ppp/pap-secrets
# /etc/ppp/pap-secrets
# This is a pap-secrets file to be used with the AUTO_PPP function of
# mgetty. mgetty-0.99 is preconfigured to startup pppd with the login option
# which will cause pppd to consult /etc/passwd (and /etc/shadow in turn)
# after a user has passed this file. Don't be disturbed therefore by the fact
# that this file defines logins with any password for users. /etc/passwd
# (again, /etc/shadow, too) will catch passwd mismatches.
# This file should block ALL users that should not be able to do AUTO_PPP.
# AUTO_PPP bypasses the usual login program so it's necessary to list all
# system userids with regular passwords here.
# ATTENTION: The definitions here can allow users to login without a
# password if you don't use the login option of pppd! The mgetty Debian
# package already provides this option; make sure you don't change that.

# INBOUND connections

# Every regular user can use PPP and has to use passwords from /etc/passwd
* hostname "" *

# UserIDs that cannot use PPP at all. Check your /etc/passwd and add any
# other accounts that should not be able to use pppd!
guest hostname "*" -
master hostname "*" -
root hostname "*" -
support hostname "*" -
stats hostname "*" -

# OUTBOUND connections

# Here you should add your userid password to connect to your providers via
# PAP. The * means that the password is to be used for ANY host you connect
# to. Thus you do not have to worry about the foreign machine name. Just
# replace password with your password.
# If you have different providers with different passwords then you better
# remove the following line.

# * password

"98xxxxxxxx" airtel "98xxxxxxxx"


cat /etc/ppp/peers/airtel
# This optionfile was generated by pppconfig 2.3.18.
connect "/usr/sbin/chat -v -f /etc/chatscripts/airtel"
user "98xxxxxxxx"
remotename airtel

Read Full Post »

Netcat can be of great help in trasferring files across network, that too in a really scalable pipeline. For bear minimum usage of file transfer, only tar and netcat utilities are required.

I tried different options and found the following superset pipeline.

sender:$ tar cf - some_directory | gzip -9 | \
  pv | gpg -c | nc -q0  25000
receiver:$ nc -l -p 25000 | gpg -d | gzip -d | \
  pv | tar xf -

Some Caveats

  • tar and nc do not offer any security, that is why gpg is used. In a trusted network, gpg pipeline elements could be removed from both sender and receiver.
  • pv could monitor the datarate very effectively. pv is not used symmetrically in the above pipe line. In the sender side it shows data rate of compressed stream whereas in the receiving side it shows the data rate of uncompressed stream. This is done deliberately so that the user could get an idea about both compressed/uncompressed data rates. By reordering gzip and pv, this could be reversed.
  • Instead of using tar and gzip separately, once could combine them and use ‘cfz’ and ‘xfz’. But, this limits the differential data rate as explained above. Fine control would be possible only by splitting them. Depending upon network throughput and CPU performance, either XZ or Lzop could be used instead of Gzip. The former offers extremely good compression, whereas the latter gives very good performance.

Read Full Post »

This is the scenario. I have a working ethernet device eth0 with an IP address 192.168.1.x. There is one more special interface in the system, say if1 which is connected to another network. That network is mainly used for transmitting multicast data.

Received multicast data has to be processed by a userland application. Due to some reason I want to project this multicast data as unicast, with a destination MAC as the MAC address of eth0 and destination IP address as the IP address of eth0. So, the application feels that the multicast is coming through eth0 interface. This has to be done for UDP packets to port 10000 only.

This is achieved by changing the driver of if1 interface the following way.

#include <ltnet/checksum.h>
#include <ltlinux/inetdevice.h>

static struct net_device *eth_dev;
static u32 local_ip;
static int init_done = 0;

static void init_fwd_process(struct net_device* net_dev)
  /* Here net_dev is the if1 device */
  struct in_ifaddr* ifa;

  if(likely(init_done == 1))

  /* Find out the net_device of eth0 */
  eth_dev = dev_get_by_name(&init_net, "eth0");

  /* First find out the IP address of eth0 */
  ifa = ((struct in_device*)eth_dev->ip_ptr)->ifa_list;
  local_ip = ifa->ifa_address;

  init_done = 1;

/* Initial 2 octets are used by the driver, for this particular driver */
#define ETH_DST_OFF 2
#define ETH_SRC_OFF 16
#define ETH_TYPE_OFF 30
#define IP_HDR_OFF 32
#define IP_TYPE_OFF (IP_HDR_OFF + 9)
#define UDP_HDR_OFF (IP_HDR_OFF + 20)

#define IP_TYPE_UDP 0x11

/* These are relative to IP packet */
#define IP_SRC_OFF 12
#define IP_DST_OFF 16
#define IP_CS_OFF 10
#define UDP_CS_OFF (20 + 6)

/* UDP Port we are interested in, 10000 in this case */
#define UDP_PORT 10000

static int fwd_mcast_packet(struct sk_buff *skb, unsigned char *data,
  int len, struct net_device  *net_dev)
  int n;
  u16 c;
  unsigned char* eth_src;
  unsigned char* eth_type;
  unsigned char* ip_hdr;
  unsigned char* ip_type;
  u32 ip_dst;
  u16 udp_dst;
  unsigned char hdr_802_3[14];

  do {

    eth_src = data + ETH_SRC_OFF;
    eth_type = data + ETH_TYPE_OFF;
    ip_hdr = data + IP_HDR_OFF;
    ip_type = data + IP_TYPE_OFF;
    ip_dst = (u32)(ntohl(*((u32 *)(data + IP_DSTABSOLUTE_OFF))));
    udp_dst = (u16)(ntohs(*((u16 *)(data + UDP_DPORT_OFF))));

    /* only IP packet */
    if((*(u16 *)eth_type) != (htons(ETH_P_IP)))
    /* only UDP */
    if(*ip_type != IP_TYPE_UDP)
    /* only multicast */
    if(!(ntohl(ip_dst) & 0xe0000000))
    /* only interesting port */
    if(ntohs(udp_dst) != UDP_PORT)

    /* We are interested in this packet
    Let other packets go through the usual path */
    /* destination is eth0 */
    memcpy(&hdr_802_3[0], eth_dev->dev_addr, 6);
    /* source is the original one */
    memcpy(&hdr_802_3[6], eth_src, 6);
    /* ethernet type is the original one */
    memcpy(&hdr_802_3[12], eth_type, 2);

    /* for aligning IP header on 16 byte */
    skb_reserve(skb, 2);
    /* copy the generated MAC header */
    memcpy(skb_put(skb, 14), hdr_802_3, 14);

    /* copy from the original IP header onwards */
    n = len - IP_HDR_OFF;
    memcpy(skb_put(skb, n), ip_hdr, n);

    skb->protocol = htons(ETH_P_IP);
    skb->pkt_type = PACKET_HOST;
    skb_reset_mac_header(skb); /* just like eth_type_trans */
    skb_pull(skb, ETH_HLEN);

    /* Destination IP address should be local_ip,
    Here skb->data is the beginning of IP header */
    memcpy(&skb->data[IP_DST_OFF], &local_ip, 4);

    /* regenerate IP checksum */
    skb->data[IP_CS_OFF] = 0;
    skb->data[IP_CS_OFF + 1] = 0;
    c = ip_fast_csum(skb->data, 5);
    memcpy(&skb->data[IP_CS_OFF], &c, 2);

    /* Only clear the UDP checksum */
    skb->data[UDP_CS_OFF] = 0;
    skb->data[UDP_CS_OFF + 1] = 0;

    /* call the main receive function */

    return len;

  return -1;

/* Inside the main packet receive function of driver */
  if(fwd_mcast_packet(skb, (unsigned char *)data, length, net_dev) > 0) {
    /* This packet has been already forwarded to IP stack,
    Just do no do anything */

Read Full Post »

There are many network throughtput and performance measurement tools available. One of the most widely used one is “iperf”. But, I found “netcat” to be a very versatile and fantastic tool to do the same.

The only “extra” requirement is to have shell access to both sides, so that netcat could be used.

First install the required tools
apt-get install netcat-traditional bwm-ng pv

Considering “machine1” and “machine2” are the two systems on both side of the link. Do the following

machine1:$ nc -l -p 5000 | pv > /dev/null
machine2:$ dd if=/dev/zero | pv | nc ip_address_or_hostname_of_machine1 5000

Netcat of machine1 opens a TCP listening socket and dumps the data to /dev/null. machine2 reads from /dev/zero and forwards that data to machine1 through netcat. Here, flow control is done by TCP stack and keeps the throughput just below the network link speed. Normally “pv” prints the pipe throughput in bytes, so multiply the real time value of “pv” by 8 to get the throughput in bits.

Tools like “bwm-ng” could be used to monitor the througput of the the interface instead of or along with “pv” also.

Some version of of netcat does not require “-p” in listening mode, so the comment could be “nc -l 5000” instead. In the sending side “cat” or “pv” could be used instead of “dd” also. But “dd” prints a nice summary when it is interrupted.

Of course, this is not completely one directional because TCP acknowledgements would be forwarded from machine1 to machine2, but that is much smaller compared to the data flowing in the forward direction.

If we have the liberty of having more shells, one more pair of netcat sessions could be opened in the reverse direction so that simultaneous/full duplex throughput could be measured.

I used this method consistently with Gigabit Ethernet and WiFi many times.

Read Full Post »

OpenEmbedded uses SRC_URI to determine where the source code is taken from. For many projects it would be convenient to use source code from local repository due to many reasons, in order to support that, it is possible to use file URI as ‘file://’

For example,
This can create problem because it is not sharable between different people due to absolute path. It may be possible to update FILESPATH varible, but I preferred to use the following method.

  • In $HOME/.bashrc, export a new environment variable, say LOCAL_REPO
    export LOCAL_REPO='/path/to/local_repo'
  • In /path/to/oe/source-me.txt, add the following lines
    if [ -z ${LOCAL_REPO} ]; then
      echo No LOCAL_REPO environment variable set
      return 2
  • In related bb files, use SRC_URI as
    SRC_URI = "file://${LOCAL_REPO}/whatever_project"

Read Full Post »

I wanted to setup a Linux kernel development system and the best choice I found was using KVM/Qemu. Both KVM and Qemu use the same virtual hardware specification, but KVM could give near native performance using special processor instructions. Since both of them use the same virtual hardware, the same virtual disk image could be used by either KVM or Qemu. So it is possible to replace ‘kvm’ command everywhere instead of ‘qemu’ at the penalty of performance.

I am using a Debian Unstable amd64 as both host and guest. It is possible to use other distributions, but package names etc. would be different.

First of all install the required packages on host, as root
apt-get install build-essential kernel-package fakeroot \
libncurses5-dev qemu kvm

Compiling the kernel

  1. Get the kernel source code. As of now I am using v3.0-rc5
    Untar/Clone it to say /path/to/linux_src
  2. Kernel Configuration.
    I found the following website for a minimal Linux Kernel Configuration
    But this configuration does not support libata, udev and ext4 which are used by current Debian systems. So, I created a configuration myself which also supports es1370 soundcard and 8139too network card, but there is no USB support. It could be downloaded from here. config-3.0.0.txt. Save it as /path/to/linux_src/.config
  3. Build the kernel. I used make-kpkg because of its simplicity, use
    a CONCURRENCY_LEVEL according to the number of processors in the build machine. It is used to set the “-jn” option for make.
    make-kpkg --initrd --revision=1custorm kernel_image

    Time Taken: With this kernel configuration and concurrency option the build takes around 4 minutes on my AMD Athlon X2 3600 machine.

Setting up a Debian virtual machine

  1. Check the host hardware could support KVM
    egrep 'svm|vmx' /proc/cpuinfo
  2. Create a filesystem image, I am using 3GB raw image format instead of qcow2, because the image could be mounted using a loopback device.
    qemu-img create -f raw debian.img 3G
  3. Get the latest x86_64 Debian netinstall CD image from,
    http://www.debian.org/devel/debian-installer/ Instead, you could install the i386 image, but the custom kernel also should be compiled for x86 architecture.
  4. Boot kvm(or qemu) and install the image. Use something like 512MB of RAM if possible
    kvm -m 512 -boot d -hda /path/to/debian.img \
    -cdrom /path/to/debian-testing-amd64-netinst.iso
  5. To transfer files to the virtual machine, there are two methods, a) using SSH to a running virtual machine, or b) mount the virtual machine image using a loopback device.
    To mount loopback, first find the sector offset
    fdisk -u -l /path/to/debian.img
    offset = sector size * start offset
    Normally that would be 32256 or 1048576
    mount -o loop,offset=1048576 /path/to/debian.img \ /path/to/mountpoint
    To ssh to a running guest, first install ssh server in the guest,
    restart guest with the “hostfwd” qemu option as given in the next section.
    Then do
    ssh root@ -p 10022
  6. To boot into the newly installed virtual machine using the stock Debian kernel
    kvm -smp 2 -m 512 -boot c -vga std -soundhw es1370 \
    -net nic -net user,hostfwd=tcp::10022-:22 \
    -hda /path/to/debian.img

    * SMP: use SMP if host supports it.
    * Networking: Realtek 8139too with “hostfwd” port forwarding so that it is possibe to SSH to the guest.
    * Sound: ES1370 card
  7. To boot using the custom kernel
    kvm -smp 2 -m 512 -boot c -vga std -soundhw es1370 \
    -net nic -net user,hostfwd=tcp::10022-:22 \
    -kernel /path/to/linux_src/arch/x86/boot/bzImage \
    -append root=/dev/sda1 \
    -hda /path/to/debian.img

Read Full Post »

Older Posts »

%d bloggers like this: