Building Carbon from the trunk

-->

WSO2 Carbon is the base platform for WSO2’s enterprise-grade middleware stack and as a start up work I spent some time on checking out Carbon from the trunk and building it. This post describes how I managed to accomplish this and some problems I encountered while doing so, and the way I got them solved.

Requirements

Before building carbon I was required to install subversion and maven. This was done using following commands ( in a Linux environment ).

$ sudo apt-get install subversion

$ sudo apt-get install maven2

Then I used the following command to checkout the carbon from trunk.

$ svn checkout https://svn.wso2.org/repos/wso2/trunk/carbon/

In the carbon_home directory the POM, "Project Object Model", which is an XML representation of a Maven project held in a file named pom.xml can be found. I could obtain more knowledge about POM from here.

http://maven.apache.org/pom.html

In the pom.xml file, which resides at the root level, the order of modules to be built can be found. For example, as per the below mentioned xml code snippet you have to start from the dependencies module and proceed with orbit, core, components, features and products.

dependencies

orbit

core

components

features

products

Dependencies directory need to be built in the order specified under the pom.xml in the dependencies directory. I used the following order of commands to accomplish this.

$ cd /home/randi/carbon/dependencies/axiom

$ mvn clean install 

-->

Before building ~/dependencies/axis2 I built the following two sub directories first.

$ cd /home/randi/carbon/dependencies/axis2/modules/tool/axis2-mar-maven-plugin

$ mvn clean install

$ cd /home/randi/carbon/dependencies/axis2/modules/tool/axis2-aar-maven-plugin

$ mvn clean install

Then the axis2 module was built.

$ cd /home/randi/carbon/dependencies/axis2

$ mvn clean install

-->

Then the same process was carried out for all the other modules inside the dependencies directory.

-->

In order to skip tests while building I used the following command.

$ mvn clean install -Dmaven.test.skip=true

 

-->

After successfully building ~/carbon/dependencies, I could build other modules as shown below.

-->

$ cd ~/carbon/orbit

$ mvn clean install -Dmaven.test.skip=true

$ cd ~/carbon/core

$ mvn clean install -Dmaven.test.skip=true

$ cd ~/carbon/components

$ mvn clean install -Dmaven.test.skip=true

$ cd ~/carbon/features

$ mvn clean install -Dmaven.test.skip=true

$ cd ~/carbon/products

$ mvn clean install -Dmaven.test.skip=true

$ sudo gedit /usr/bin/mvn

$ sudo gedit /etc/security/limits.conf 

-->

Problems encountered

while trying to accomplish the above task, I ecountered some problems as shown below.

"mvn clean install" crashes with java.lang.OutOfMemoryError: PermGen space after update to java 1.6.0_04

The system is out of resources.

Consult the following stack trace for details.

java.lang.OutOfMemoryError: PermGen space

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)

at java.lang.ClassLoader.defineClass(ClassLoader.java:616)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)

at java.net.URLClassLoader.access$000(URLClassLoader.java:58)

at java.net.URLClassLoader$1.run(URLClassLoader.java:197)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

at org.codehaus.plexus.compiler.javac.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:56)

at com.sun.tools.javac.main.JavaCompiler.(JavaCompiler.java:300)

at com.sun.tools.javac.main.JavaCompiler.instance(JavaCompiler.java:72)

at com.sun.tools.javac.main.Main.compile(Main.java:340)

at com.sun.tools.javac.main.Main.compile(Main.java:279)

at com.sun.tools.javac.main.Main.compile(Main.java:270)

at com.sun.tools.javac.Main.compile(Main.java:87)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.codehaus.plexus.compiler.javac.JavacCompiler.compileInProcess(JavacCompiler.java:549)

at org.codehaus.plexus.compiler.javac.JavacCompiler.compile(JavacCompiler.java:156)

at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:605)

at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:128)

at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)

[INFO] 1error

[INFO] -------------------------------------------------------------

[INFO] ------------------------------------------------------------------------

[ERROR] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Compilation failure

Failure executing javac, but could not parse the error:

I could overcome this by appending following line to /usr/bin/mvn.

sudo gedit /usr/bin/mvn (to open the file from gedit).

Then I added following line to the file.

export MAVEN_OPTS="-Xmx1024M -XX:MaxPermSize=128M"

  -------------------------------------------------------------------------------------------------------------------------------------

-->

When building the dependencies, I got the following build error message.

[INFO] ------------------------------------------------------------------------

[INFO] Building Apache Synapse - Distribution

[INFO] task-segment: [install]

[INFO] ------------------------------------------------------------------------

[INFO] [remote-resources:process {execution: default}]

[INFO] [site:attach-descriptor {execution: default-attach-descriptor}]

[INFO] [assembly:single {execution: distribution-package}]

[INFO] Reading assembly descriptor: src/main/assembly/bin.xml

[INFO] Processing DependencySet (output=synapse-${synapse.version}/lib)

[INFO] ------------------------------------------------------------------------

[ERROR] BUILD ERROR

[INFO] ------------------------------------------------------------------------

[INFO] Failed to create assembly: Failed to retrieve OS environment variables. Reason: Cannot run program "env": java.io.IOException: error=24, Too many open files

[INFO] ------------------------------------------------------------------------

[INFO] For more information, run Maven with the -e switch

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 3 minutes 44 seconds

[INFO] Finished at: Tue Sep 28 10:07:38 IST 2010

[INFO] Final Memory: 377M/665M

[INFO] ------------------------------------------------------------------------

to overcome this I did the following modifications to files.

I added following two lines to /etc/security/limits.conf .

soft nofile 2048

hard nofile 4096

further to set up user limits according to /etc/security/limits.conf, I added following line to /etc/pam.d/login

session required pam_limits.so

Then rebooting and building dependencies again solved my problem.

A month to remember...

Its coming to an end of a happening and important month in my life. Important things are meant to be cherished. Hence thought of spending sometime on revising the exciting moments I encountered throughout the month.


The beginning of this month was the time of playing the final trump after spending tough 4 years of academic progression. On 3rd of September the final exams in final year was ended and it was the hardest exam I have ever faced during the university I must say. However,though the four years we spent inside UOM is tough we really enjoyed each and every second we spent there.Of course the friendship we shared became the catalyst for that for sure. Hence that feeling of departure we felt during those last days was heart-felting. As a cure for this all of us tried to spend those few days in a remarkable manner and that made us to get even closer I guess. Despite the busy schedules me and my bunch of friends always found sometime to hangup with and have a walk around here and there in the University premises. Its my pleasure to take this moment to pay my gratitude to all my university buddies who paid their contribution to bring this four year of stay at uni a remarkable period.

Second week of this month was the opportunity we got to show off our colors as the cream of the country since the engineering exhibition 'Extreme Odyssey' was in action. The exhibition held throughout days 9,10 and 11th and we were lucky enough to exhibit our final year project to the crowd. In line with the exhibition Cs & Es conference was on the play and we could published our research paper of final year project at this conference too.

13th September was a special day in my life indeed as it was a new beginning. It was the day on which I started investing the gain from my entire academic life to the industry. It was the day at which I started my career as a software engineer at Wso2 Inc, a leading middleware company, the place where I always wanted to be in. and the time we joined there was a important period to the WSO2 as well since they were celebrating their 5 years of creating a global brand. Hence my first week at work was accompanied by WSO2-Con 2010,which was held on 14th and 15th of this month at HNB towers and the 5 year celebration party , which was held on 17th at Waters Edge.



I was so lucky enough to start my career by working to the first 100% open source cloud platform for enterprise applications, WSO2 Stratos and I was assigned to work on the Billing and Metering component of it. Last week of this month was spent on getting used with various technologies using at WSO2 and setting up the environment.

In overall this month was happening and sometimes there were moments where I felt like life race at a speed that I can hardly manage. But still it is the race for success, and in such a race no matter how hard you burn your energy, sooner or later you will definitely yield the profits.

Support Vector Machines for Speech Recognition

Support Vector Machines are a way of classifying data. it says that SVM is considered to be easier than using neural networks, but I'm yet to provide a comment on that :).

These days I’m trying to find a way to apply a SVM to improve accuracy of our speech recognition application. To achieve this I'm going to use LIBSVM.

The procedure for training the SVM is to

• Transform data to the format of an SVM package
•  Randomly try a few kernels and parameters
•  Test

First, I thought to check whether I could use this to classify and predict phonemes. First step was to prepare the input files.

The format of training and testing data file is: 

<label><index1>:<value1><index2><value2> 
.
.
.

Each line contains an instance and is ended by a '\n' character.  For

classification, is an integer indicating the class label
(multi-class is supported). For regression, is the target
value which can be any real number. For one-class SVM, it's not used
so can be any number.  Except using precomputed kernels (explained in
another section), : gives a feature (attribute) value.
is an integer starting from 1 and is a real
number. Indices must be in ASCENDING order. Labels in the testing file are only used to calculate accuracy or errors.

Hence I analyzed a voice signal using MATLAB to check whether I can represent phonemes in vector format. For this first I recorded a voice signal and saved in .wav format . When this signal is plotted, I observed that some phonemes are intuitively separable whereas some are not. However even these separable phonemes contain different number of samples. But according to input file format each data element need to be represented in a fixed size vector. So now, I’m trying to check whether I can represent every phoneme in a fixed sized vector using different transforms, filters etc.

Want to type e-mails in Sinhalese ??

The easiest way of typing from a native language is to use a transliteration application. For Sinhalese the most famous application was the UCSC real-time Unicode converter that can be found here.

Now this is becoming replaced with Google transliteration labs, which provides Unicode conversion support for number of languages including Sinhalese.  

Compared to Unicode converter this provides an intuitive transliteration scheme.

Transliteration Vs Translation

Translation is the process of converting meaning of a word from one language to another where as transliteration is the process of converting sound of a word to another language. Apparently, transliteration allows typing other languages phonetically in English letters through these converters.

To embed the feature to Gmail and make the life easy refer this.

Using neural networks for speech recognition

Our 'Sinhala Speech Recognition system' is now showing an elementary behavior, but we need to improve it more in order to get a satisfactory performance. To achieve that we are using optimization techniques on several faces such as improving and training language model and acoustic model, noise filtering techniques and machine learning approaches.

I’m going to use a neural network based approach to overcome the uncertainty due to variations of user and environment noise.  My intention is to use this blog entry to provide a step by step process to illustrate how to use a neural network in speech recognition.

Introduction to neural networks

A neural network is a collection of unique processing elements named neurons which are connected with similar other elements. First artificial neural network was designed in late 50’s and it’s much simpler than any biological neural network. Biological neural networks are far more complex and lots of studies are happening to discover the secrets behind these biological systems. Neural networks perform their role in applications that have limitations of using regular computer programs such as image recognition, speech recognition and making decisions.

Neural network differ from regular programming due to its requirement of training before performing task where as in regular programming task is programmed.

Structure of a neural network

A neural network consists of set of inputs, a weight (that multiplies each input ) and output per neuron. The output is calculated by an activation function applied on the sum of all inputs that are multiplied by the weights.

Avatar, Na'vi and networking

Nice film with breath taking animations. But I wonder whether the fanciness of surrounding has degrade the quality of effort. However rather than animations I find the concept of Na’vi creatures’ networking much more interesting. Those creatures have powerful network to which they can upload and down load information from. Rather than artificial wired / wireless network topologies existing on earth their network is via the fibers running though trees and other creatures.Their sacred trees are the backbone of their network and all the creatures including Na’vies and animals are plug and play workstations. All the creatures have a device (similar to USB device) which can be plug to this network or other creatures via corresponding port and facilitate the communication among them. On one hand, it could be seen to earth people that these creatures are machines rather living beings, but still those creatures live with nature unlike to human kind on earth.

I wonder whether Na’vi kind shows the possibility of ‘genetic globalization/ genetic networking’ that may be possible thorough genetic evolution someday.
Selective mutation by mother nature ensures that those features relevant to a species get enhanced and those are unnecessary get depreciated and vanished. Communication on the other hand is an essential for any kind of species for their existence, reproduction, evolvement and even to interact with other species. Hence similar to the way our ancestors and other living beings at early stages of evolution came up and improved their own languages or sounds or gestures to communicate, the evolvement of these methods also be technically possible with genetic evolution. However, human kind has deviate from the natural evolving pattern and pace, and has gone for artificial communication and network topologies. This has generated a world that could have very different if such evolvement happen naturally.

What if someone didn’t invent electricity, vacuum tubes, resistors and finally all digital media? Will us be not linked to each other as we are today via internet? Will we be not able to communicate globally? What I think is if we adhered to the pace of evolution by Mother Nature, she would have found us a way to be networked among each other just like Na` vi creatures do, because nature of genetic evolution is to upgrade towards positive direction and degrade towards negative direction. Hence had we lived with nature and had we let nature to evolve and develop us, entire world could have been networked and globalized genetically rather than digitally.