Thursday, December 31, 2015

Java performance tuning - standard utils


All utils from this post are located at $JAVA_HOME/bin directory

1. Java process identification

First of all we have to identify process id of our java application.
In UNIX we can use
ps -aux

In Windows - we can open TaskManager

Or in both UNIX/Windows, we can just run utility :
jps

which will show processes related with java:
13528 Jps
14056 Gems

2. jmap

jmap can be used  for checking "memory map".

Example of usage :
jmap -heap 118984

Example of results :

using thread-local object allocation.
Parallel GC with 13 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 536870912 (512.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 134217728 (128.0MB)
   used     = 66957592 (63.855735778808594MB)
   free     = 67260136 (64.1442642211914MB)
   49.887293577194214% used
From Space:
   capacity = 22347776 (21.3125MB)
   used     = 5406752 (5.156280517578125MB)
   free     = 16941024 (16.156219482421875MB)
   24.19369157807918% used
To Space:
   capacity = 22347776 (21.3125MB)
   used     = 0 (0.0MB)
   free     = 22347776 (21.3125MB)
   0.0% used
PS Old Generation
   capacity = 357957632 (341.375MB)
   used     = 1262024 (1.2035598754882812MB)
   free     = 356695608 (340.1714401245117MB)
   0.3525623948702398% used
PS Perm Generation
   capacity = 31588352 (30.125MB)
   used     = 31448256 (29.99139404296875MB)
   free     = 140096 (0.13360595703125MB)
   99.55649474844398% used

3. jinfo 

jinfo can be used for checking java options which were used for application. For example we found that something is wrong with PermSize on previous step, so we want to check java options :

 jinfo -flag PermSize 118984
Picked up _JAVA_OPTIONS: -XX:PermSize=21757952

4. jstat

jstat can be used for getting statistic from JVM during specified periods of time.

For example, statistic from garbage collector which were gathered 5 times with interval 1sec(1000ms) :

bash-4.1$ jstat -gc 1996103 1000 5

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
768.0  1856.0  0.0    0.0   123712.0 91910.9   687232.0    3504.9   26304.0 16454.5     48    0.134   8      0.305    0.439
768.0  1856.0  0.0    0.0   123712.0 93250.5   687232.0    3504.9   26304.0 16454.5     48    0.134   8      0.305    0.439
768.0  1856.0  0.0    0.0   123712.0 94294.1   687232.0    3504.9   26304.0 16454.5     48    0.134   8      0.305    0.439
768.0  1856.0  0.0    0.0   123712.0 95926.5   687232.0    3504.9   26304.0 16454.5     48    0.134   8      0.305    0.439
768.0  1856.0  0.0    0.0   123712.0 98104.9   687232.0    3504.9   26304.0 16454.5     48    0.134   8      0.305    0.439


5. jstack

with jstack we can check which processes are now running inside our JVM session.

jstack 12264

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000053247000 nid=0x2cec waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000053246000 nid=0x22f8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000531ea000 nid=0x3c18 in Object.wait() [0x000000005453f000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fd5870b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000fd5870b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000531e3000 nid=0x60bc in Object.wait() [0x000000005443f000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fd586af8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000fd586af8> (a java.lang.ref.Reference$Lock)

"VM Thread" os_prio=2 tid=0x00000000531dd000 nid=0x1934 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000000000221d800 nid=0x4334 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000000000221f000 nid=0x18f8 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002220800 nid=0x23b4 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002224000 nid=0x30a0 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x00000000545e8000 nid=0x3ddc waiting on condition

6. Visual tools 

Of course, if it's possible, much more easier to open "visual" tool like JConsole or VisualVM.

Wednesday, December 30, 2015

Groovy and database

related posts : BDD with groovy and Spock

1. Intro

Groovy can help to implement a lot of things. One of them - it can be use for database testing -  Grooovy can make it very simple and easy. In this post - example of how it can used for testing some features which were implemented using Oracle DB : I had to test views and triggers which were created for views. For that I had to deploy to DB scripts. Before deploying i had to re-point views to tests views/tables. And, of course, run some tests with data modification.

2. DB connection with bootstrap of  DB connection library

Before connecting to DB, we have to load driver class. For this purpose we don't need any classpath settings - we can just load needed class in our code :

def jarFile = new File("C://MyProject/java/lib/global/oracle/ojdbc6.jar");
this.class.classLoader.rootLoader.addURL(jarFile.toURI().toURL());


And then the class is loaded - we can connect to DB:

import groovy.sql.Sql
def sql = Sql.newInstance("jdbc:oracle:thin:@localhost:1522:MYDB", "MYUSERNAME", "MYPASSWORD","oracle.jdbc.driver.OracleDriver")


3. Test the connection 

To test the connection there are a lot of options, I decided just to get a time from Oracle :

def testConnection(sql) {
   sql.eachRow("SELECT sysdate as dt FROM dual") {
     println "current date : ${it.dt} "
   }
}

- if this procedure is returning time - everything is working and we can move forward. 


4. Execute statement

On previous step we executed SELECT statement. For executing just SQL with no return (for example DDL operation) we can use execute method :

sql.execute("create table TEST_TABLE as select * from "+sourceTable+" where 0=1")

Also very often i needed to execute SQL operation in "silent mode" - without any error notification. For example, I have to DROP the table, but if doesn't exist - I will have an exception. But I don't needed - if table not exists - that's exactly what i want! So I wrapped execute method with try/catch :

def executeSilent(sql, statement) {
  //println statement
  try {     
     sql.execute(statement) 
  } catch(Exception e) {
  }
}

Now we can make some table modification this way :

  sql.eachRow("select * from user_tab_columns where table_name='"+sourceTable+"'") {
   def modifySql="ALTER TABLE TEST_TABLE MODIFY("+it.column_name+"  NULL)"
   executeSilent(sql, modifySql)         
  }


5. Deploy script to DB 

Also I needed before every test deploy to DB script. That is also simple for implementing : I used symbol "/" as a "block end" in SQL scripts, so, to split file content for SQL statements for execution them "one by one" I used fileContent.split("/") method  :

def deploySqlScript(sql, filePath, tableName, viewName) {
  def fileContent = new File(filePath).text
  fileContent=fileContent.toUpperCase()
  fileContent=fileContent.replace(viewName, "V_TEST_TABLE")  
  fileContent=fileContent.replace(tableName, "TEST_VIEW")  
  for (String s:fileContent.split("/")) {
    try {
       sql.execute(s.trim())
    } catch (Exception e) {
         println "Failed to execute:" +s.trim()+" Exception:"+e.toString()
    }
  }
}


6. Simple test

For my tests, I needed to compare count of records, so I created function :

def selectDataCount(sql, tableName, id) {
  def selectSql="select count(*) as cnt from "+tableName+" where id="+id; 
  def result=0;
  sql.eachRow(selectSql) {
     result=it.cnt
  }
  return result;
}


And for checking, I created analog of "ASSERT" :

def checkCount(sql, tableName, id, expectedCnt) {
   def actualCnt=selectDataCount(sql, tableName, id)
   if (expectedCnt!=actualCnt) {
      println "!!![Failed] for $tableName, $id. Expected count:$expectedCnt actual count:$actualCnt"
   } else {
      println "[OK] for $tableName, $id. Expected count:$expectedCnt actual count:$actualCnt"
   }
}

After that, I was able to write my tests : 

def runTest(sql, tableName) {
  def id=1
  deleteData(sql, tableName)   

  println "\ninsert first row"
  insertData(sql, tableName, id)  
  checkCount(sql, tableName, id, 1)

  println "\ncheck for delete:"
  deleteData(sql, tableName, id)
  checkCount(sql, tableName, id, 0)  
.....................................................................
}


7. Full source

def jarFile = new File("C://MyProject/java/lib/oracle/ojdbc6.jar");
this.class.classLoader.rootLoader.addURL(jarFile.toURI().toURL());

import groovy.sql.Sql
def sql = Sql.newInstance("jdbc:oracle:thin:@localhost:1522:MYDB", "MYUSERNAME", "MYPASSWORD","oracle.jdbc.driver.OracleDriver")

def testConnection(sql) {
   sql.eachRow("SELECT sysdate as dt FROM dual") {
     println "current date : ${it.dt} "
   }
}

def executeSilent(sql, statement) {
  //println statement
  try {     
     sql.execute(statement) 
  } catch(Exception e) {
  }
}

def prepareTestTable(sql,sourceTable) {
   executeSilent(sql, "drop table TEST_TABLE") 
   executeSilent(sql, "drop view  V_TEST_TABLE") 

   // create test table 
   sql.execute("create table TEST_TABLE as select * from "+sourceTable+" where 0=1")  

   // make colums "nullable"
   sql.eachRow("select * from user_tab_columns where table_name='"+sourceTable+"'") {
   def modifySql="ALTER TABLE TEST_TABLE MODIFY("+it.column_name+"  NULL)"
   executeSilent(sql, modifySql)         
  }

  // create unique index on SERVICE_ID, SERVICE_VERSION
  sql.execute("CREATE UNIQUE INDEX IDX_TEST_MIG_SRV_UNQ ON TEST_MIG (SERVICE_ID, SERVICE_VERSION)")
  println "Test table for $sourceTable is ready"
}

def deploySqlScript(sql, filePath, tableName, viewName) {
  def fileContent = new File(filePath).text
  fileContent=fileContent.toUpperCase()
  fileContent=fileContent.replace(viewName, "V_TEST_TABLE")  
  fileContent=fileContent.replace(tableName, "TEST_VIEW")  
  for (String s:fileContent.split("/")) {
    try {
       sql.execute(s.trim())
    } catch (Exception e) {
         println "Failed to execute:" +s.trim()+" Exception:"+e.toString()
    }
  }
}

def deleteData(sql, tableName, id=null) {
   def delSql="delete from "+tableName;
   if (id!=null) {
     delSql+=" where id="+id;
   }  
   sql.executeUpdate(delSql);
}

def insertData(sql, tableName, id) {    
    def insertStr="insert into "+tableName+" (id) values ("+id+")"
    sql.execute(insertStr)
}

def selectDataCount(sql, tableName, id) {
  def selectSql="select count(*) as cnt from "+tableName+" where id="+id; 
  def result=0;
  sql.eachRow(selectSql) {
     result=it.cnt
  }
  return result;
}

def checkCount(sql, tableName, id, expectedCnt) {
   def actualCnt=selectDataCount(sql, tableName, id)
   if (expectedCnt!=actualCnt) {
      println "!!![Failed] for $tableName, $id. Expected count:$expectedCnt actual count:$actualCnt"
   } else {
      println "[OK] for $tableName, $id. Expected count:$expectedCnt actual count:$actualCnt"
   }
}

def runTest(sql, tableName) {
  def id=1
  deleteData(sql, tableName)   

  println "\ninsert first row"
  insertData(sql, tableName, id)  
  checkCount(sql, tableName, id, 1)

  println "\ncheck for delete:"
  deleteData(sql, tableName, id)
  checkCount(sql, tableName, id, 0)  
...........................................................................................................
}

def prepareAndRunTest(sql, tableName, viewName, scriptPath) {
   prepareTestTable(sql, tableName)
   deploySqlScript(sql, scriptPath, tableName, viewName)
   runTest(sql, "V_TEST_VIEW")  
}

testConnection(sql) 
//prepareAndRunTest(sql, "TABLE1", "V_TABLE1", "scripts/V_TABLE1.sql")
//prepareAndRunTest(sql, "TABLE2", "V_TABLE2", "scripts/V_TABLE2.sql")
........................................................................................................

Sunday, December 13, 2015

Java networking : NIO

Related posts : Java networking : UDP vs TCP

1. Intro

From wiki : https://en.wikipedia.org/wiki/Non-blocking_I/O_(Java)
Non-blocking I/O (usually called NIO, and sometimes called "New I/O") is a collection of Java programming language APIs that offer features for intensive I/O operations. 
Main elements : 

  • NIO data transfer is based on buffers
  • Channels are designed to provide for bulk data transfers to and from NIO buffers.
  • A selector  provides a mechanism for waiting on channels and recognizing when one or more become available for data transfer.


2. NIO server

Main concept of NIO : we can have one server thread which can work with multiple client threads in non-blocking mode. This is implemented by selectors, which have to be used of both sides : client and server. Server is "listening" the channel using selector :
Iterator<SelectionKey> keys = selector.selectedKeys().iterator();
After that we can check what exactly clients wants from server :
SelectionKey key = keys.next();
...
if (key.isConnectable()){  // client want to connect to server 
if (key.isWritable()){   // client want server to write something to channel
if (key.isReadable()){ // client want to  read from server 

 
After understanding what exactly client wants from server (connect, read or write), we can get client channel and  proceed with that operation :
SocketChannel channel = (SocketChannel) key.channel();
...
channel.accept();
..
ByteBuffer readBuffer = ByteBuffer.allocate(1000);
channel.read(readBuffer);
....
channel.write(ByteBuffer.wrap(data));

Also,  as it was mentioned before - data transfer is based on buffers.

Full code of server :

package com.demien.networking;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.*;

public class NioServer implements Runnable {

    public final static String ADDRESS = "127.0.0.1";
    public final static int PORT = 6666;
    public final static long TIMEOUT = 10000;

    private ServerSocketChannel serverChannel;
    private Selector selector;
    /**     * This hashmap is important. It keeps track of the data that will be written to the clients.     * This is needed because we read/write asynchronously and we might be reading while the server     * wants to write. In other words, we tell the Selector we are ready to write (SelectionKey.OP_WRITE)     * and when we get a key for writting, we then write from the Hashmap. The write() method explains this further.     */    private Map<SocketChannel, byte[]> clientMessages = new HashMap<SocketChannel, byte[]>();

    public static void main(String args[]) {
        Thread serverThread = new Thread(new NioServer());
        serverThread.run();
    }

    public NioServer() {
        init();
    }

    private void init() {
        System.out.println("initializing server");
        // We do not want to call init() twice and recreate the selector or the serverChannel.        if (selector != null) return;
        if (serverChannel != null) return;

        try {
            // This is how you open a Selector            selector = Selector.open();
            // This is how you open a ServerSocketChannel            serverChannel = ServerSocketChannel.open();
            // You MUST configure as non-blocking or else you cannot register the serverChannel to the Selector.            serverChannel.configureBlocking(false);
            // bind to the address that you will use to Serve.            serverChannel.socket().bind(new InetSocketAddress(ADDRESS, PORT));

            /**             * Here you are registering the serverSocketChannel to accept connection, thus the OP_ACCEPT.             * This means that you just told your selector that this channel will be used to accept connections.             * We can change this operation later to read/write, more on this later.             */            serverChannel.register(selector, SelectionKey.OP_ACCEPT);

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Override    public void run() {
        System.out.println("Now accepting connections...");
        try {
            // A run the server as long as the thread is not interrupted.            while (!Thread.currentThread().isInterrupted()) {
                /**                 * selector.select(TIMEOUT) is waiting for an OPERATION to be ready and is a blocking call.                 * For example, if a client connects right this second, then it will break from the select()                 * call and run the code below it. The TIMEOUT is not needed, but its just so it doesn't                 * block undefinitely.                 */                selector.select(TIMEOUT);

                /**                 * If we are here, it is because an operation happened (or the TIMEOUT expired).                 * We need to get the SelectionKeys from the selector to see what operations are available.                 * We use an iterator for this.                 */                Iterator<SelectionKey> keys = selector.selectedKeys().iterator();

                while (keys.hasNext()) {
                    SelectionKey key = keys.next();
                    // remove the key so that we don't process this OPERATION again.                    keys.remove();

                    // key could be invalid if for example, the client closed the connection.                    if (!key.isValid()) {
                        continue;
                    }
                    /**                     * In the server, we start by listening to the OP_ACCEPT when we register with the Selector.                     * If the key from the keyset is Acceptable, then we must get ready to accept the client                     * connection and do something with it. Go read the comments in the accept method.                     */                    if (key.isAcceptable()) {
                        System.out.println("Accepting connection");
                        accept(key);
                    }
                    /**                     * If you already read the comments in the accept() method, then you know we changed                     * the OPERATION to OP_WRITE. This means that one of these keys in the iterator will return                     * a channel that is writable (key.isWritable()). The write() method will explain further.                     */                    if (key.isWritable()) {
                        System.out.println("Writing...");
                        write(key);
                    }
                    /**                     * If you already read the comments in the write method then you understand that we registered                     * the OPERATION OP_READ. That means that on the next Selector.select(), there is probably a key                     * that is ready to read (key.isReadable()). The read() method will explain further.                     */                    if (key.isReadable()) {
                        System.out.println("Reading connection");
                        read(key);
                    }
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            closeConnection();
        }

    }

    /**     * We registered this channel in the Selector. This means that the SocketChannel we are receiving     * back from the key.channel() is the same channel that was used to register the selector in the accept()     * method. Again, I am just explaning as if things are synchronous to make things easy to understand.     * This means that later, we might register to write from the read() method (for example).     */    private void write(SelectionKey key) throws IOException {
        SocketChannel channel = (SocketChannel) key.channel();
        writeToChannel(channel);
        key.interestOps(SelectionKey.OP_READ);
    }

    private void writeToChannel(SocketChannel channel) throws IOException {
        /**         * The hashmap contains the object SockenChannel along with the information in it to be written.         * In this example, we send the "Hello from server" String and also an echo back to the client.         * This is what the hashmap is for, to keep track of the messages to be written and their socketChannels.         */        byte[] data = clientMessages.get(channel);
        if (data != null) {
            clientMessages.remove(channel);

            // Something to notice here is that reads and writes in NIO go directly to the channel and in form of            // a buffer.            channel.write(ByteBuffer.wrap(data));
        }

        // Since we wrote, then we should register to read next, since that is the most logical thing        // to happen next. YOU DO NOT HAVE TO DO THIS. But I am doing it for the purpose of the example        // Usually if you register once for a read/write/connect/accept, you never have to register again for that unless you        // register for none (0). Like it said, I am doing it here for the purpose of the example. The same goes for all others.
    }

    // Nothing special, just closing our selector and socket.    private void closeConnection() {
        System.out.println("Closing server down");
        if (selector != null) {
            try {
                selector.close();
                serverChannel.socket().close();
                serverChannel.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

    /**     * Since we are accepting, we must instantiate a serverSocketChannel by calling key.channel().     * We use this in order to get a socketChannel (which is like a socket in I/O) by calling     * serverSocketChannel.accept() and we register that channel to the selector to listen     * to a WRITE OPERATION. I do this because my server sends a hello message to each     * client that connects to it. This doesn't mean that I will write right NOW. It means that I     * told the selector that I am ready to write and that next time Selector.select() gets called     * it should give me a key with isWritable(). More on this in the write() method.     */    private void accept(SelectionKey key) throws IOException {
        ServerSocketChannel serverSocketChannel = (ServerSocketChannel) key.channel();
        SocketChannel socketChannel = serverSocketChannel.accept();
        socketChannel.configureBlocking(false);

        socketChannel.register(selector, SelectionKey.OP_WRITE);
        byte[] hello = new String("Hello from server").getBytes();
        clientMessages.put(socketChannel, hello);
    }

    /**     * We read data from the channel. In this case, my server works as an echo, so it calls the echo() method.     * The echo() method, sets the server in the WRITE OPERATION. When the while loop in run() happens again,     * one of those keys from Selector.select() will be key.isWritable() and this is where the actual     * write will happen by calling the write() method.     */    private void read(SelectionKey key) throws IOException {
        SocketChannel channel = (SocketChannel) key.channel();
        ByteBuffer readBuffer = ByteBuffer.allocate(1024);
        readBuffer.clear();
        int read;
        try {
            read = channel.read(readBuffer);
        } catch (IOException e) {
            System.out.println("Reading problem, closing connection");
            key.cancel();
            channel.close();
            return;
        }
        if (read == -1) {
            System.out.println("Nothing was there to be read, closing connection");
            channel.close();
            key.cancel();
            return;
        }
        // IMPORTANT - don't forget the flip() the buffer. It is like a reset without clearing it.        readBuffer.flip();
        byte[] data = new byte[1000];
        readBuffer.get(data, 0, read);
        System.out.println("Received: " + extractString(data));

        echo(key, data);
    }

    private void echo(SelectionKey key, byte[] data) throws IOException {
        SocketChannel socketChannel = (SocketChannel) key.channel();
        clientMessages.put(socketChannel, data);
        writeToChannel(socketChannel);
        key.interestOps(SelectionKey.OP_WRITE);
    }

    public static String extractString(byte[] array) {
        List<Byte> wrapper = new ArrayList<>();
        for (int i = 0; i < array.length; i++) {
            if (array[i] != 0) {
                wrapper.add(array[i]);
            } else {
                break;
            }
        }
        byte[] resultArray = new byte[wrapper.size()];
        for (int i = 0; i < wrapper.size(); i++) {
            resultArray[i] = wrapper.get(i);
        }


        return new String(resultArray);
    }
}


3. NIO client

Client implementation is based on the same principals : register selector and wait for "events" to understand what to do : connect, read or write.
Full code of client :

package com.demien.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.Iterator;


public class NioClient implements Runnable {

    private Selector selector;
    private BufferedReader stdIn = new BufferedReader( new InputStreamReader(System.in));

    public static void main(String[] args) {
        NioClient clientThread = new NioClient();
        Thread thread = new Thread(clientThread);
        thread.start();
    }

        @Override        public void run() {
            SocketChannel channel;
            try {
                selector = Selector.open();
                channel = SocketChannel.open();
                channel.configureBlocking(false);

                channel.register(selector, SelectionKey.OP_CONNECT);
                channel.connect(new InetSocketAddress(NioServer.ADDRESS, NioServer.PORT));

                System.out.println("Starting....");

                while (!Thread.interrupted()){

                    selector.select(1000);

                    Iterator<SelectionKey> keys = selector.selectedKeys().iterator();

                    while (keys.hasNext()){
                        SelectionKey key = keys.next();
                        keys.remove();

                        if (!key.isValid()) continue;

                        if (key.isConnectable()){
                            System.out.println("I am connected to the server");
                            connect(key);
                        }
                        if (key.isWritable()){
                            write(key);
                        }
                        if (key.isReadable()){
                            read(key);
                        }
                    }
                }
            } catch (Exception e1) {
                   e1.printStackTrace();
            } finally {
                close();
            }
        }

        private void close(){
            try {
                selector.close();
            } catch (IOException e) {
                  e.printStackTrace();
            }
        }

        private void read (SelectionKey key) throws IOException {
            SocketChannel channel = (SocketChannel) key.channel();
            ByteBuffer readBuffer = ByteBuffer.allocate(1000);
            readBuffer.clear();
            int length;
            try{
                length = channel.read(readBuffer);
            } catch (IOException e){
                System.out.println("Reading problem, closing connection");
                key.cancel();
                channel.close();
                return;
            }
            if (length == -1){
                System.out.println("Nothing was read from server");
                channel.close();
                key.cancel();
                return;
            }
            readBuffer.flip();
            byte[] buff = new byte[1024];
            readBuffer.get(buff, 0, length);
            System.out.println("Received: " + NioServer.extractString(buff));
            key.interestOps(SelectionKey.OP_WRITE);
        }

        private void write(SelectionKey key) throws IOException {
            SocketChannel channel = (SocketChannel) key.channel();
            String userInput = stdIn.readLine();
                channel.write(ByteBuffer.wrap(userInput.getBytes()));
            // lets get ready to read.            key.interestOps(SelectionKey.OP_READ);
        }

        private void connect(SelectionKey key) throws IOException {
            SocketChannel channel = (SocketChannel) key.channel();
            if (channel.isConnectionPending()){
                channel.finishConnect();
            }
            channel.configureBlocking(false);
            channel.register(selector, SelectionKey.OP_READ);
        }

} 

4. Running
After starting we can send messages to server by input (console) and watch "echo" output :
Starting....
I am connected to the server
Received: Hello from server
hello. i'm a client
Received: hello. i'm a client
how are you ?
Received: how are you ?



Java networking : UDP vs TCP

Related topic : Java networking : NIO

1. Intro 

Both TCP and UDP are protocols used for sending bits of data — known as packets — over the Internet. They both build on top of the Internet protocol. So, what's the difference ? 
In shorts, UDP - is more simple because it doesn't guarantee package delivery. 

Wiki : 
https://en.wikipedia.org/wiki/User_Datagram_Protocol
The User Datagram Protocol (UDP) is one of the core members of the Internet protocol suite. The protocol was designed by David P. Reed in 1980 and formally defined in RFC 768.
UDP uses a simple connectionless transmission model with a minimum of protocol mechanism. It has no handshaking dialogues, and thus exposes the user's program to any unreliability of the underlying network protocol. There is no guarantee of delivery, ordering, or duplicate protection. UDP provides checksums for data integrity, and port numbers for addressing different functions at the source and destination of the datagram.

https://en.wikipedia.org/wiki/Transmission_Control_Protocol
The Transmission Control Protocol (TCP) is a core protocol of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP). Therefore, the entire suite is commonly referred to as TCP/IP. TCP provides reliable, ordered, and error-checked delivery of a stream of octets between applications running on hosts communicating over an IP network. TCP is the protocol that major Internet applications such as the World Wide Webemailremote administration and file transferrely on. Applications that do not require reliable data stream service may use the User Datagram Protocol (UDP), which provides a connectionless datagram service that emphasizes reduced latency over reliability.




2. UDP Server.

Let's now create a server for UDP protocol. In Java we can use classes DatagramSocket and DatagramPacket. Also I had to create procedure to extract Strings for readed bytes. Server is reading data from socket and sending it back to client.

package com.demien.networking;

import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.List;

public class UdpServer {
    public final static int PORT_NUMBER = 6666;

    public static String extractString(byte[] array) {
        List<Byte> wrapper=new ArrayList<>();
        for (int i=0; i<array.length;i++) {
            if (array[i]!=0) {
                wrapper.add(array[i]);
            } else {
                break;
            }
        }
        byte[] resultArray=new byte[wrapper.size()];
        for (int i=0; i<wrapper.size();i++) {
            resultArray[i]=wrapper.get(i);
        }


        return  new String(resultArray);
    }

    public static void main(String args[]) throws Exception {
        System.out.println("Starting.... waiting for client connections......");
        try (
                DatagramSocket serverSocket = new DatagramSocket(PORT_NUMBER);
        ) {
            byte[] receiveData = new byte[1024];
            byte[] sendData;
            while (true) {
                DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);
                serverSocket.receive(receivePacket);
                String inputLine = extractString(receivePacket.getData());
                System.out.println("received: " + inputLine);
                InetAddress IPAddress = receivePacket.getAddress();
                int port = receivePacket.getPort();
                sendData = inputLine.getBytes();
                DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, port);
                serverSocket.send(sendPacket);
            }
        }
    }
}


3. UDP Client.

Client application is reading data from system.it(console), sending it to server and showing server responce.

package com.demien.networking;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;

public class UdpClient {
    public static void main(String args[]) throws Exception {
        System.out.println("Starting.... ");
        try (
                BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
                DatagramSocket clientSocket = new DatagramSocket();
        ) {
            InetAddress IPAddress = InetAddress.getByName("localhost");
            byte[] sendData;
            byte[] receiveData = new byte[1024];
            System.out.println("Enter message :  ");
            while (true) {
                String userInput = inFromUser.readLine();
                sendData = userInput.getBytes();
                DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, UdpServer.PORT_NUMBER);
                clientSocket.send(sendPacket);
                DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);
                clientSocket.receive(receivePacket);
                String receivedSentence = UdpServer.extractString(receivePacket.getData());
                System.out.println("response from server:" + receivedSentence);
            }
        }

    }
}



4. TCP Server. 

The same logic as for UDP server, but using another classes for TCP protocol.

package com.demien.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;

public class TcpServer {

    public final static int PORT_NUMBER=6666;

    public static void main(String[] args) throws IOException {
        System.out.println("Starting.... waiting for client connections......");
        try (
                ServerSocket serverSocket = new ServerSocket(PORT_NUMBER);
                Socket clientSocket = serverSocket.accept();
                PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);
                BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
        ) {
            System.out.println("Client is connected");
            String inputLine;
            while ((inputLine = in.readLine()) != null) {
                System.out.println("received:"+inputLine);
                out.println(inputLine);
            }
        } catch (IOException e) {
            System.out.println("Exception caught when trying to listen on port "                    + PORT_NUMBER + " or listening for a connection");
            System.out.println(e.getMessage());
        }
    }
}

5. TCP Client.

Also, the same client logic : read from console, send to server, display response.

package com.demien.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.Socket;
import java.net.UnknownHostException;

public class TcpClient {

    public final static String HOST_NAME="localhost";

    public static void main(String[] args) throws IOException {

        System.out.println("Starting.... ");
        try (
                Socket echoSocket = new Socket(HOST_NAME, TcpServer.PORT_NUMBER);
                PrintWriter out = new PrintWriter(echoSocket.getOutputStream(), true);
                BufferedReader in = new BufferedReader(new InputStreamReader(echoSocket.getInputStream()));
                BufferedReader stdIn = new BufferedReader( new InputStreamReader(System.in))
        ) {
            System.out.println("Connected to server. Enter message :  ");
            String userInput;
            while ((userInput = stdIn.readLine()) != null) {
                out.println(userInput);
                System.out.println("response from server: " + in.readLine());
            }
        } catch (UnknownHostException e) {
            System.err.println("Unknown host:" + HOST_NAME);
            System.exit(1);
        } catch (IOException e) {
            System.err.println("Couldn't get I/O for the connection to " +
                    HOST_NAME);
            System.exit(1);
        }
    }
}


Saturday, November 21, 2015

Spring WS with Spring Boot

Related posts :   Simple Spring boot application

1. Intro 

Usually, to make a web service we had to configure web server (like oracle weblogic, glassfish) and deploy there our web service. With SpringWS  - everything is much more simple : our SpringWS-based web service can be deployed just to Tomcat - we don't need web servers. But with using of SpringBoot for running SpringWS-based web service  - we even don't need a Tomcat : we can just run our application using main(String[] args)  function and that's it!


2. Project structure

SpringWS use "contract first" approach for web services. That means: we have to start with "contract" : our web services have to be "explained" using XSD schemes for domain entities and operations. In this example we have only one entity class : Category.xsd and operation file CategoryOperations.xsd  with 2 operations: save category and getCategoryById.

After that, based on XSD files we have to generate .java classes. Highlighted files on next picture were generated - we were not creating them manually :





3. pom.xml 

Most interesting thing here located in the bottom of file  - it's a plugin for generating java src files based on our XSD schemes.
<?xml version="1.0"?><project xmlns="http://maven.apache.org/POM/4.0.0"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <artifactId>com.demien</artifactId>
   <groupId>springws</groupId>
   <version>1.0</version>
   <packaging>jar</packaging>

   <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
                <version>1.2.7.RELEASE</version>
      <relativePath/>
   </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-ws</artifactId>
        </dependency>
        <dependency>
            <groupId>wsdl4j</groupId>
            <artifactId>wsdl4j</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.ws.xmlschema</groupId>
            <artifactId>xmlschema-core</artifactId>
            <version>2.0.1</version>
        </dependency>
    </dependencies>


   <build>
      <plugins>
         <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>jaxb2-maven-plugin</artifactId>
            <version>1.4</version>
            <executions>
               <execution>
                  <goals>
                     <goal>xjc</goal>
                  </goals>
                  <phase>generate-sources</phase>
               </execution>
            </executions>
            <configuration>
               <clearOutputDir>false</clearOutputDir>
               <outputDirectory>src/main/java</outputDirectory>
               <schemaDirectory>src/main/resources</schemaDirectory>
               <enableIntrospection>false</enableIntrospection>
            </configuration>
         </plugin>
      </plugins>
   </build>

</project>


4. Entity schema (Category.xsd)

Just regular java POJO class "transalted" to XSD language:

<?xml version="1.0" encoding="UTF-8"?><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"            xmlns="http://com/demien/springws/domain" targetNamespace="http://com/demien/springws/domain" elementFormDefault="qualified" attributeFormDefault="unqualified">
   <xs:element name="Category" type="Category"/>
   <xs:complexType name="Category">
      <xs:sequence>
         <xs:element name="CategoryId" type="xs:integer"/>
         <xs:element name="CategoryName" type="xs:string"/>
         <xs:element name="CategoryDescription" type="xs:string"/>
         <xs:element name="CategoryParentId" type="xs:integer"/>
      </xs:sequence>
   </xs:complexType>
</xs:schema>


5. Operations schema (CategoryOperations.xsd)

Here we have 2 operations : save and getbyId. For both of them we have to define data types for request and response. So, in total we have 4 types :


<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"             xmlns="http://com/demien/springws/domain/operation"            xmlns:category="http://com/demien/springws/domain" targetNamespace="http://com/demien/springws/domain/operation" elementFormDefault="qualified">
   <xsd:import namespace="http://com/demien/springws/domain" schemaLocation="Category.xsd"/>
   <xsd:element name="CategorySaveRequest">
      <xsd:complexType>
         <xsd:sequence>
            <xsd:element name="category" type="category:Category"/>
         </xsd:sequence>
      </xsd:complexType>
   </xsd:element>
   <xsd:element name="CategorySaveResponse">
      <xsd:complexType>
         <xsd:sequence>
            <xsd:element name="category" type="category:Category"/>
         </xsd:sequence>
      </xsd:complexType>
   </xsd:element>

   <xsd:element name="CategoryGetByIdRequest">
      <xsd:complexType>
         <xsd:sequence>
            <xsd:element name="categoryId" type="xsd:integer"/>
         </xsd:sequence>
      </xsd:complexType>
   </xsd:element>
   <xsd:element name="CategoryGetByIdResponse">
      <xsd:complexType>
         <xsd:sequence>
            <xsd:element name="category" type="category:Category"/>
         </xsd:sequence>
      </xsd:complexType>
   </xsd:element>

</xsd:schema>



6. Web service configuration(WSConfig.java)

Here we have 4 beans :
 - messageDispatcherServlet - it will intercept our calls to url /ws/ and interpret them as web service calls
- Wsdl11Definition - it is responsible for showing WSDL - description of our web service. This WSDL file will be automatically generated based on CategoryOperations.xsd file. After all, it will be accessable by path ws/beanName.wsdl  - so in our case :  /ws/CategoryService.wsdl.
- Category schema - it will be accessable by path /ws/Category.xsd
- Category operation schema - it will be accessable by path /ws/CategoryOperations.xsd

package com.demien.springws;

import org.springframework.boot.context.embedded.ServletRegistrationBean;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.ws.config.annotation.EnableWs;
import org.springframework.ws.transport.http.MessageDispatcherServlet;
import org.springframework.ws.wsdl.wsdl11.DefaultWsdl11Definition;
import org.springframework.xml.xsd.SimpleXsdSchema;
import org.springframework.xml.xsd.XsdSchema;


@Configuration@EnableWspublic class WSConfig {

    @Bean    public ServletRegistrationBean messageDispatcherServlet(ApplicationContext applicationContext) {
        MessageDispatcherServlet servlet = new MessageDispatcherServlet();
        servlet.setApplicationContext(applicationContext);
        servlet.setTransformWsdlLocations(true);
        return new ServletRegistrationBean(servlet, "/ws/*");
    }

    @Bean(name = "CategoryService")
    public DefaultWsdl11Definition defaultWsdl11Definition(XsdSchema CategoryOperations) {
        DefaultWsdl11Definition result = new DefaultWsdl11Definition();
        result.setPortTypeName("CategoryWS");
        result.setLocationUri("/ws");
        result.setTargetNamespace("http://com/demien/springws/domain/operation");

        result.setSchema(CategoryOperations);
        return result;
    }

    @Bean    public XsdSchema Category() {
        return new SimpleXsdSchema(new ClassPathResource("Category.xsd"));
    }

    @Bean    public XsdSchema CategoryOperations() {
        return new SimpleXsdSchema(new ClassPathResource("CategoryOperations.xsd"));
    }

}



7. App main file(App.java)

Here everything is pretty simple - just running :
package com.demien.springws;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplicationpublic class App {
    public static void main(String[] args) {
        SpringApplication.run(new Class<?>[]{App.class}, args);
    }
}


8. Service(business logic) class(CategoryService.java)

Here I made "emulation" of service implementation : my services is storing data in HashMap instead of saving in DB and generate ID by increment of static variable :

package com.demien.springws.service;

import com.demien.springws.domain.Category;
import org.springframework.stereotype.Service;
import java.math.BigInteger;
import java.util.HashMap;
import java.util.Map;

@Servicepublic class CategoryService {

    private Map<BigInteger,Category> storage=new HashMap<BigInteger,Category>();
    private int id=1;

    public Category save(Category category) {
        category.setCategoryId(BigInteger.valueOf(id));
        id++;
        storage.put(category.getCategoryId(), category);
        return category;
    }

    public Category getById(BigInteger categoryId) {
        Category result=storage.get(categoryId);
        return result;
    }
}

9. Build app  - for generating sources from schema files

Now development part is almost over  and we need to generate classes from XDS schemas. Everything is defined in our pom.xml, so, all we need is to run :
  mvn clean compile

During compiling we will see in output list of generated files :
[INFO] com\demien\springws\domain\operation\CategoryGetByIdRequest.java
[INFO] com\demien\springws\domain\operation\CategoryGetByIdResponse.java
[INFO] com\demien\springws\domain\operation\CategorySaveRequest.java
[INFO] com\demien\springws\domain\operation\CategorySaveResponse.java
[INFO] com\demien\springws\domain\operation\ObjectFactory.java
[INFO] com\demien\springws\domain\operation\package-info.java
[INFO] com\demien\springws\domain\Category.java
[INFO] com\demien\springws\domain\ObjectFactory.java
[INFO] com\demien\springws\domain\package-info.java


10. Web service endpoint(CategoryServiceEndpoint.java)

All files were generated, now one step more to go: create a web service endpoint  - "wrapper" for CategoryService with needed for endpoint annotations :

package com.demien.springws.service;

import com.demien.springws.domain.Category;
import com.demien.springws.domain.operation.CategoryGetByIdRequest;
import com.demien.springws.domain.operation.CategoryGetByIdResponse;
import com.demien.springws.domain.operation.CategorySaveRequest;
import com.demien.springws.domain.operation.CategorySaveResponse;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.ws.server.endpoint.annotation.Endpoint;
import org.springframework.ws.server.endpoint.annotation.PayloadRoot;
import org.springframework.ws.server.endpoint.annotation.RequestPayload;
import org.springframework.ws.server.endpoint.annotation.ResponsePayload;

@Endpointpublic class CategoryServiceEndpoint {
    private static final String TARGET_NAMESPACE = "http://com/demien/springws/domain/operation";


    @Autowired    CategoryService categoryService;

    public CategoryServiceEndpoint() {
    }

    @PayloadRoot(localPart = "CategorySaveRequest", namespace = TARGET_NAMESPACE)
    public @ResponsePayload    CategorySaveResponse save(@RequestPayload CategorySaveRequest request)       {
        CategorySaveResponse response = new CategorySaveResponse();
        Category category=categoryService.save(request.getCategory());
        response.setCategory(category);
        return response;
     }

    @PayloadRoot(localPart = "CategoryGetByIdRequest", namespace = TARGET_NAMESPACE)
    public @ResponsePayload    CategoryGetByIdResponse save(@RequestPayload CategoryGetByIdRequest request)       {
        CategoryGetByIdResponse response = new CategoryGetByIdResponse();
        Category category=categoryService.getById(request.getCategoryId());
        response.setCategory(category);
        return response;
    }

}


11. Running and testing of our web service using SoapUI 

Now we can run our application(main procedure in App.java class) and check our services.

First of all we can check our resources - by next 2 links we have to be able to open our schemas :
http://localhost:8080/ws/Category.xsd
http://localhost:8080/ws/CategoryOperations.xsd

Next - we can check generation of WSDL file(from CategoryOperations.xsd) by SpringWS:
http://localhost:8080/ws/CategoryService.wsdl

And if everything is OK  - we can call web service from client, for example using SoapUI.
After import of WSDL link to SoapUI - it creates 2 operations : CategorySave and CategoryGetById.

Execution of CategorySave :
In request I filled only CategoryName field:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:oper="http://com/demien/springws/domain/operation" xmlns:dom="http://com/demien/springws/domain">
   <soapenv:Header/>
   <soapenv:Body>
      <oper:CategorySaveRequest>
         <oper:category>
            <dom:CategoryId>?</dom:CategoryId>
            <dom:CategoryName>First Category</dom:CategoryName>
            <dom:CategoryDescription>?</dom:CategoryDescription>
            <dom:CategoryParentId>?</dom:CategoryParentId>
         </oper:category>
      </oper:CategorySaveRequest>
   </soapenv:Body>
</soapenv:Envelope>

Returned response - we can see that ID was generated :
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <SOAP-ENV:Body>
      <ns3:CategorySaveResponse xmlns:ns2="http://com/demien/springws/domain" xmlns:ns3="http://com/demien/springws/domain/operation">
         <ns3:category>
            <ns2:CategoryId>1</ns2:CategoryId>
            <ns2:CategoryName>First Category</ns2:CategoryName>
            <ns2:CategoryDescription>?</ns2:CategoryDescription>
         </ns3:category>
      </ns3:CategorySaveResponse>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>


Execution of CategoryGetById:
Request - I filled CategoryId :
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:oper="http://com/demien/springws/domain/operation">
   <soapenv:Header/>
   <soapenv:Body>
      <oper:CategoryGetByIdRequest>
         <oper:categoryId>1</oper:categoryId>
      </oper:CategoryGetByIdRequest>
   </soapenv:Body>
</soapenv:Envelope>


Returned response - category which was created on previous step :
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <SOAP-ENV:Body>
      <ns3:CategoryGetByIdResponse xmlns:ns2="http://com/demien/springws/domain" xmlns:ns3="http://com/demien/springws/domain/operation">
         <ns3:category>
            <ns2:CategoryId>1</ns2:CategoryId>
            <ns2:CategoryName>First Category</ns2:CategoryName>
            <ns2:CategoryDescription>?</ns2:CategoryDescription>
         </ns3:category>
      </ns3:CategoryGetByIdResponse>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

12. The end. 

Source code can be downloaded from here.