Popular Posts

Hazelcast Persistence with MySQL

What is hazelcast?

What is hazelcast?

Java collections are used to store data in a different formats. like List, Set and Map etc. but these are restricted to single Thread. :(

What if we want to share same collection in different Threads we go for concurrent implementations like ConcurrentHashMap.

How do we share the same collection Object across JVMs? the answer is Hazelcast.

Tip
hazelcast is a in-memory datagrid based on java. data is evenly distributed among the nodes. This allows for horizontal scalability both in terms of available storage space and processing power. Hence it being used as most popular cache framework for java based applications. It is highly available for distributed cache for applications.

Why Hazelcast?

There are five main reasons why i prefer to go for hazelcast

  1. One of the main features of Hazelcast is not having a master node. Each node in the cluster is configured to be the same in terms of functionality. The oldest node manages the cluster members, i.e. automatically performs the data assignment to nodes. When a new node joins to the cluster or a node goes down, this data assignment is repeated across the nodes and the data distribution comes to a balance again.

  2. Another main feature is the data being persisted entirely in-memory. This is fast. In the case of a failure, such as a node crash, no data will be lost since Hazelcast keeps copies of data across all the nodes of cluster. Data is kept in partition slices and each partition slice is owned by a node and backed up on another node.

  3. There is an option to persist data permanently to relational database like MySQL. this we will cover more in the next sections.

  4. Hazelcast supports a number of distributed collections and features.like Maps ,List, Set and Queue.

  5. It is open source and entire framework comes in a small JAR file which you can easily add to your classpath. It also supports client server model.

Getting Started with (IMDG)

WARN: In computing, Hazelcast IMDG is an open source in-memory data grid based on Java

  • Make sure you have JDK installed and configured in your system.

  • download hazelcast binary distribution (IMDG) from https://hazelcast.org/download/#imdg (3.12.1.zip is the latest)

  • Extract the archive and go to lib folder which contains different jars file

  • From bin folder and run the script file and console look like this.

  • This script basically runs the (com.hazelcast.core.server.StartServer) java class (which runs as server)

  • There is another way we can run or connect to the hazelcast using below class com.hazelcast.console.ConsoleApp ( on hazelcast-3.5 )

Run the below command and could see the terminal output where you can perform some operations as well . (create / add /delete data into collection)

ConsoleApp.sh
tvajjala$java -cp hazelcast-3.12.1.jar  com.hazelcast.console.ConsoleApp


Members {size:2, ver:2} [
Member [192.168.1.10]:5701 - a6750beb-a83b-4edd-afd0-52173a69a15b this
Member [192.168.1.10]:5702 - 03e0bb99-039a-4563-9248-94e00cdaedc7
]

Jul 05, 2019 10:18:39 PM com.hazelcast.internal.cluster.ClusterService
INFO: [192.168.1.10]:5702 [dev] [3.12.1]

Members {size:2, ver:2} [
Member [192.168.1.10]:5701 - a6750beb-a83b-4edd-afd0-52173a69a15b
Member [192.168.1.10]:5702 - 03e0bb99-039a-4563-9248-94e00cdaedc7 this
]

hazelcast[default] > jvm
Memory max: 1820MB
Memory free: 127MB 6%
Used Memory:9MB
# procs: 4
OS info: x86_64 Mac OS X 10.14.5
JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.151-b12

hazelcast[default] > whoami
Member [192.168.1.10]:5702 - 03e0bb99-039a-4563-9248-94e00cdaedc7 this
hazelcast[default] >

Similar to object reference in java programing, hazelcast uses the concept called namespace.

when first time start console it will have namespace value default. below are some simple commands to know about your cluster information via console

namespace.sh
hazelcast[storage] > ns warehouse
namespace: warehouse
hazelcast[warehouse] > m.put BOOKS 10
null
hazelcast[warehouse] > m.put PENCILS 100
null
hazelcast[warehouse] > m.put PENS 50
null
hazelcast[warehouse] > m.entries
PENCILS: 100
PENS: 50
BOOKS: 10
Total 3
hazelcast[warehouse] > m.iterator
1 BOOKS
2 PENS
3 PENCILS
hazelcast[warehouse] > m.values
100
50
10
Total 3
hazelcast[warehouse] > m.keys
BOOKS
PENS
PENCILS
Total 3
Tip
Hazelcast supports different types of collections similar to Java. these can be accesible using below shortcuts
  • m - Map

  • mm - MultiMap

  • l - List

  • s - Set

  • q - Queue

Run below command to print all the supported operations on these collections.

help.sh
hazelcast[default] > help

Java Client

We can write Java client to read entries that are added through terminal

HazelcastClient.java
package com.tvajjala.documentation;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IMap;

/**
* @author ThirupathiReddy Vajjala
*/
public class HazelcastClient {

public static void main(String[] args) {


    ClientConfig clientConfig=new ClientConfig();
    clientConfig.getGroupConfig().setName("dev");
    clientConfig.getNetworkConfig().addAddress("192.168.1.10");

    HazelcastInstance client= HazelcastClient.newHazelcastClient(clientConfig);

    IMap<String,String> map= client.getMap("warehouse");

    map.entrySet().stream().forEach(e-> System.out.println(e.getKey() + "  =  "+ e.getValue()));

    client.getLifecycleService().shutdown();//shutdown client
}
}

Adding spring flavor

add following dependencies

filename.xml
<!-- https://mvnrepository.com/artifact/com.hazelcast/hazelcast-spring -->
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-spring</artifactId>
    <version>3.12.1</version>
</dependency>


<!-- https://mvnrepository.com/artifact/com.hazelcast/hazelcast -->
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>3.12.1</version>
</dependency>

Persisting in-memory datastore into MySQL

MySQLMapStore.java
package com.trvajjala.model.store;
import java.io.ByteArrayInputStream;
import java.io.ObjectInputStream;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.log4j.Logger;
import org.springframework.dao.DataAccessException;
import org.springframework.dao.EmptyResultDataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.PreparedStatementCallback;
import com.hazelcast.core.MapStore;


/**
 *
 * mapstore will persist into mysql Database
 *
 */
public class MySQLMapStore implements MapStore<String, Object>, SQLQuery {


 private static Logger logger = Logger.getLogger(MySQLMapStore.class
   .getSimpleName());
 {
  logger.info("Hazelcast MySQL MapStore Instance Created. ");
 }
 /**
  * jdbcTemplate will handle DB connection
  */
 private JdbcTemplate jdbcTemplate;
 /**
  * setting method injected through IOC
  *
  * @param jdbcTemplate
  */
 public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
  this.jdbcTemplate = jdbcTemplate;
 }

 /**
  * load object from Database using given KEY
  */
 public Object load(String key) {
  logger.info(" loading persisted entry " + key);
  try {
   byte[] bytes = jdbcTemplate.queryForObject(SELECT_QUERY,
     new Object[] { key }, byte[].class);
   return deserialize(bytes);
  } catch (EmptyResultDataAccessException e) {
   return null;
  }
 }
 public Map<String, Object> loadAll(Collection<String> keys) {
  Map<String, Object> map = new HashMap<String, Object>();
  for (Iterator<String> iterator = keys.iterator(); iterator.hasNext();) {
   String map_key = iterator.next();
   Object map_value = load(map_key);
   map.put(map_key, map_value);
  }
  logger.info(" total " + map.size() + " objects loaded ");
  return map;
 }


 public Set<String> loadAllKeys() {
  List<String> list = jdbcTemplate.queryForList(ALL_KEYS, String.class);
  logger.info(" total keys found " + list.size());
  return new HashSet<String>(list);
 }


 public void delete(final String key) {
  jdbcTemplate.execute(DELETE_KEY,
    new PreparedStatementCallback<Boolean>() {
     public Boolean doInPreparedStatement(PreparedStatement ps)
       throws SQLException, DataAccessException {
      ps.setString(1, key);
      return ps.execute();
     }
    });
  logger.info(" entry with key " + key + "deleted ");
 }


 public void deleteAll(Collection<String> keys) {


  for (Iterator<String> iterator = keys.iterator(); iterator.hasNext();) {
   String map_key = iterator.next();
   delete(map_key);
  }


  if (!keys.isEmpty()) {
   logger.info(" total " + keys.size() + "objects deleted. ");
  }
 }


 /**
  * store method save object into DB . if KEY already exist it will update
  * the record with new value
  */
 public void store(final String key, final Object value) {


  try {


   jdbcTemplate.execute(INSERT_QUERY,
     new PreparedStatementCallback<Boolean>() {


      public Boolean doInPreparedStatement(
        PreparedStatement ps) throws SQLException,
        DataAccessException {
       ps.setString(1, key);
       ps.setObject(2, value);
       return ps.execute();
      }


     });


   logger.info(" entry persisted successfully [ " + key + " = "
     + value + " ]");


  } catch (org.springframework.dao.DuplicateKeyException dke) {
   logger.info(" updating  value with existing key " + key);
   update(key, value);
  }
 }


 private void update(final String key, final Object value) {


  jdbcTemplate.execute(UPDATE_QUERY,
    new PreparedStatementCallback<Boolean>() {


     public Boolean doInPreparedStatement(PreparedStatement ps)
       throws SQLException, DataAccessException {
      ps.setObject(1, value);
      ps.setString(2, key);
      return ps.execute();
     }
    });
 }


 public void storeAll(Map<String, Object> map) {


  Set<Entry<String, Object>> set = map.entrySet();
  for (Iterator<Entry<String, Object>> iterator = set.iterator(); iterator
    .hasNext();) {
   Entry<String, Object> entry = iterator.next();
   store(entry.getKey(), entry.getValue());
  }


  logger.info(" total  " + map.size() + " objects saved.");


 }


 /**
  * convert byte stream into Object
  *
  * @param objectBytes
  * @return
  */
 public Object deserialize(byte[] objectBytes) {


  try {
   ObjectInputStream objectIn = new ObjectInputStream(
     new ByteArrayInputStream(objectBytes));


   return objectIn.readObject();
  } catch (Exception e) {
   return null;
  }
 }


}

Queries are placed under below interface

SQLQuery.java
package com.trvajjala.model.store;


/**
* MySQL Queries which are used in MySQLMapStore
*
*/

public interface SQLQuery {

    String INSERT_QUERY = "INSERT INTO tbl_hazelmap(map_key,map_value) values(?,?)";
    String UPDATE_QUERY = "UPDATE tbl_hazelmap SET map_value=? WHERE map_key=?";
    String SELECT_QUERY = "SELECT map_value FROM tbl_hazelmap WHERE map_key=?";
    String ALL_KEYS = "SELECT map_key FROM  tbl_hazelmap";
    String DELETE_KEY = "DELETE FROM tbl_hazelmap WHERE map_key=?";

}

You can have DB script as show below to support any kind Map value . or change db datatypes as per your requirement

sql
CREATE DATABASE  IF NOT EXISTS `hazel_db`;
USE `hazel_db`;


CREATE TABLE `tbl_hazelmap` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `map_key` varchar(45) DEFAULT NULL,
  `map_value` blob,
  PRIMARY KEY (`id`),
  UNIQUE KEY `map_key_UNIQUE` (`map_key`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET='utf8'
spring-hazelcast.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
 xmlns:hz="http://www.hazelcast.com/schema/spring"
 xsi:schemaLocation="http://www.springframework.org/schema/context
          http://www.springframework.org/schema/context/spring-context-3.0.xsd
          http://www.springframework.org/schema/beans
          http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
          http://www.hazelcast.com/schema/spring
          http://www.hazelcast.com/schema/spring/hazelcast-spring-3.2.xsd">


 <context:property-placeholder location="classpath:hazel-server.properties" />


 <hz:hazelcast id="instance">
  <hz:config>
   <hz:group name="${hazel.username}" password="${hazel.password}" />
   <hz:properties>
    <hz:property name="hazelcast.merge.first.run.delay.seconds">5</hz:property>
    <hz:property name="hazelcast.merge.next.run.delay.seconds">5</hz:property>
    <hz:property name="hazelcast.logging.type">log4j</hz:property>
    <!-- reference : http://hazelcast.org/docs/latest/manual/html/logging.html -->
   </hz:properties>


   <hz:network port="${hazel.network.port}"
    port-auto-increment="${hazel.port.autoincrement}">
    <hz:join>
     <hz:multicast enabled="false" />
     <hz:tcp-ip connection-timeout-seconds="5"
      enabled="${hazel.tcp.enabled}">
      <hz:member>${hazel.tcp.member}</hz:member>
     </hz:tcp-ip>
    </hz:join>
   </hz:network>


   <hz:map name="store" backup-count="1" in-memory-format="BINARY"
    statistics-enabled="true">
    <hz:map-store enabled="true" implementation="mysqlMapStore"
     write-delay-seconds="${hazel.mapstore.delay}">
    </hz:map-store>
   </hz:map>

  </hz:config>
 </hz:hazelcast>


 <hz:map instance-ref="instance" id="store" name="store" />





 <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
  <property name="dataSource" ref="dataSource" />
 </bean>


 <bean name="dataSource"
  class="org.springframework.jdbc.datasource.DriverManagerDataSource">
  <property name="driverClassName" value="${db.driverClassName}" />
  <property name="url" value="${db.url}" />
  <property name="username" value="${db.username}" />
  <property name="password" value="${db.password}" />
 </bean>


 <bean id="mysqlMapStore" class="com.tvajjala.model.store.MySQLMapStore">
  <property name="jdbcTemplate" ref="jdbcTemplate" />
 </bean>


</beans>
hazel-server.properties
#DB PROPERTIES
db.driverClassName=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/hazel_db
db.username=root
db.password=XXXXX




#HAZELCAST PROPERTIES
hazel.username=dev
hazel.password=dev
hazel.tcp.enabled=true
hazel.tcp.member=127.0.0.1
hazel.network.port=5701
hazel.port.autoincrement=true


#mysql store delay seconds
hazel.mapstore.delay=1

Now all the map entries that you store into hazelcast map instance will be persisted into DB with 1 sec delay.

No comments:

Post a Comment