Thursday, August 10, 2017

Curl Command example with client Key to https Server

To communicate with https Server by curl command, if a client private key is required, it is not simple like as i thought. my_client.12 is my client key and I need to convert to the other format. Before starting, check your key.
openssl pkcs12 -info -in my_client.12



Check the Server sertificate
openssl s_client -showcerts -connect www.domain.com:443


If you see that all right. you will be able to communicate with the https server which requires client key.

Convert the client key
openssl pkcs12 -in my_client.12 -out client.pem -clcerts -nokeys
openssl pkcs12 -in my_client.12 -out key.pem -nocerts
openssl rsa -in key.pem -out newkey.pem


And now finally, you can communicate with the server.
curl -v -G  -k --key newkey.pem --cert ./client.pem:password https://www.domain.com/path

Download Files from AWS S3 Bucket via SQS message



Our Infrastructure put a log file in S3 bucket. I need to get only this new Log file from the S3 bucket and parse it put into DB. The Log parse Servers are clustered which might be inturruped while downloading and might download again which another cluster server is currently parsing the log file.
There is a solution. When new files are created, the S3 Bucket will send a message to a Simple Queue Service(SQS). The Log Parsing Servers will pull the messages every minute. If a message already pulled, SQS will hide a message in determined period until the delete action. The delete action will be called from parsing server after downloading.
To run the code, you will need a python3 or higher and boto3(Amazon Web Services (AWS) SDK for Python)
yum install epel-release
yum install python34-pip
pip3 install boto3



There is a python3 code sample.
import boto3
import json
import traceback
import os
import logging, sys
import tarfile

#logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

# Get the service resource
sqs = boto3.resource('sqs', region_name='eu-central-1')
s3_client = boto3.client('s3', region_name='eu-central-1')

# Get the queue
queue = sqs.get_queue_by_name(QueueName='Your SQS Queue Name')

for message in queue.receive_messages(MaxNumberOfMessages=10):
  try:
    if message is not None:
      # Parsing event message from s3 bucket
      s3 = json.loads(message.body)['Records'][0]['s3']
      bucket = s3['bucket']['name']
      key = s3['object']['key']
      
      logging.debug('bucket :'+bucket)
      logging.debug('key :'+key)
      
      # Get filename and directory from key
      filename = os.path.basename(key)
      directory = '/your_prefix_dir/' + os.path.dirname(key)
      
      logging.debug('filename :'+filename)
      logging.debug('directory :'+directory)
      
      # Create Directory if it is not exist
      if not os.path.exists(directory):
          os.makedirs(directory)  
                          
      # Download Log File
      s3_client.download_file(bucket, key, directory + filename)
      logging.debug('Download completed')
      
      # Extract tar.gz File
      tar = tarfile.open(directory + filename, "r:gz")
      tar.extractall(directory)
      tar.close()  
      logging.debug('extract file completed')
         
      # Remove tar.gz File
      os.remove(directory + filename)
      
      # Remove SQS message    
      message.delete()
      
  except ValueError:
    # SQS is dedicated for S3 event message. If there is wrong message from other service, leave message body and remove the message
    logging.error('Message format is not valid. Delete message :' + message.body)
    message.delete()
  except Exception:
    logging.error(traceback.format_exc()) 
  else:
    logging.info('finish')



If you see message from the S3 bucket like below command. It is ready to pull the log file by the python script. Watch out, the SQS hide the message default 20 second. it won't be visiable.
aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/your account number/queueName --max-number-of-messages 1 --region eu-central-1

you can excute the python script like below 20 seconds later. flock is for avoiding concurrent excution.
flock -n /home/your Dir/lock/s3-copy.lock -c "/usr/bin/python3 /your_prefix_dir/s3-copy.py"

The log file are downloaded and the tar.gz files are extracted.

Thursday, August 3, 2017

Generate Unique Code for Coupons or Vouchers in java

Please see the Maven Dependency
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>commons-lang</groupId>
   <artifactId>commons-lang</artifactId>
  </dependency>



import java.math.BigInteger;

public class Coupon {

    private byte numberOfChar;

    private BigInteger code;

    private BigInteger crc;

    public byte getNumberOfChar() {
        return numberOfChar;
    }

    public void setNumberOfChar(byte numberOfChar) {
        this.numberOfChar = numberOfChar;
    }

    public BigInteger getCode() {
        return code;
    }

    public void setCode(BigInteger code) {
        this.code = code;
    }

    public BigInteger getCrc() {
        return crc;
    }

    public void setCrc(BigInteger crc) {
        this.crc = crc;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof Coupon)) {
            return false;
        }

        Coupon c = (Coupon) o;
        if (code.equals(c.code)) {
            return true;
        }
        return false;

    }

    @Override
    public int hashCode() {
        int hash = 5;
        hash = 89 * hash + (code != null ? code.hashCode() : 0);
        hash = 89 * hash + (crc != null ? crc.hashCode() : 0);
        return hash;
    }

}




public class CouponGenerateError extends Exception {

    public CouponGenerateError(String message) {
        super(message);
    }

    private static final long serialVersionUID = -8232285755941178115L;

}




import java.math.BigInteger;
import java.security.SecureRandom;
import java.util.LinkedHashSet;
import java.util.Set;

public class CouponGenerator {

    private SecureRandom random = new SecureRandom();

    /**
     * Represent of number of CRC chracter. If it is 1. One Character of CRC code will be generated and attached next to
     * coupon code
     */
    static int CRC_SIZE = 1;

    /**
     * CRC Generator should never be changed unless you like to invalidate all previously published coupons. It is a key
     * value to prove if the CRC value is right or not
     */
    static int CRC_GENERATOR = 31;

    /**
     * @param numberOfChar
     *            is define how long is the Voucher code. Voucher code consists of coupon code and CRC code
     * @param numberOfCoupon
     *            is define how many Vouchers must be generated
     * @return
     * @throws CouponGenerateError
     */
    public Set getCouponSet(byte numberOfChar, long numberOfCoupon) throws CouponGenerateError {

        isValidNumberOfChar(numberOfChar);

        Set couponSet = new LinkedHashSet();

        while (numberOfCoupon > couponSet.size()) {
            couponSet.add(generateCoupon(numberOfChar));
        }

        return couponSet;
    }

    /**
     * @param numberOfChar
     *            It must be bigger than 2 and smaller than 12. If the parameter is 2, it will have 32 cases of coupon
     *            code. It is meaning less to generate. 
* If the parameter is bigger than 12. Internal long type variable can not calculate it. It will make a * byte overflow and return garbage value. Because, Long is 8 byte which is 64 bits. Each coupon chracter * takes 5 bit. 13 Chracters will take 65 bits * @throws CouponGenerateError */ private void isValidNumberOfChar(byte numberOfChar) throws CouponGenerateError { if (numberOfChar < 3 || numberOfChar > 12) { throw new CouponGenerateError( "Invalid numberOfChar for Coupon chracters. It must be bigger than 2 and smaller than 12"); } } /** * @param numberOfChar * length of Alphanumeric code value is defined by numberOfChar * @param numberOfChar * must be bigger than 3 or equal and smaller than 12 or equal chracters * @return * @throws CouponGenerateError */ public Coupon getCoupon(byte numberOfChar) throws CouponGenerateError { isValidNumberOfChar(numberOfChar); return generateCoupon(numberOfChar); } private Coupon generateCoupon(byte numberOfChar) throws CouponGenerateError { Coupon coupon = new Coupon(); coupon.setNumberOfChar(numberOfChar); // Create 5 random bits per character. 5 bits represent Base32 Encoding coupon.setCode(new BigInteger((numberOfChar - CRC_SIZE) * 5, random)); coupon.setCrc(CouponUtil.calculateCrc(coupon)); return coupon; } }



import java.math.BigInteger;

import org.apache.commons.lang.StringUtils;

public class CouponUtil {

    /**
     * @param coupon
     *            
     * 7.  Base 32 Encoding with Extended Hex Alphabet
     * 
     *    The following description of base 32 is derived from [7].  This
     *    encoding may be referred to as "base32hex".  This encoding should not
     *    be regarded as the same as the "base32" encoding and should not be
     *    referred to as only "base32".  This encoding is used by, e.g.,
     *    NextSECure3 (NSEC3) [10].
     * 
     *    One property with this alphabet, which the base64 and base32
     *    alphabets lack, is that encoded data maintains its sort order when
     *    the encoded data is compared bit-wise.
     * 
     *    This encoding is identical to the previous one, except for the
     *    alphabet.  The new alphabet is found in Table 4.
     * 
     *                  Table 4: The "Extended Hex" Base 32 Alphabet
     * 
     *          Value Encoding  Value Encoding  Value Encoding  Value Encoding
     *              0 0             9 9            18 I            27 R
     *              1 1            10 A            19 J            28 S
     *              2 2            11 B            20 K            29 T
     *              3 3            12 C            21 L            30 U
     *              4 4            13 D            22 M            31 V
     *              5 5            14 E            23 N
     *              6 6            15 F            24 O         (pad) =
     *              7 7            16 G            25 P
     *              8 8            17 H            26 Q
     * 
     * @return
     */
    public static String getVoucherString(Coupon coupon) {

        byte numberOfChar = coupon.getNumberOfChar();

        String code = coupon.getCode().toString(32);

        if (!isCodeSizeRight(numberOfChar, code)) {
            code = leftPadding0(numberOfChar, code);
        }
        // Debug purpose
        // System.out.println("code : " + code);
        // System.out.println("bit code : " + coupon.getCode().toString(2));
        // System.out.println("CRC : " + coupon.getCrc().toString(32));
        return code + coupon.getCrc().toString(32);
    }

    private static String leftPadding0(byte numberOfChar, String code) {

        return StringUtils.repeat("0", numberOfChar - CouponGenerator.CRC_SIZE).substring(
                code.length() % numberOfChar)
                + code;

    }

    private static boolean isCodeSizeRight(byte numberOfChar, String code) {
        return code.length() % (numberOfChar - CouponGenerator.CRC_SIZE) == 0;
    }

    /**
     * It validate the CRC value from the coupon.
     * 
     * @param coupon
     * @return
     */
    static boolean isValidCrc(Coupon coupon) {

        if (coupon.getCrc().equals(calculateCrc(coupon))) {
            return true;
        } else {
            return false;
        }

    }

    /**
     * It cancluate CRC value and return it. coupon parameter must be set code value to calculate CRC value.
     * 
     * @param coupon
     * @return
     */
    static BigInteger calculateCrc(Coupon coupon) {

        BigInteger code = coupon.getCode();
        String crcValue = String.valueOf(code.longValue() % CouponGenerator.CRC_GENERATOR);
        // Debug purpose
        // System.out.println("code.toString(2) : " + code.toString(2));
        // System.out.println("code.longValue() : " + code.longValue());
        // System.out.println("crcValue : " + crcValue);
        return new BigInteger(crcValue, 10);
    }
}




import java.math.BigInteger;
import java.util.Set;

import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

public class CouponGeneratorTest {

    static CouponGenerator couponGenerator = new CouponGenerator();

    static Set couponSet;

    @BeforeClass
    public static void init() throws CouponGenerateError {

        couponSet = couponGenerator.getCouponSet((byte) 10, 1000);
    }

    /**
     * Test if the CouponGenerator generate requested number of coupons
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCount() throws Exception {

        Assert.assertEquals("Coupon creation didn't match the requested number of coupon", 1000, couponSet.size());
    }

    /**
     * Test if the CodeGenerator generate exact size of vaucher code chracter size.
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCharacterSize() throws Exception {

        for (Coupon coupon : couponSet) {

            // System.out.println(CouponUtil.getVoucherString(coupon));
            Assert.assertEquals("Coupon Characters didn't match the requested number of character", 10, CouponUtil
                    .getVoucherString(coupon).length());
        }

    }

    /**
     * Test if the generated CRC value passes validation check. 
* If it fails, one of generation or validation is wrong * * @throws Exception */ @Test public void testCrcCreation() throws Exception { for (Coupon coupon : couponSet) { Assert.assertTrue("Checksum value is not valid", CouponUtil.isValidCrc(coupon)); } } /** * Test if the given coupon passes CRC validation check. * * @throws Exception */ @Test public void testCrcValidation() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("f", 32)); Assert.assertTrue("CRC value value is not valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the given coupon fails CRC validation check. * * @throws Exception */ @Test public void testCrcValidation2() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("2", 32)); Assert.assertFalse("CRC value must not be valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the CouponGenerator accept requesting less than 3 chracters or bigger than 12 chracters. In this case, it * must throw Exception */ @Test public void testCouponMinusCharacterSize() { try { couponGenerator.getCoupon((byte) 2); Assert.fail("small charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) -6); Assert.fail("Minus charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) 13); Assert.fail("Bigger than 12 charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } } }


Tuesday, February 16, 2016

Image Resizing by java, Change JPEG image quality

I have seen sample code from others to resize image and change jpeg quality by java. It was pretty good. but, just I didn't like one thing that most of them didn't close the stream of file reading or writing. So, The image files were not able to removed or rewrited. I guess probably, it will cause to create jomby process which is never die unless restart JVM. Thease kind of things must be avoided to make a stable application. It will not be visible for now. but, It will capture the application one day and make difficult to find out why application is not stable.

Add Maven Dependendy to resize image
<dependency>
 <groupId>org.imgscalr</groupId>
 <artifactId>imgscalr-lib</artifactId>
 <version>4.2</version>
</dependency>


import javax.imageio.IIOImage;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.ImageWriteParam;
import javax.imageio.ImageWriter;
import javax.imageio.plugins.jpeg.JPEGImageWriteParam;
import javax.imageio.stream.FileImageOutputStream;

import org.imgscalr.Scalr;
import org.imgscalr.Scalr.Method;
import org.imgscalr.Scalr.Mode;
...

public class ImageUtil {

    public static void downsizeJpg(Path sourcePath, Path targetPath, int targetWidth) throws IOException {

        try (InputStream isSrc = new FileInputStream(sourcePath.toFile())) {

            // Don't Use "ImageIO.read(File file)". It will not close the stream
            // Use Stream to close the resource    
            BufferedImage sourceImage = ImageIO.read(isSrc);

            double width = sourceImage.getWidth();
            double height = sourceImage.getHeight();
            double ratio = width / height;
            int trgHeight = (int) (targetWidth / ratio);

            // If taget width is smaller than source width, start to downsize
            if (targetWidth < width) {

                BufferedImage targetImage = Scalr.resize(sourceImage, Method.QUALITY, Mode.FIT_EXACT, targetWidth,
                        trgHeight);
                saveImage(targetImage, targetPath, 0.7f);

                // prevent upsizing. just copy the source image to target image
            } else {

                saveImage(sourceImage, targetPath, 0.7f);
            }
        }
    }

    private static void saveImage(RenderedImage image, Path targetPath, float quality) throws FileNotFoundException,
            IOException {

        // set compression level 
        JPEGImageWriteParam jpegParams = new JPEGImageWriteParam(null);
        jpegParams.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
        jpegParams.setCompressionQuality(quality);

        final ImageWriter writer = ImageIO.getImageWritersByFormatName("jpg").next();

        try (FileImageOutputStream fios = new FileImageOutputStream(targetPath.toFile())) {

            // specifies where the jpg image has to be written
            writer.setOutput(fios);

            // save the image
            writer.write(null, new IIOImage(image, null, null), jpegParams);
            writer.dispose();
        }

    }
}

quality can be fromo 0f to 1f. 1 is the highst quality and file size get big. 0 is lowest quality and file size get small.

Thursday, May 28, 2015

Print current JVM's Java System Property

It print out all System Properties on console.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.getProperties().list(System.out);
    }
}


Then you pick up one property which you need and use it.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.out.println(System.getProperty("user.home"));
        System.out.println(System.getProperty("file.separator"));

...

        // create file object which is under user's home directory
        File file = new File (System.getProperty("user.home") + System.getProperty("file.separator") + "setting.xml");
    }
}


Referred from : System Properties

Wednesday, May 20, 2015

a Java Class which scans ports in an IP

I had to run port scanner software for security check reason, before deliver the product. I tried to get one utilillity from internet. but, the Websites were blocked in my company to download. so, I just made a Java Class which can do the port scanning. it takes around 20 min to run throught all the ports. If I increase Thread size and shorten the timeout, it will take less then 5 min.

Fornow, I have a little problem. When I let it run quickly, Firewall or security software get activated and deny to response if the port is opened. So, I had to let it run slowly to find all the opened port.

package com.devtrigger;

import java.net.InetSocketAddress;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;

class PortScanner {

    public static void main(final String... args) throws InterruptedException, ExecutionException {
        final ExecutorService es = Executors.newFixedThreadPool(15);

        final String ip = "127.0.0.1";

        final int timeout = 200;
        final List<Future<ScanResult>> futures = new ArrayList<>();
        for (int port = 1; port <= 65535; port++) {
            futures.add(portIsOpen(es, ip, port, timeout));
        }
        es.awaitTermination(200L, TimeUnit.MILLISECONDS);
        int openPorts = 0;
        for (final Future<ScanResult> f : futures) {
            if (f.get().isOpen()) {
                openPorts++;
                System.out.println(f.get().getPort());
            }
        }
        System.out.println("There are " + openPorts + " open ports on host " + ip + " (probed with a timeout of "
                + timeout + "ms)");
    }

    public static Future<ScanResult> portIsOpen(final ExecutorService es, final String ip, final int port,
            final int timeout) {
        return es.submit(new Callable<ScanResult>() {
            @Override
            public ScanResult call() {
                try {
                    Socket socket = new Socket();
                    socket.connect(new InetSocketAddress(ip, port), timeout);
                    socket.close();
                    return new ScanResult(port, true);
                } catch (Exception ex) {
                    return new ScanResult(port, false);
                }
            }
        });
    }

    public static class ScanResult {
        private int port;

        private boolean isOpen;

        public ScanResult(int port, boolean isOpen) {
            super();
            this.port = port;
            this.isOpen = isOpen;
        }

        public int getPort() {
            return port;
        }

        public void setPort(int port) {
            this.port = port;
        }

        public boolean isOpen() {
            return isOpen;
        }

        public void setOpen(boolean isOpen) {
            this.isOpen = isOpen;
        }

    }
}

Friday, February 6, 2015

How to create and run Apache JMeter Test Scripts from a Java program


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

 <modelVersion>4.0.0</modelVersion>

 <groupId>myportal</groupId>
 <artifactId>loadtest</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>

 <name>Zug Portal Load Test Tool</name>
 <url>http://maven.apache.org</url>

 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.12</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.apache.jmeter</groupId>
   <artifactId>ApacheJMeter_http</artifactId>
   <version>2.11</version>
  </dependency>

 </dependencies>
</project>
package myportal.loadtest;

import java.net.URL;

import org.apache.jmeter.control.LoopController;
import org.apache.jmeter.engine.StandardJMeterEngine;
import org.apache.jmeter.protocol.http.sampler.HTTPSampler;
import org.apache.jmeter.testelement.TestElement;
import org.apache.jmeter.testelement.TestPlan;
import org.apache.jmeter.threads.SetupThreadGroup;
import org.apache.jmeter.util.JMeterUtils;
import org.apache.jorphan.collections.HashTree;

public class SampleJmeter {

    public static void main(String[] args) {
        // Engine
        StandardJMeterEngine jm = new StandardJMeterEngine();
        URL property = SampleJmeter.class.getClassLoader().getResource("jmeter.properties");

        // jmeter.properties
        JMeterUtils.loadJMeterProperties(property.getPath());

        HashTree hashTree = new HashTree();

        // HTTP Sampler
        HTTPSampler httpSampler = new HTTPSampler();
        httpSampler.setDomain("www.google.com");
        httpSampler.setPort(80);
        httpSampler.setPath("/");
        httpSampler.setMethod("GET");

        // Loop Controller
        TestElement loopCtrl = new LoopController();
        ((LoopController) loopCtrl).setLoops(1);
        ((LoopController) loopCtrl).addTestElement(httpSampler);
        ((LoopController) loopCtrl).setFirst(true);

        // Thread Group
        SetupThreadGroup threadGroup = new SetupThreadGroup();
        threadGroup.setNumThreads(1);
        threadGroup.setRampUp(1);
        threadGroup.setSamplerController((LoopController) loopCtrl);

        // Test plan
        TestPlan testPlan = new TestPlan("MY TEST PLAN");

        hashTree.add("testPlan", testPlan);
        hashTree.add("loopCtrl", loopCtrl);
        hashTree.add("threadGroup", threadGroup);
        hashTree.add("httpSampler", httpSampler);

        jm.configure(hashTree);

        jm.run();
    }
}

meter.properties file is from the JMeter installation /bin directory.