Thursday, November 4, 2021

AWS Secret Manager sample code with golang aws-sdk-go-v2 SDK

AWS Secret Manager provides sample code. It is very good to use for quick start. 

But, I see often, they provide sample code with old version of SDK. 

In this time, I program with Golang. and the aws-sdk-go is old version sdk. 

As soon as I pushe the code, I know, it will be alerted by source scanning software. 

I had to modify the sample code. as below

 

package aws

// Use this code snippet in your app.
// If you need more information about configurations or implementing the sample code, visit the AWS docs:
// https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/setting-up.html

import (
	"context"
	"encoding/base64"
	"errors"
	"fmt"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
	"github.com/aws/smithy-go"
)

func getSecret() {
	secretName := "arn:aws:secretsmanager:eu-central-1:.... put your secretmanager ARN"
	region := "eu-central-1"

	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion(region),
	)
	if err != nil {
		// handle error
	}

	//Create a Secrets Manager client
	svc := secretsmanager.NewFromConfig(cfg)
	input := &secretsmanager.GetSecretValueInput{
		SecretId:     aws.String(secretName),
		VersionStage: aws.String("AWSCURRENT"), // VersionStage defaults to AWSCURRENT if unspecified
	}

	// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
	// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html

	result, err := svc.GetSecretValue(context.TODO(), input)
	if err != nil {

		var apiErr smithy.APIError
		if errors.As(err, &apiErr) {
			code := apiErr.ErrorCode()
			message := apiErr.ErrorMessage()
			// handle error code
			fmt.Println("error code: " + code + " message : " + message)
			return
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return
	}

	// Decrypts secret using the associated KMS CMK.
	// Depending on whether the secret is a string or binary, one of these fields will be populated.
	var secretString, decodedBinarySecret string
	if result.SecretString != nil {
		secretString = *result.SecretString
	} else {
		decodedBinarySecretBytes := make([]byte, base64.StdEncoding.DecodedLen(len(result.SecretBinary)))
		len, err := base64.StdEncoding.Decode(decodedBinarySecretBytes, result.SecretBinary)
		if err != nil {
			fmt.Println("Base64 Decode Error:", err)
			return
		}
		decodedBinarySecret = string(decodedBinarySecretBytes[:len])
	}

	// Your code goes here.
	fmt.Println(secretString)
	fmt.Println(decodedBinarySecret)
}

Friday, December 21, 2018

maven with NTLM Proxy Server

While I am migrating my project to Java 11, I was facing many issues. I was not sure if the 3rd Party Framesworks work fine. I had a doubt. So I tested several Frameworks. So far, Eclipslink and Jmokit was not running fine on Java11. I have reported to Jmokit. Eclipselink has apparently fixed something in the latest code, but the latest version is not released yet. It is laid in Snapshot repository. I am using Company Nexus Repository which I am not allow to change the configuration for adding Snapshot Repository. To access the Repository, I had to configure Proxy in maven setting.xml. But, I was not able to access the Repository yet. I realized the Proxy Server is required NTML based Authentication. Maven doesn't support NTLM Proxy Authentication. It required another way to solve.

I found out that either use CNTLM or wagon-http-lightweight. CNTLM would be good for various reson. If you use several application behind Proxy, like a VirtualBox, Git, Yum, SVN, ... But, I just need for a maven. so I have chosen wagon-http-lightweight.

It is fairly easy to use this. You can download the file from https://mvnrepository.com/artifact/org.apache.maven.wagon/wagon-http-lightweight/2.2 This is just 15 Kbytes. locate the dowonloaded file under M2_HOME/lib/ext
Now, You need to configure the maven xml files. You can use the password as plaintext in the configuration. But, I recommend to use encrypted password for the security. maven will recognize automatically when the password starts with "{" and will decrypt it before authenticate with Proxy Server. I will create a master password and user password by maven. masterpassword will be used for decrypting userpassword with in maven. userpassword is your NTLM userpassword

USER_HOME/.m2/settings-security.xml
mvn -emp masterpassword

{SPg1nt21S2MHuw0Hy8MJaEF7Gc7dK25UWGDYKHupNCw=}
<settingsSecurity>
  <master>{SPg1nt21S2MHuw0Hy8MJaEF7Gc7dK25UWGDYKHupNCw=}</master>
</settingsSecurity>
 


Set up your proxy in USER_HOME/.m2/setting.xml

mvn -ep userpassword
{7ut6v4FFiJMHtwsmYrsmLMcPoDBGmbz/kgcQ6Vks+/0=}

<proxies>
    <proxy>
   <id>internet-proxy</id>
   <active>true</active>
   <protocol>http</protocol>
   <host>###ProxyHost###</host>
   <username>###Username###</username>
   <password>{7ut6v4FFiJMHtwsmYrsmLMcPoDBGmbz/kgcQ6Vks+/0=}</password>
   <port>###Proxy Port###</port>
   <nonProxyHosts>localhost|127.0.0.1</nonProxyHosts>
    </proxy>
 </proxies>  


PS. Security is good. but it makes develpers crazy. I hope there is a simple way to secure and easy to devlop without configure extra. it takes so much time to configure and get it.

Thursday, August 10, 2017

Curl Command example with client Key to https Server

To communicate with https Server by curl command, if a client private key is required, it is not simple like as i thought. my_client.12 is my client key and I need to convert to the other format. Before starting, check your key.
openssl pkcs12 -info -in my_client.12



Check the Server sertificate
openssl s_client -showcerts -connect www.domain.com:443


If you see that all right. you will be able to communicate with the https server which requires client key.

Convert the client key
openssl pkcs12 -in my_client.12 -out client.pem -clcerts -nokeys
openssl pkcs12 -in my_client.12 -out key.pem -nocerts
openssl rsa -in key.pem -out newkey.pem


And now finally, you can communicate with the server.
curl -v -G  -k --key newkey.pem --cert ./client.pem:password https://www.domain.com/path

Download Files from AWS S3 Bucket via SQS message



Our Infrastructure put a log file in S3 bucket. I need to get only this new Log file from the S3 bucket and parse it put into DB. The Log parse Servers are clustered which might be inturruped while downloading and might download again which another cluster server is currently parsing the log file.
There is a solution. When new files are created, the S3 Bucket will send a message to a Simple Queue Service(SQS). The Log Parsing Servers will pull the messages every minute. If a message already pulled, SQS will hide a message in determined period until the delete action. The delete action will be called from parsing server after downloading.
To run the code, you will need a python3 or higher and boto3(Amazon Web Services (AWS) SDK for Python)
yum install epel-release
yum install python34-pip
pip3 install boto3



There is a python3 code sample.
import boto3
import json
import traceback
import os
import logging, sys
import tarfile

#logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

# Get the service resource
sqs = boto3.resource('sqs', region_name='eu-central-1')
s3_client = boto3.client('s3', region_name='eu-central-1')

# Get the queue
queue = sqs.get_queue_by_name(QueueName='Your SQS Queue Name')

for message in queue.receive_messages(MaxNumberOfMessages=10):
  try:
    if message is not None:
      # Parsing event message from s3 bucket
      s3 = json.loads(message.body)['Records'][0]['s3']
      bucket = s3['bucket']['name']
      key = s3['object']['key']
      
      logging.debug('bucket :'+bucket)
      logging.debug('key :'+key)
      
      # Get filename and directory from key
      filename = os.path.basename(key)
      directory = '/your_prefix_dir/' + os.path.dirname(key)
      
      logging.debug('filename :'+filename)
      logging.debug('directory :'+directory)
      
      # Create Directory if it is not exist
      if not os.path.exists(directory):
          os.makedirs(directory)  
                          
      # Download Log File
      s3_client.download_file(bucket, key, directory + filename)
      logging.debug('Download completed')
      
      # Extract tar.gz File
      tar = tarfile.open(directory + filename, "r:gz")
      tar.extractall(directory)
      tar.close()  
      logging.debug('extract file completed')
         
      # Remove tar.gz File
      os.remove(directory + filename)
      
      # Remove SQS message    
      message.delete()
      
  except ValueError:
    # SQS is dedicated for S3 event message. If there is wrong message from other service, leave message body and remove the message
    logging.error('Message format is not valid. Delete message :' + message.body)
    message.delete()
  except Exception:
    logging.error(traceback.format_exc()) 
  else:
    logging.info('finish')



If you see message from the S3 bucket like below command. It is ready to pull the log file by the python script. Watch out, the SQS hide the message default 20 second. it won't be visiable.
aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/your account number/queueName --max-number-of-messages 1 --region eu-central-1

you can excute the python script like below 20 seconds later. flock is for avoiding concurrent excution.
flock -n /home/your Dir/lock/s3-copy.lock -c "/usr/bin/python3 /your_prefix_dir/s3-copy.py"

The log file are downloaded and the tar.gz files are extracted.

Thursday, August 3, 2017

Generate Unique Code for Coupons or Vouchers in java

Please see the Maven Dependency
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>commons-lang</groupId>
   <artifactId>commons-lang</artifactId>
  </dependency>



import java.math.BigInteger;

public class Coupon {

    private byte numberOfChar;

    private BigInteger code;

    private BigInteger crc;

    public byte getNumberOfChar() {
        return numberOfChar;
    }

    public void setNumberOfChar(byte numberOfChar) {
        this.numberOfChar = numberOfChar;
    }

    public BigInteger getCode() {
        return code;
    }

    public void setCode(BigInteger code) {
        this.code = code;
    }

    public BigInteger getCrc() {
        return crc;
    }

    public void setCrc(BigInteger crc) {
        this.crc = crc;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof Coupon)) {
            return false;
        }

        Coupon c = (Coupon) o;
        if (code.equals(c.code)) {
            return true;
        }
        return false;

    }

    @Override
    public int hashCode() {
        int hash = 5;
        hash = 89 * hash + (code != null ? code.hashCode() : 0);
        hash = 89 * hash + (crc != null ? crc.hashCode() : 0);
        return hash;
    }

}




public class CouponGenerateError extends Exception {

    public CouponGenerateError(String message) {
        super(message);
    }

    private static final long serialVersionUID = -8232285755941178115L;

}




import java.math.BigInteger;
import java.security.SecureRandom;
import java.util.LinkedHashSet;
import java.util.Set;

public class CouponGenerator {

    private SecureRandom random = new SecureRandom();

    /**
     * Represent of number of CRC chracter. If it is 1. One Character of CRC code will be generated and attached next to
     * coupon code
     */
    static int CRC_SIZE = 1;

    /**
     * CRC Generator should never be changed unless you like to invalidate all previously published coupons. It is a key
     * value to prove if the CRC value is right or not
     */
    static int CRC_GENERATOR = 31;

    /**
     * @param numberOfChar
     *            is define how long is the Voucher code. Voucher code consists of coupon code and CRC code
     * @param numberOfCoupon
     *            is define how many Vouchers must be generated
     * @return
     * @throws CouponGenerateError
     */
    public Set getCouponSet(byte numberOfChar, long numberOfCoupon) throws CouponGenerateError {

        isValidNumberOfChar(numberOfChar);

        Set couponSet = new LinkedHashSet();

        while (numberOfCoupon > couponSet.size()) {
            couponSet.add(generateCoupon(numberOfChar));
        }

        return couponSet;
    }

    /**
     * @param numberOfChar
     *            It must be bigger than 2 and smaller than 12. If the parameter is 2, it will have 32 cases of coupon
     *            code. It is meaning less to generate. 
* If the parameter is bigger than 12. Internal long type variable can not calculate it. It will make a * byte overflow and return garbage value. Because, Long is 8 byte which is 64 bits. Each coupon chracter * takes 5 bit. 13 Chracters will take 65 bits * @throws CouponGenerateError */ private void isValidNumberOfChar(byte numberOfChar) throws CouponGenerateError { if (numberOfChar < 3 || numberOfChar > 12) { throw new CouponGenerateError( "Invalid numberOfChar for Coupon chracters. It must be bigger than 2 and smaller than 12"); } } /** * @param numberOfChar * length of Alphanumeric code value is defined by numberOfChar * @param numberOfChar * must be bigger than 3 or equal and smaller than 12 or equal chracters * @return * @throws CouponGenerateError */ public Coupon getCoupon(byte numberOfChar) throws CouponGenerateError { isValidNumberOfChar(numberOfChar); return generateCoupon(numberOfChar); } private Coupon generateCoupon(byte numberOfChar) throws CouponGenerateError { Coupon coupon = new Coupon(); coupon.setNumberOfChar(numberOfChar); // Create 5 random bits per character. 5 bits represent Base32 Encoding coupon.setCode(new BigInteger((numberOfChar - CRC_SIZE) * 5, random)); coupon.setCrc(CouponUtil.calculateCrc(coupon)); return coupon; } }



import java.math.BigInteger;

import org.apache.commons.lang.StringUtils;

public class CouponUtil {

    /**
     * @param coupon
     *            
     * 7.  Base 32 Encoding with Extended Hex Alphabet
     * 
     *    The following description of base 32 is derived from [7].  This
     *    encoding may be referred to as "base32hex".  This encoding should not
     *    be regarded as the same as the "base32" encoding and should not be
     *    referred to as only "base32".  This encoding is used by, e.g.,
     *    NextSECure3 (NSEC3) [10].
     * 
     *    One property with this alphabet, which the base64 and base32
     *    alphabets lack, is that encoded data maintains its sort order when
     *    the encoded data is compared bit-wise.
     * 
     *    This encoding is identical to the previous one, except for the
     *    alphabet.  The new alphabet is found in Table 4.
     * 
     *                  Table 4: The "Extended Hex" Base 32 Alphabet
     * 
     *          Value Encoding  Value Encoding  Value Encoding  Value Encoding
     *              0 0             9 9            18 I            27 R
     *              1 1            10 A            19 J            28 S
     *              2 2            11 B            20 K            29 T
     *              3 3            12 C            21 L            30 U
     *              4 4            13 D            22 M            31 V
     *              5 5            14 E            23 N
     *              6 6            15 F            24 O         (pad) =
     *              7 7            16 G            25 P
     *              8 8            17 H            26 Q
     * 
     * @return
     */
    public static String getVoucherString(Coupon coupon) {

        byte numberOfChar = coupon.getNumberOfChar();

        String code = coupon.getCode().toString(32);

        if (!isCodeSizeRight(numberOfChar, code)) {
            code = leftPadding0(numberOfChar, code);
        }
        // Debug purpose
        // System.out.println("code : " + code);
        // System.out.println("bit code : " + coupon.getCode().toString(2));
        // System.out.println("CRC : " + coupon.getCrc().toString(32));
        return code + coupon.getCrc().toString(32);
    }

    private static String leftPadding0(byte numberOfChar, String code) {

        return StringUtils.repeat("0", numberOfChar - CouponGenerator.CRC_SIZE).substring(
                code.length() % numberOfChar)
                + code;

    }

    private static boolean isCodeSizeRight(byte numberOfChar, String code) {
        return code.length() % (numberOfChar - CouponGenerator.CRC_SIZE) == 0;
    }

    /**
     * It validate the CRC value from the coupon.
     * 
     * @param coupon
     * @return
     */
    static boolean isValidCrc(Coupon coupon) {

        if (coupon.getCrc().equals(calculateCrc(coupon))) {
            return true;
        } else {
            return false;
        }

    }

    /**
     * It cancluate CRC value and return it. coupon parameter must be set code value to calculate CRC value.
     * 
     * @param coupon
     * @return
     */
    static BigInteger calculateCrc(Coupon coupon) {

        BigInteger code = coupon.getCode();
        String crcValue = String.valueOf(code.longValue() % CouponGenerator.CRC_GENERATOR);
        // Debug purpose
        // System.out.println("code.toString(2) : " + code.toString(2));
        // System.out.println("code.longValue() : " + code.longValue());
        // System.out.println("crcValue : " + crcValue);
        return new BigInteger(crcValue, 10);
    }
}




import java.math.BigInteger;
import java.util.Set;

import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

public class CouponGeneratorTest {

    static CouponGenerator couponGenerator = new CouponGenerator();

    static Set couponSet;

    @BeforeClass
    public static void init() throws CouponGenerateError {

        couponSet = couponGenerator.getCouponSet((byte) 10, 1000);
    }

    /**
     * Test if the CouponGenerator generate requested number of coupons
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCount() throws Exception {

        Assert.assertEquals("Coupon creation didn't match the requested number of coupon", 1000, couponSet.size());
    }

    /**
     * Test if the CodeGenerator generate exact size of vaucher code chracter size.
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCharacterSize() throws Exception {

        for (Coupon coupon : couponSet) {

            // System.out.println(CouponUtil.getVoucherString(coupon));
            Assert.assertEquals("Coupon Characters didn't match the requested number of character", 10, CouponUtil
                    .getVoucherString(coupon).length());
        }

    }

    /**
     * Test if the generated CRC value passes validation check. 
* If it fails, one of generation or validation is wrong * * @throws Exception */ @Test public void testCrcCreation() throws Exception { for (Coupon coupon : couponSet) { Assert.assertTrue("Checksum value is not valid", CouponUtil.isValidCrc(coupon)); } } /** * Test if the given coupon passes CRC validation check. * * @throws Exception */ @Test public void testCrcValidation() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("f", 32)); Assert.assertTrue("CRC value value is not valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the given coupon fails CRC validation check. * * @throws Exception */ @Test public void testCrcValidation2() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("2", 32)); Assert.assertFalse("CRC value must not be valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the CouponGenerator accept requesting less than 3 chracters or bigger than 12 chracters. In this case, it * must throw Exception */ @Test public void testCouponMinusCharacterSize() { try { couponGenerator.getCoupon((byte) 2); Assert.fail("small charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) -6); Assert.fail("Minus charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) 13); Assert.fail("Bigger than 12 charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } } }


Tuesday, February 16, 2016

Image Resizing by java, Change JPEG image quality

I have seen sample code from others to resize image and change jpeg quality by java. It was pretty good. but, just I didn't like one thing that most of them didn't close the stream of file reading or writing. So, The image files were not able to removed or rewrited. I guess probably, it will cause to create jomby process which is never die unless restart JVM. Thease kind of things must be avoided to make a stable application. It will not be visible for now. but, It will capture the application one day and make difficult to find out why application is not stable.

Add Maven Dependendy to resize image
<dependency>
 <groupId>org.imgscalr</groupId>
 <artifactId>imgscalr-lib</artifactId>
 <version>4.2</version>
</dependency>


import javax.imageio.IIOImage;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.ImageWriteParam;
import javax.imageio.ImageWriter;
import javax.imageio.plugins.jpeg.JPEGImageWriteParam;
import javax.imageio.stream.FileImageOutputStream;

import org.imgscalr.Scalr;
import org.imgscalr.Scalr.Method;
import org.imgscalr.Scalr.Mode;
...

public class ImageUtil {

    public static void downsizeJpg(Path sourcePath, Path targetPath, int targetWidth) throws IOException {

        try (InputStream isSrc = new FileInputStream(sourcePath.toFile())) {

            // Don't Use "ImageIO.read(File file)". It will not close the stream
            // Use Stream to close the resource    
            BufferedImage sourceImage = ImageIO.read(isSrc);

            double width = sourceImage.getWidth();
            double height = sourceImage.getHeight();
            double ratio = width / height;
            int trgHeight = (int) (targetWidth / ratio);

            // If taget width is smaller than source width, start to downsize
            if (targetWidth < width) {

                BufferedImage targetImage = Scalr.resize(sourceImage, Method.QUALITY, Mode.FIT_EXACT, targetWidth,
                        trgHeight);
                saveImage(targetImage, targetPath, 0.7f);

                // prevent upsizing. just copy the source image to target image
            } else {

                saveImage(sourceImage, targetPath, 0.7f);
            }
        }
    }

    private static void saveImage(RenderedImage image, Path targetPath, float quality) throws FileNotFoundException,
            IOException {

        // set compression level 
        JPEGImageWriteParam jpegParams = new JPEGImageWriteParam(null);
        jpegParams.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
        jpegParams.setCompressionQuality(quality);

        final ImageWriter writer = ImageIO.getImageWritersByFormatName("jpg").next();

        try (FileImageOutputStream fios = new FileImageOutputStream(targetPath.toFile())) {

            // specifies where the jpg image has to be written
            writer.setOutput(fios);

            // save the image
            writer.write(null, new IIOImage(image, null, null), jpegParams);
            writer.dispose();
        }

    }
}

quality can be fromo 0f to 1f. 1 is the highst quality and file size get big. 0 is lowest quality and file size get small.

Thursday, May 28, 2015

Print current JVM's Java System Property

It print out all System Properties on console.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.getProperties().list(System.out);
    }
}


Then you pick up one property which you need and use it.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.out.println(System.getProperty("user.home"));
        System.out.println(System.getProperty("file.separator"));

...

        // create file object which is under user's home directory
        File file = new File (System.getProperty("user.home") + System.getProperty("file.separator") + "setting.xml");
    }
}


Referred from : System Properties