Monday, 8 March 2021

How to copy the files/directories into Kubernetes (K8S) Pods from local to remote pods (either ways)

Sometimes, we may get into situations where we need files from Kubernetes (K8S) pods to local file system & vice versa.

K8S has built-in commands for copying resources into & from K8S pods.

Case 1: To copy into local system from remote pods

kubectl cp <namespace_of_pod>/<pod_generated_name>:<source_copying_dir_path> <destination_local_dir_path>

kubectl cp catalog /catalog-app-es1xy2323sds:/appl/repositories     ~/

(here, I am copying the directory /appl/repositories from remote pod named catalog-app-es1xy2323sds (generated name) from Kubernetes cluster to local root path ~/ 

Case 2: To copy from local system to remote pods

kubectl cp <source_local_dir_path> <namespace_of_pod>/<pod_generated_name>:<destination_copying_dir_path> 

kubectl cp  ~/config  catalog /catalog-app-es1xy2323sds:/appl/repositories 

(here, I am copying the directory ~/config from local file system to remote pod named catalog-app-es1xy2323sds (generated name) in the namespace catalog)

you can search for all the namespaces using

kubectl get namespaces

you can search all the pods for a particular namespace using

kubectl get pods -n <namespace_name>

How to extract a TAR file in Linux / unix OS or on MAC

You can use tar command to extract, create, list, update, add/replace the files in the given .tar compressed file.

tar command provides the various options:

-t          List

-x         Extract

-c         Create

-r        Add/Replace


-v        Verbose

-f        <filename> Location of archieve


For Example, to list the contents/files in an given .tar file

tar -tf <.tar filename>            Lists all the files

tar -xvf <.tar filename>        Extracts all the files

Friday, 12 February 2021

How to do I send parameters to bash script / shell script?

You can send parameters from command-line to a bash script using $1 $2 ...

$1 represents parameter 1

$2 represents parameter 2

likewise

For Example:

startup.sh

#!/bin/bash

echo hello $1 $2
on command line :
./startup.sh dinesh dontha
output:
hello dinesh dontha

Thursday, 11 February 2021

Most used Linux commands when working on servers / when ssh to remote systems

tail -100f catalina.out

let's say we want to run the latest current running logs with last 100 lines, then you should use the above command.

cd <specific folder>

useful to change to mentioned directory

cd ..

useful to change back to parent directory, one level down

mkdir <dir_name>

creates a new directory with given name <dir_name>

mkdir -p <dir1/dir2/dir3>

creates the directory structure, which fails to do without -p option

rm -r <dir1/dir2/dir3>

removes the complete directory structure, regardless of number of internal directories, which fails to remove without -r option.



How to install correct combination of nodejs and npm on mac / windows ?

 I would recommend different installation process on mac rather than installing the nodejs using brew command line.

  • Just go to  https://nodejs.org/en/download/releases/
  • For example, if you want nodejs v6.9.4 and its compatible npm v3.10.10 on MAC, then download the node-v6.9.4.pkg file of specified version, which automatically installs its compatible npm version 3.10.10.
  • Likewise, you don't need npm to install nodejs, but you can directly install required nodejs version which brings its compatible npm version automatically.

Note: If you are working on Mac / Windows, then you find the respective installers for installation. If you are working on Linux based OS, you would need to download .tar files.

Index of /download/release/v6.9.4/


../
docs/                                              05-Jan-2017 21:30                   -
win-x64/                                           05-Jan-2017 21:04                   -
win-x86/                                           05-Jan-2017 21:23                   -
SHASUMS256.txt                                     05-Jan-2017 23:33                3865
SHASUMS256.txt.asc                                 05-Jan-2017 23:33                4369
SHASUMS256.txt.sig                                 05-Jan-2017 23:33                 287
node-v6.9.4-aix-ppc64.tar.gz                       05-Jan-2017 21:08            17397618
node-v6.9.4-darwin-x64.tar.gz                      05-Jan-2017 21:29            12041778
node-v6.9.4-darwin-x64.tar.xz                      05-Jan-2017 21:30             8316788
node-v6.9.4-headers.tar.gz                         05-Jan-2017 21:35              476961
node-v6.9.4-headers.tar.xz                         05-Jan-2017 21:35              346992
node-v6.9.4-linux-arm64.tar.gz                     05-Jan-2017 20:52            13216193
node-v6.9.4-linux-arm64.tar.xz                     05-Jan-2017 20:52             8413528
node-v6.9.4-linux-armv6l.tar.gz                    05-Jan-2017 21:27            13058040
node-v6.9.4-linux-armv6l.tar.xz                    05-Jan-2017 21:33             8291812
node-v6.9.4-linux-armv7l.tar.gz                    05-Jan-2017 21:03            13040155
node-v6.9.4-linux-armv7l.tar.xz                    05-Jan-2017 21:05             8304476
node-v6.9.4-linux-ppc64.tar.gz                     05-Jan-2017 20:49            13994448
node-v6.9.4-linux-ppc64.tar.xz                     05-Jan-2017 20:50             8522364
node-v6.9.4-linux-ppc64le.tar.gz                   05-Jan-2017 20:48            13749594
node-v6.9.4-linux-ppc64le.tar.xz                   05-Jan-2017 20:48             8655784
node-v6.9.4-linux-s390x.tar.gz                     05-Jan-2017 20:46            14201628
node-v6.9.4-linux-s390x.tar.xz                     05-Jan-2017 20:46             8933844
node-v6.9.4-linux-x64.tar.gz                       05-Jan-2017 20:50            13928019
node-v6.9.4-linux-x64.tar.xz                       05-Jan-2017 20:51             9348020
node-v6.9.4-linux-x86.tar.gz                       05-Jan-2017 20:52            13411488
node-v6.9.4-linux-x86.tar.xz                       05-Jan-2017 20:53             8935060
node-v6.9.4-sunos-x64.tar.gz                       05-Jan-2017 21:10            14679106
node-v6.9.4-sunos-x64.tar.xz                       05-Jan-2017 21:11             9442348
node-v6.9.4-sunos-x86.tar.gz                       05-Jan-2017 23:31            13617149
node-v6.9.4-sunos-x86.tar.xz                       05-Jan-2017 23:31             8765124
node-v6.9.4-win-x64.7z                             05-Jan-2017 21:05             6868142
node-v6.9.4-win-x64.zip                            05-Jan-2017 21:05            12239442
node-v6.9.4-win-x86.7z                             05-Jan-2017 21:24             5982803
node-v6.9.4-win-x86.zip                            05-Jan-2017 21:24            11026266
node-v6.9.4-x64.msi                                05-Jan-2017 21:06            12791808
node-v6.9.4-x86.msi                                05-Jan-2017 21:24            11489280
node-v6.9.4.pkg                                    05-Jan-2017 21:24            15526075
node-v6.9.4.tar.gz                                 05-Jan-2017 21:31            26379683
node-v6.9.4.tar.xz                                 05-Jan-2017 21:32            15519264

Monday, 8 February 2021

How to create a Kafka Topic using command line utility ?

 Just a quick introduction, we can create Kafka Topic. using command line utility and also using programming APIs.

This post shows command line utility which help us in creating the Kafka Topic.

For your information, under /bin folder of Kafka installation, there are many Kafka Utilities available for various uses, one of which to kafka-topics.sh helps to create topics, describe topics & other topic related stuff.

Before executing the topic command as below, ensure your Zookeeper Server and Kafka Cluster (same order - start Zookeeper 1st then Kafka Cluster service next) has already been started.

Lets say Kafka cluster is having one node with its Kafka Service node running at a default port 9092 & we are trying to create a topic named error-events

$ bin/kafka-topics.sh --create --topic error-events --bootstrap-server localhost:9092

Console Output:

Topic created.

To ensure the topic creation, we can describe the topic:

$ bin/kafka-topics.sh --describe --topic error-events --bootstrap-server localhost:9092
Topic:error-events  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: error-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0


It prints all the information related to given kafka topic - like partition count, replicator factor, replicas, in-sync replicas (Isr), Leader ...




Saturday, 6 February 2021

What is SPLUNK?

Splunk is a product/tool which captures the real-time data & indexes the data, extracts interesting fields to analyse. We can create various Splunk objects like Alerts, Reports, Dashboards using this real-time data. By doing so, we can analyse various issues happening in Applications that feeds data to the Splunk.

For Example, 

  • you can find all the 500 errors of your application happened during last 12 Hrs.
  • you can find all the intruders or users who are trying to hack the system. For example, if from a particular IP address, if we see some unusual 403 forbidden errors, let say, when the user is trying to enter the password with a lot of trials.
  • you can create an alert with the above mentioned scenarios, for example, if we find 500 errors more than 10 in last 4 hrs, we should get an alert. 
  • We can create Dashboards with all the charts & reports in one place for a better monitoring and analysis.
 
Splunk provides a search query language of its own, which does have a lot of commands for searching & reporting, often called as "Transforming commands" & "Non-transforming commands". Using these commands - we can analyse & capture the realtime statistics of the application in one place.



Friday, 5 February 2021

How to force the maven to redownload all the packages ?

First Way: 

You can use mvn option -U : which forces the maven to download the dependency into local repository (.m2 folder)

Eg:

mvn clean package -U

Second Way:

Use purge-local-repository

Eg:

mvn dependency:purge-local-repository clean package


Third Way:

You can manually directly delete the .m2 folder in computers home directory (like in linux - mac, ~)





@SerializedName annoation: GSON: How to use a different name for serialised fields

 you can use GSON annotation @SerializedName annotation for using a different names for serialized fields.

Maven Dependency Needed:

<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency> For Example:

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.annotations.SerializedName;

public class GsonExposeAnnotationTest {
public static void main(String[] args) {

Student student = new Student("Dinesh", "ROLL123", "Hyderabad - 500081");
Gson gson = new GsonBuilder().setPrettyPrinting().create();
System.out.println(gson.toJson(student));

}
}

class Student {
@SerializedName("student-name")
String name;
@SerializedName("student-rollno")
String rollNo;
@SerializedName("student-address")
String address;

public Student(String name, String rollNo, String address) {
this.name = name;
this.rollNo = rollNo;
this.address = address;
}
} Output: Serialised output - fields name are different than in POJO { "student-name": "Dinesh", "student-rollno": "ROLL123", "student-address": "Hyderabad - 500081" }

@Expose annotation : GSON : How to exclude / include the fields in serialisation & deserialisation ?

You can use @Expose annotation to exclude or include the fields using GSON library.

By default, @Expose serialize & deserialize attributes have true. So you can mention @Expose directly, which intends @Expose(serialize = true, deserialize = true)
Maven Dependency Need: Add this dependency in your pom.xml file.
For Example:

<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.annotations.Expose;

public class GsonExposeAnnotationTest {
public static void main(String[] args) {
Customer customer = new Customer("Dinesh", "CID124", new Integer(12));
Gson gson = new GsonBuilder().setPrettyPrinting().create();
System.out.println(gson.toJson(customer));

gson = new GsonBuilder().setPrettyPrinting().excludeFieldsWithoutExposeAnnotation().create();
System.out.println(gson.toJson(customer));


}
}

class Customer {
@Expose(serialize = true, deserialize = true)
public String customerName;
@Expose(serialize = true, deserialize = true)
public String customerId;
@Expose(serialize = false, deserialize = false)
public Integer totalAssets;

public Customer(String customerName, String customerId, Integer totalAssets) {
this.customerName = customerName;
this.customerId = customerId;
this.totalAssets = totalAssets;
}
}

Input:
{
  "customerName": "Dinesh",
  "customerId": "CID124",
  "totalAssets": 12
}

output: Only exposed fields are in the seen in the output after serialisation.
{ "customerName": "Dinesh", "customerId": "CID124" }


Saturday, 16 January 2021

How to use comments in python

Single line comments
You can use # symbol for single line comments

For example,

#print function is used to output something
print ('Hello World')

Block Comments  ( Triple quotes comments)
Three quotes comments are considered as block comments in python.

For example,

"""
Author: Dinesh Dontha
Component: Payment Processing
Licence: ----
"""
print('Start of the program')
#some processing
print('End of the program')

How to print in python

You can use print () function in python to output something in python. 

    For example: 

    print ('Hello World ') 

You can also pass some argument into print function. 

     For example, 

     name = "Dinesh" 
     print (' Your are %s.', % name);

Tuesday, 15 September 2020

How to create CRON expressions & what are the components of a typical CRON expression? Usage of CRON expression in @Scheduled

CRON expressions are mainly used for scheduling tasks, which should run frequently at a fixed rate, lets say, running a task every day at mid night 12'0 Clock, running a task every hour & like wise.

For Example, An Online shopping cart application might have many schedulers that schedule the tasks, for instance, once an order is placed successfully by a customer,  a shipping order scheduler might processes these successful orders placed in previous hour or before some time.

A typical CRON expression: "0/5 * * ? * *"

CRON expression basically consists of 7 components. Each component will have range of allowed values, Allowed special characters. For example, "Seconds" component will have range of values allowed from 0 to 59.

CRON expression components:

"Seconds" "Minutes" "Hours" "Day of month" "month" "Day of week" "Year"
All are mandatory parameters, except "Year".
"Seconds" - a mandatory parameter
     Range of values:         0-59
     Allowed characters     , - * /
Examples: 
    10,20      executes at 10th second and 20th second
    *             executes at every second
    0/5          executes every 5 seconds starting from 0th second
"Minutes" - a mandatory parameter
     Range of values:         0-59
     Allowed characters     , - * /
Examples: 
    10,20      executes at 10th minute and 20th minute 
    *             executes at every minute
    0/5          executes every 5 minutes starting from 0th minute
"Hours" - a mandatory parameter
     Range of values:         0-23
     Allowed characters     , - * /
Examples: 
    10,20      executes at 10th hour and 20th hour 
    *             executes at every hour
    0/5          executes every 5 hours starting from mid night 12:00 AM
"Day of month" - a mandatory parameter
     Range of values:         1-31
     Allowed characters    , - * ? / L W C
Examples: 
    10,20      executes at 10th day and 20th day 
    *             executes at every day
    1/5          executes every 5 days starting from first day
    L represents the last day of the month. L is allowed only with "Day of month" 
    & "Day of week".
    L represents 31st in Jan, 28th in Feb (non-leap years)
    W represents the nearest week-day. Only allowed to use when it is a single day, 
    not allowed with range of values.
    12W 
    say if 12th day of the month is Saturday, it will trigger on 11th day (Friday), 
    if 12th day is Sunday, it will trigger on 13th day (Monday), 
    if 12th day is a Monday, as it is weekday, it will trigger on 12th day (Monday)
    C represents the calendar. Only allowed for "Day of month" & 
    "Day of week" components.
    Values are calculated against an associated calendar if any. 
    If no calendar associated, if it is declared like
    5C represents the first day included by the calendar on or after the 5th.
"Month" - a mandatory parameter
     Range of values:         0-11 or JAN-DEC
     Allowed characters    , - * /
Examples: 
    1,2                      executes at FEB & MAR months 
   FEB,MAR          executes at FEB & MAR months 
    *                         executes at every month
"Day of Week" - a mandatory parameter
     Range of values:         0-11 or JAN-DEC
     Allowed characters    , - * ? / L C #
  L represents the last day of the month. L is allowed only with "Day of month" 
    & "Day of week"
    1-7 (1 - sun, 2- mon, 3-tues, 4-wed, 5- thur, 6-fri, 7-sat)
    6L represents last Friday of the month.
    C represents the calendar. Only allowed for "Day of month" 
    & "Day of week" components.
    Values are calculated against an associated calendar if any. 
    If no calendar associated, if it is declared like
    1C represents the first day included by the calendar on or after Sunday.
    7C represents the first day included by the calendar on or after Saturday.
   # is used to represent a single day, like nth XXX day. 
    Only allowed for "Day of week".
   7#2 represents the 2nd Saturday of the month.
 "Year" - an optional parameter
     Range of values:         empty or 1970-2099
     Allowed characters    , - * /
Examples: 
    2000-2010         executes for range of years from 2000 to 2010
   2000,2006          executes for years 2000 and 2006
    *                         executes at every year
Examples with all components combined:
0 0 12 * * ?Fire at 12:00 PM (noon) every day
0 20 10 ? * *Fire at 10:20 AM every day
0 15 11 * * ?Fire at 11:15 AM every day
0 15 10 * * ? *Fire at 10:15 AM every day
0 20 20 * * ? 2010Fire at 08:20 PM every day during the year 2010
0 * 14 * * ?Fire every minute starting at 2:00 PM and ending at 2:59 PM, every day
0 0/5 14 * * ?Fire every 5 minutes starting at 2:00 PM and ending at 2:55 PM, every day
0 0/5 14,18 * * ?Fire every 5 minutes starting at 2:00 PM and ending at 2:55 PM,
AND fire every 5 minutes starting at 6:00 PM and ending at 6:55 PM,
every day
0 0-5 16 * * ?Fire every minute starting at 4:00 PM and ending at 4:05 PM, every day
0 10,22 14 ? 3 FRIFire at 2:10 PM and at 2:22 PM every Friday in the month of March
0 15 10 ? * MON-FRIFire at 10:15 AM every Monday, Tuesday, Wednesday, Thursday and Friday
0 15 10 15 * ?Fire at 10:15 AM on the 15th day of every month
0 15 10 L * ?Fire at 10:15 AM on the last day of every month
0 15 10 ? * 7LFire at 10:15 AM on the last Saturday of every month
0 15 10 ? * 7LFire at 10:15 AM on the last Saturday of every month
0 15 10 ? * 7L 2002-2004Fire at 10:15 AM on every last Saturday of every month
during the years 2002, 2003 and 2004
0 15 10 ? * 6#3Fire at 10:15 AM on the third Friday of every month
0 0 12 1/5 * ?Fire at 12 PM (noon) every 5 days every month,
starting on the first day of the month
0 6 6 6 6 ?Fire every June 6 at 6:06 AM
How to use CRON expression in spring scheduling @Scheduled annotation?
As we know that, firstly, we need to enable scheduling using @EnableScheduling 
and use @Scheduled annotation on tasks.
Full example: Spring boot scheduling example
Example:
    @Scheduled(cron = "0/5 * * ? * *")
    public void reportCurrentTime() throws InterruptedException {
        //Thread.sleep(6000l);
        logger.info(formatter.format(new Date()));
    }