Category Archives: Cloud

Blogs about the information of Cloud technologies

Map EBS Storage with Linux EC2 instance in AWS

Map EBS Storage with Linux EC2 instance in AWS

In this blog, we are going to configure the EBS Storage to the redhat linux EC2 instance in Amazon web services.

Following are the steps

1. Create an EC2 instance of Redhat linux from AWS Console.

2. Create addition volume of EBS storage from AWS Console.

3. Attach the EC2 instance with EBS volume from EBS tab by right click on EBS Storage which is created in step 2.
Note: Both EC2 instance and EBS volume should be in same availability zone.

4. Open the putty session with private key.

5. Now first step is to check the disk present as shown below before attached the new EBS volume in step 3

-- command executed before step 3 it show only one volume
[ec2-user@ip-172-31-7-21 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /

6. After step 3 executed, command show the volume is attached

[ec2-user@ip-172-31-7-21 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /
xvdf 202:80 0 1G 0 disk

7. Switch to root user

sudo su -

8. Check the file system type which need to be formatted and already existed/used by Linux system

[root@ip-172-31-7-21 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /
xvdf 202:80 0 1G 0 disk
[root@ip-172-31-7-21 ~]#
[root@ip-172-31-7-21 ~]# file -s /dev/xvda
/dev/xvda: x86 boot sector; partition 1: ID=0xee, active, starthead 0, startsector 1, 20971519 sectors, code offset 0x63
[root@ip-172-31-7-21 ~]#
[root@ip-172-31-7-21 ~]# file -s /dev/xvda1
/dev/xvda1: data
[root@ip-172-31-7-21 ~]# file -s /dev/xvda2
/dev/xvda2: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
[root@ip-172-31-7-21 ~]#

Note: It show that linux system is using XFS filesystem.

8. Make a file system XFS as shown below example:

[root@ip-172-31-7-21 data1]# mkfs.xfs /dev/xvdf
meta-data=/dev/xvdf isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

Note: You can make other filesystem also like ext4 of one partition

[root@ip-172-31-7-21 ~]# mkfs -t ext4 /dev/xvdg
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

9. Mount the device to a new folder

mkdir /data1
mount /dev/xvdf /data1

10. Now check the lsblk command for output

[root@ip-172-31-7-21 data1]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /
xvdf 202:80 0 1G 0 disk /data1

11. You can check with df -Th command for file format:

[root@ip-172-31-7-21 data2]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/xvda2 xfs 10G 925M 9.1G 10% /
devtmpfs devtmpfs 474M 0 474M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 13M 483M 3% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs tmpfs 100M 0 100M 0% /run/user/1000
tmpfs tmpfs 100M 0 100M 0% /run/user/0
/dev/xvdf xfs 976M 2.6M 907M 1% /data1

12. For make change permanent add entry in /etc/fstab file.
vi /etc/fstab

# /etc/fstab
# Created by anaconda on Fri Mar 23 17:41:14 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=50a9826b-3a50-44d0-ad12-28f2056e9927 / xfs defaults 0 0
/dev/xvdf /data1 xfs defaults 0 0

13. Check the entry is good

mount -a
Example:
[root@ip-172-31-7-21 data2]# mount -a
[root@ip-172-31-7-21 data2]#

Note: If you type wrong name like /dev/svdf then you got following message
[root@ip-172-31-7-21 data2]# mount -a
mount: special device /dev/svdg does not exist

14. Now your drive is ready to use.

For Unmount the drive

[root@ip-172-31-7-21 data2]# umount /data1
[root@ip-172-31-7-21 data2]#

Advertisements

Map EBS Storage with Window EC2 instance in AWS

Map EBS Storage with Window EC2 instance in AWS

We are showing the example of mapping the EBS Storage device in Windows EC2 instance of Amazon web services. We created seperate EBS Storage and tried to map this storage with one of my instance EC2 which is running windows Server 2016

Note: For Mapping EBS storage and EC2 instance both in same zone.

following are the steps configured disk in Window EC2 instance

1. Create an Window EC2 instance and login in into it.

2. Go to EBS Storage on right side and create a volume.
Important: Before create check EC2 instance zone both EBS volume and EC2 instance should be in same zone.

Elastic Block Storage 1.PNG

3. Start Creating the EBS Volume from CREATE VOLUME button on AWS console.

Create EBS 2

4. Select the Size as you need and Zone (same as EC2 instance). Press CREATE Volume button at last.
In example i am choosing min as i need to show example. Always pick min as you need because it is model pay as you grow.

Create Volume 3

5. Attach the created EBS to EC2 instance.

Attached Volume 4

6. EBS volume is only attached if both are in same Zone.

Attach instance 6

7. Go to EC2 instance, Open the Computer Management –> Select Disk Management.

Computer Management 7.PNG

8. In Disk Management is show offline then online by right click on it.

Online the disk 8

10. Right Click then initialise disk for use.

Initialize Disk 9.PNG

11. Complete the initialize disk process.

MBT 10

12. Create new volume and select the drive name as D: drive and finish the process by next button.

New Sample Volume 11

13. Your device is formatted and ready for use.

Disable or Delete the Drive EBS

1. First detached the volume on AWS Console at EBS console –> volume –> go to action and select deattach the volume.

Deattach volume 13

2. Delete the volume.

Delete volume.PNG

Configure VNC for Amazon Cloud EC2 Redhat version from Windows

Configure VNC for Amazon Cloud EC2 Redhat version from Windows

1. Start the Redhat EC2 Machine from Amazon Web Service Cloud.

2. Connect with Redhat EC2 Machine with putty.

Connect with Linux EC2 instance

3. Switch ec2-user to root user for install the following packages.

sudo su -

4. Install the following packages in same sequence with root user.

yum groupinstall 'Server with GUI'
yum install -y pixman pixman-devel libXfont
yum -y install tigervnc-server

5. You can set password for ec2-user

passwd ec2-user

6. Open the /etc/ssh/sshd_config file and set following parameter

cd /etc/ssh
vi sshd-config
--set the following parameter to YES option and uncomment with NO option
password authentication parameter to yes
ChallengeResponseAuthentication to yes

Change SSHD config file

7. Restart the sshd service using the below command service sshd restart

[root@ip-172-31-16-82 ssh]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service

8. Exit the root user and connect with ec2-user

[root@ip-172-31-16-82 ssh]# exit
logout
[ec2-user@ip-172-31-16-82 ~]$

9. Setup a VNC password with command vncpasswd

[ec2-user@ip-172-31-16-82 ~]$ vncpasswd
Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used
[ec2-user@ip-172-31-16-82 ~]$

10. Start a VNC Server using the below command vncserver :1

[ec2-user@ip-172-31-16-82 ~]$ vncserver :1
xauth: file /home/ec2-user/.Xauthority does not exist

New 'ip-172-31-16-82.us-east-2.compute.internal:1 (ec2-user)' desktop is ip-172-31-16-82.us-east-2.compute.internal:1
Creating default startup script /home/ec2-user/.vnc/xstartup
Creating default config /home/ec2-user/.vnc/config
Starting applications specified in /home/ec2-user/.vnc/xstartup
Log file is /home/ec2-user/.vnc/ip-172-31-16-82.us-east-2.compute.internal:1.log
[ec2-user@ip-172-31-16-82 ~]$

VNC Putty connection.PNG

11. In AWS Cloud, we need to go to Security group of EC2 Redhat Linux instance and open the VNC port 5901 for connectivity, for multiple connectivity 5900-5910 port need to open.

Go to EC2 Dashboard --> Select the Redhat machine --> Below on description tab, Select the security group to modified --> In Security group go to inbound --> add new rule ---> TCP --> PORT RANGE : 5900-5910 , SORUCE: anywhere --> Save it

12. Install your vnc viewer of ULTRA VNC Viewer or as you want.

For connectivity use public address or Public IP with PORT 5901
Enter : at vnc viewer window
Enter password set for VNC Server in step 9

VNCViewer.PNG

Managed Users in Identity & Access Management(IAM) of AWS cloud

Create & managed permission for USER in Identity & Access Management of AWS cloud

IAM service manages the access management to AWS resources for your users.Its manages as:
Authentication: who can use your AWS resources
Authorization: What resources can use and in what ways

Four Components:
USERS Create & Manage User, permission to allow or deny access of AWS resources
GROUPS Define group of permission assigned to user like Development team has same permission assign to a user then you create a group of common permission of development team which will easy to assign to the USER added in development team. All team having same permission.
ROLES Roles are similar to User. Roles are assigned to application but USER are assigned to Peoples. Example EC2 application want to access S3 Services with help of Roles.
POLICES It is basically permissions.

Basic Login with IAM

A. Login with the main account to Manage the User and Permission then select services from upper right side.

Services

B. Go to the Security Identity & Compliance module and select IAM from it.

SecuritySetting

C. Now highlighted URL is used to login for other user which you are going to create in next step.(IAM USER-SIGNIN LINK)
This page also show you resource how much user, roles, Policies groups present in it.

IAM3.PNG

Groups
In left panel, you can manage groups like create, modify the groups. You create new group then attach policy with it like administrator full access permission to group.
Note: You have option in Group Action as Add users to group, delete group, Edit group Name & Remove Users from Group

Steps to Create a new group
A. Create a new group
Groups1

B. Add new group name and attach policy

Groupattachpolicy3
Groupattachpolicy3

C. Finish the Creating of Group.

Grouppolicy4.PNG

USERS

A. Create user
Note: Access type as Programmatic access or AWS Management Console access
Programmatic: access used in development, application interacting with API.
AWS Management console: used to deploy resources, Remove or give access

CreateUser1.PNG

B. When using AWS Management Console login it will ask you for assigning password.

createuser2.PNG

C. Assign Permission to new create user.

addusertogroup3.PNG

D. Review user before create.

Reviewuserbeforecreate.PNG

After creating User you can login with New URL Provided by AWS

A. Copy the URL on IAMS first page and open into new browser.

URLforlogin

B. It will go for new signin page as below.

signinAWSuser

C. Now You can login with you new created user in AWS with IAMS with this URL.
In that way you can have only one user as super user and you can create other user with different permission as you want to manage your security.

Roles
Role is used by application and user are assigned to peoples. If i want that my AWS service want to use the S3 service then i need to create a role with S3 permission and assigned to that EC2 which running AWS Service.

A. Create a Role.
e.g Create a role with S3 service then attached it with EC2 that AWS service is hosted. Go t0 IAM –> Dashboard –> Role –> Amazon EC2 –> What type of role (amazon s3 full access) –> Name (s3fullaccess_role)

B. Assigned Role to the EC2 instance.
EC2 console –> Start the instance —> attach the policy from ACTION tab –> INSTANCE SETTINGS –> ATTACH/REPLACE IAM ROLE –> Select the IAM role –> Apply.

Note: With this example, EC2 instance is able to interact with S3

Policy:
Policy is basically permission you can give to users, groups or roles. You can create your own policies.

A. Create a Policy.
Create Policy –> Policy generator –> AWS Service : EC2 , Actions: All actions , Resource : * (all resources) —> Create policy

Glacier Storage in AWS

Glacier Storage in AWS

Glacier is storage class in S3, It is very low cost storage service that provides secure, durable and flexible storage for data backup and archival.
Glacier uses vault. Amazon glacier data stored as archives. An archive is any object such as a photo, video, or document which you store in vault.
A single archive can be large as 40 terabytes.

Indirectly use the Glacier by creating Lifecycle rule in S3 storage property

--Creating Lifecycle Rule in S3 Properties.

s3--> Create bucket --> upload some backup --> start upload --> Now backup in S3 storage -->
Bucket property --> lifecycle ---> Versioning need to be turned on ----> create role that object in S3 stored archive to glacier storage class in 30 days

Directly by awscli command (only cli command used in Glacier Service

--upload
aws glacier upload-archive --account-id - --vault-name --body

aws glacier upload-archive --account-id - --vault-name edureka --body backupforglacer

Understand Simple Storage Service S3 in AWS

Simple Storage Service in AWS cloud

–Amazon Simple Storage Service is a storage designed to make web-scale computing easier for developers.
–S3 object can be accessed with HTTP protocol from anywhere as long as you have permission.
–S3 is web based storage so it has own its security for web based storage.
–it worked on Object Storage Mechanism.
–Each Amazon S3 object has data , a key & metadata.
–Each object can contain upto 5 TB of data.
–Uniquely identified with in a bucket by a key(name) and a version ID.

Bucket
–Used to store objects which consist of data and metadata
–The bucket can be configured and created in any specific region.
–When an object is added to the bucket, Amazon S3 generate a unique version ID & assign to the object.
–By Default, only 100 buckets can be created in each AWS Accounts.

Example:You have a bucket BUCKET1 and object in it photos as folder and Singapore.jpg is an image object then you can access it with help of URL from outside the world direclty as : (HTTP://BUCKET1.S3.AMAZONAWS.COM/PHOTOS/SINGAPORE.JPG)

It is used like application has images, videos and any file then application pick things from S3 storage.

Access Control Line Permission in S3
Bucket permission specifies who is allowed the access to the objects in a bucket and what particular permission have been granted.
Everyone : for every user grant anonymous access
Log Delivery : Grant access to bucket when bucket is used to store server logs
Me : Refer to AWS root account , Not to an IAM user
Authenticated Users : For anyone with AWS account can perform the appropriate action

Note: Permission at bucket level and User level

Bucket Policy:
It allows users to authorize policies which either grant or deny access to any number of accounts and across a range or set of keys.
JSON Script is used to define a policy.

Type of S3 Storage Classes
Standard: Durability is great & 99.99% availability
Standard-Infrequent Access : it has lower availability 99.90%
Glacier: It take long time to retrieval time for data.
Reduced Redundancy Storage(RRS): It store non-critical, reproducible data at lower levels of redundancy than S3 Standard Storage

Cross-Region Replication
It automatically copying data from one region to other region. It is a bucket level features that copy asynchronous copying of object across bucket from one region to other.
It provide region failure like flood or earthquake failed region, then it help to overcome with region failure and make your data more protective.

Versioning
Version is keeping data multiple variants of objects in the same bucket. It help to recover objects which is accidental deleted or overwrite.
Every version of every object is stored in Amazon S3 bucket as RESERVED , RETRIEVED & RESTORED.
By Default Versioning Feature is disabled.
Example
If Version feature is enabled then it come to cost and it help to protect the data which accidentally deleted by user.
It maintained the Version of object when object is updated or deleted then we restore that object by replacing older version.

Understand Elastic Block Storage in AWS

Elastic Block Store

Primary Storage for data that require frequent updates or storage for a database application. Provide Block Storage Volumes for EC2 instance to store stuff E.g Database, web server data etc.
Major Used for following services:
Data change frequently
Require Long-term persistence
Frequently read or write operations

EBS Snapshot
–Can back up the data on the EBS volumes to Amazon S3 by taking point in time snapshot.
–Snapshots are incremental backup
–Snapshots of encrypted volumes are automatically encrypted.
–Volumes that are created from encrypted snapshots are also automatically encrypted.
–Cost of Snapshot means snapshot use the S3 volume for backup then S3 volume usage cost will be applied.
–EBS Volume can be attached to Same Region in which instance is present. Means instance & EBS volume in same region for attach/use.

Note:
EBS Volume has multiple copies like replication only on one zone availability if complete zone failure then it will lost. It does not replica on other zone. It high availability in same one zone.
Volume cannot moved from one zone to other in EBS. If EBS volume is failed then it is available in same zone but not in other zone if complete zone failed.

EBS Volume
EBS volumes that are attached to EC2 instance are exposed as storage volumes that persist independently from the life of the instance. If EC2 instance terminate then DBS volume is available.
It is independent from EC2 instance. You can use encryption with EBS volume.

EBS type:
Provisioned IOPS(SSD) Used for Production environment like large database Oracle, MangoDB etc or Application used large user base.
General Purpose(SSD) used for development purpose but you can start with General Purpose in future you need you move it to Provisioned IOPS.
Magnetic low cost devices used for cold storage data, it is used to storing the data which is less accessed like log files, archiving purpose.