Apache Sqoop Overview & Import Data From MySQL to HDFS

Overview on Sqoop

Sqoop is open source s/w from Apache used for transfer data between RDBMS(Oracle, SQL Server, MySQL) and HDFS.

MySQL Database
Connecting to MySQL Database in cloudera VM:

root user: root/cloudera
other user: cloudera/cloudera

[cloudera@quickstart ~]$ mysql -u root -p

Streaming data into Hadoop using Apache Flume

Flume:Flume a hadoop echo system s/w used for streaming the logs file from applications int o HDFS.

In this post let's discuuss about following topics.

• Overview on Flume
• Streaming log files data into HDFS
• Streaming Twitter App logs into HDFS

Overview on Apache Pig

What is pig?

• Implemented by Yahoo.
• Pig Hadoop echo system s/w from apache foundation used for analysing the data.
• Pig uses pig latin language.
• Data flow language.
• handle structured, semi-structured and un-structured
• Replacement of mapreduce(not 100%)
• Pig internally uses MapReduce.

Apache Hive UDF'S (User Defined Functions)

In this post, let's discuss about Hive UDF's.

• Creating UDF
• How to packaging UDF(creating jar file)
• Add jar file in to hive
• Test UDF

Steps to create and test UDF's

1) Implement the code for UDF in Java
2) Package java class into jar file copy in some location
3) Add jar file in to Hive CLI
4) Create temporary function in hive
5) Use hive UDF BY  using Query.

Prerequiste: Table should have some data.

Problem statement-1
Find the maximum marks obtained out of four subject by an student.

Package java class into jar file copy in some location.

SELECT CLASS IN ECLIPSE-->RIGHT-->EXPORT-->JAVA-->JAR--> BROWSE THE LOCATION-->PROFILE FILENAME WITH .JAR Extension.

Add jar file in to Hive CLI

hive> add jar /home/cloudera/training/HiveUDFS/getMaxMarks.jar;

Create temporary function in hive

hive> create temporary function getmaxmarks as 'udfs.GetMaxMarks';

Use hive UDF BY  using Query

hive> select getmaxmarks(10,20,30,40) from dummy;   // sanity test

There are 2 types of UDF'S

1) Regular UDF( UDF) ---> Applied on more number of rows in a table
2) User Defined aggregate function (UDAF) --> Group of result sets.

Problem statement-2: Find the mean of marks obtained in maths by all the students.

Package java class into jar file copy in some location

Right click onth package-->export-->java-->provide jar file name.

Add jar file in to Hive CLI

hive> add jar /home/cloudera/training/HiveUDFS/getMeanMarks.jar;

Create temporary function in hive

hive> create temporary function getmeanmarks as 'udaf.GetMeanMarks';

Use functions with queries

hive> select getmeanmarks(social)from t_student_record;

JavaScriptExecutor in Selenium WebDriver

JavaScriptExecutor in Selenium WebDriver:

In general, we do click on an element using click() method in Selenium.

Sometimes web controls don’t react well against selenium commands and we may face issues with the above statement (click()). To overcome such kind of situation, we use JavaScriptExecutor interface.

Scrolling Web Pages using Selenium WebDriver

There are 3 different ways to Scroll Web Pages.

1.Scroll page by proving pixel number.
2. Scroll page till we find a web element on the page
3. Scroll page till bottom of the page

To scroll web page using Selenium, we can use JavaScriptExecutor interface that helps to execute JavaScript methods through Selenium Webdriver.

Syntax:
window.scrollBy(xnum, ynum)

Parameters:

• xnum is a Number
• Required. How many pixels to scroll by, along the x-axis (horizontal). Positive values will scroll to the right, while negative values will scroll to the left
• ynum is a Number
• Required. How many pixels to scroll by, along the y-axis (vertical). Positive values will scroll down, while negative values scroll up

Return Value: No return value

Examples:

Scroll page by proving pixel number

  js.executeScript("window.scrollBy(0,500)");

Scroll page till we find a web element on the page

  js.executeScript("arguments[0].scrollIntoView();",Element );

 Scroll page till bottom of the page

  js.executeScript("window.scrollTo(0, document.body.scrollHeight)");

Test Script

import org.openqa.selenium.By;

import org.openqa.selenium.JavascriptExecutor;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

import org.openqa.selenium.chrome.ChromeDriver;

public class Scrolling {

public static void main(String[] args) {

System.setProperty("webdriver.chrome.driver", "C://Drivers/chromedriver_win32/chromedriver.exe");

WebDriver driver = new ChromeDriver();

driver.get("https://www.countries-ofthe-world.com/flags-of-the-world.html");

driver.manage().window().maximize(); // maximum browser window

JavascriptExecutor js = (JavascriptExecutor) driver;

// 1. scrolling by using pixel

js.executeScript("window.scrollBy(0,1000)", "");

// 2. scrolling page till we find element

WebElement flag = driver

.findElement(By.xpath("//*[@id='content']/div[2]/div[2]/table[1]/tbody/tr[86]/td[1]/img"));

js.executeScript("arguments[0].scrollIntoView();", flag);

// 3. scroll page till bottom

js.executeScript("window.scrollTo(0,document.body.scrollHeight)");

}

}

How To Run WebDriver Test Case using Microsoft Edge Browser

Download Microsoft WebDriver from here to launch Edge Browser. Download the proper version of the driver based on your OS build number. If the extension of the downloaded file is “.msi“, you need to install it to get the “.exe” driver.

How to Write Excel Files Using Apache POI In Selenium WebDriver

Let’s see how to Read excel files using Apache POI in Selenium WebDriver:

How To Resize Browser Window Using Selenium WebDriver

To resize browser window to particular dimensions, we use ‘Dimension’ class to resize the browser window.

How to Read Excel Files Using Apache POI In Selenium WebDriver

Let’s see how to Read excel files using Apache POI in Selenium WebDriver:

Handling Excel Files Using Apache POI In Selenium WebDriver

Selenium supports only Web browser automation. We need to get the help of third party API like Apache POI to handle (read and write) excel files using Selenium WebDriver.

How To Find Broken Links Using Selenium WebDriver

One of the key test case is to find broken links on a webpage. Due to existence of broken links, your website reputation gets damaged and there will be a negative impact on your business. It’s mandatory to find and fix all the broken links before release. If a link is not working, we face a message as 404 Page Not Found.

Double Click Action In Selenium WebDriver

In some scenarios, we may need to do double click action on a particular element to move further. In such cases, we use Actions class in Selenium WebDriver to work on Mouse and Keyboard Actions. Check out the below code for detailed explanation of Actions Class.

Right Click Action (Context Click) In Selenium

In some scenarios, we may need to do right click action / context click on an element to do some actions. We use Actions class in Selenium WebDriver to work on Mouse and Keyboard Actions. 

Drag And Drop In Selenium WebDriver

In some applications, we may face a situation to automate drag and drop an item from one location to another location.  Selenium has provided an “Actions” class to handle this kind of scenarios. 

Mouse Hover Actions Using Actions Class In Selenium

Sometimes, sub menu items render in DOM only when we mouse hover on main menu. In that case, we face difficulty to click on sub menu item. In order to perform mouse hover actions, we need to chain all of the actions that we want to achieve in one go. To do this we need to make the driver move to the parent element that has child elements and click on the child element.

How To Handle Javascript Alerts/PopUps In Selenium WebDriver

In this post, we see how to handle javascript alerts/popus. Alerts are basically popup boxes that take your focus away from the current browser and forces you to read the alert message. You need to do some action such as accept or dismiss the alert box to resume your task on the browser.

How To Upload Files Using AutoIT in Selenium

Selenium can not handle file downloading because browsers use native dialogs for downloading files. Sometime we need to download file from AUT(Application Under Test). There are several ways to automate download file in Selenium but here we see download file using AutoIT in Selenium WebDriver.

AutoIt Introduction:

AutoIt Tool is an open source tool. It is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows out-of-the-box with no annoying “runtimes” required!

Now the question is how we do download file using AutoIT Tool in Selenium WebDriver.

Steps to integrate autoit with selenium webdriver

1) Write AutoIT script for file uploading( AutoIT Editor)

ControlFocus() --> focus on the text box
ControlSetText() --> providing path of a file
ControlClick() --> clicking on open button

2) Compile AutoIT script and generate .exe file

Tools-->Compile-->Select x64--> Compile --> generated .exe file

3) use and integrate .exe file in selenium webdriver script

Ex: 

Runtime.getRuntime().exec("C://autoitfiles/fileupload.exe"+" "+"C:\\SeleniumPractice\\Fruites\\apple.jpg");

Uploading Single File

AutoIT script

ControlFocus("File Upload","","Edit1")
Sleep(3000)
ControlSetText("File Upload","","Edit1","C:\SeleniumPractice\Fruites\apple.jpg")
Sleep(3000)
ControlClick("File Upload","","Button1")
Sleep(3000)

Note: We need to compile AutoIT script and generate .exe file (SingleFileUpload.exe)

Selenium Test case

import java.io.IOException;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class UploadSingleFile {

public static void main(String[] args) throws IOException, InterruptedException {

System.setProperty("webdriver.gecko.driver","C://Drivers/geckodriver-v0.19.1-win64/geckodriver.exe");
WebDriver driver=new FirefoxDriver();

driver.get("http://demo.automationtesting.in/Register.html");

WebElement button=driver.findElement(By.xpath("//*[@id='imagesrc']"));

JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", button);

Runtime.getRuntime().exec("C://SeleniumPractice/SingleFileUpload.exe"); // execute .exe file

//driver.quit();
}

}

Uploading Multiple Files

AutoIT script

Sleep(500)

ControlFocus("File Upload","","Edit1")

Sleep(500)

ControlSetText("File Upload","","Edit1",$CmdLine[1])

Sleep(500)

ControlClick("File Upload","","Button1")

Sleep(500)

Selenium Test case

import java.io.IOException;

import org.openqa.selenium.By;

import org.openqa.selenium.JavascriptExecutor;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

import org.openqa.selenium.firefox.FirefoxDriver;

public class UploadMultipleFiles {

public static void main(String[] args) throws IOException, InterruptedException {

System.setProperty("webdriver.gecko.driver","C://Drivers/geckodriver-v0.19.1-win64/geckodriver.exe");

WebDriver driver=new FirefoxDriver();

driver.get("http://demo.automationtesting.in/Register.html");

WebElement button=driver.findElement(By.xpath("//*[@id='imagesrc']"));

JavascriptExecutor executor = (JavascriptExecutor)driver;

Thread.sleep(3000);

//round1- first file

executor.executeScript("arguments[0].click();", button);

Runtime.getRuntime().exec("C://SeleniumPractice/MultipleFilesUpload.exe"+" "+ "C:\\SeleniumPractice\\Fruites\\apple.jpg");

Thread.sleep(5000);

//round2-  second file

executor.executeScript("arguments[0].click();", button);

Runtime.getRuntime().exec("C://SeleniumPractice/MultipleFilesUpload.exe"+" "+ "C:\\SeleniumPractice\\Fruites\\Mangoes.jpg");

Thread.sleep(5000);

//round3-  third file file

executor.executeScript("arguments[0].click();", button);

Runtime.getRuntime().exec("C://SeleniumPractice/MultipleFilesUpload.exe"+" "+ "C:\\SeleniumPractice\\Fruites\\PineApple.jpg");

Thread.sleep(5000);

driver.quit();

}

}

TestNG Listeners & Extent Reports in Selenium

In this post, we see TestNG listeners. Listeners “listen” to the event defined in the selenium script and behave accordingly. The main purpose of using listeners is to create logs. There are many types of listeners such as WebDriver Listeners and TestNG Listeners.

Here in this post, we see TestNG Listeners. Using TestNG listeners we could generate logs and customize TestNG Reports.

How to use HashMap in Selenium WebDriver

In this article, I discussed how to create hash map in Java for data driven tests and read the data from the HasMap object. Please go through below example.

1) HashMap in Java

Example:

public class HashMapExample {

public static void main(String[] args) {

HashMap hm = new HashMap();

// Adding pairs to HashMap
hm.put(101, "John");
hm.put(102, "Scott");
hm.put(103, "David");

// Printing pairs from HashMap
System.out.println(hm);

// Remove a pair from HashMap by using key
hm.remove(102);

System.out.println(hm);

// Retrive a single pair from HashMap using key value
System.out.println(hm.get(103));

// Reading Keys and values from HashMap
for (Map.Entry m : hm.entrySet()) {

System.out.println((m.getKey() + "  " + m.getValue()));
}

}

}

Output:

{101=John, 102=Scott, 103=David}
{101=John, 103=David}
David
101  John
103  David

2) How to use HashMap in Selenium WebDriver Test case

Example:

import java.util.HashMap;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class LoginTestHM {

// method return HashMap object with data pairs
static HashMap logindata() {
HashMap hm = new HashMap();
hm.put("x", "mercury@mercury");
hm.put("y", "mercury1@mercury1");
hm.put("z", "mercury2@mercury2");

return hm;
}

public static void main(String[] args) throws Exception {

System.setProperty("webdriver.chrome.driver", "C://Drivers/chromedriver_win32/chromedriver.exe");
WebDriver driver = new ChromeDriver();

driver.get("http://newtours.demoaut.com/");

// Login as X
String credentials = logindata().get("x"); // Retriving value 'x' from
// HashMap

// Login as y
// String credentials = logindata().get("y");

// Login as z
// String credentials = logindata().get("x");

String uarr[] = credentials.split("@"); // separting value of 'x' int o
// 2 parts using delimeter '@'

driver.findElement(By.name("userName")).sendKeys(uarr[0]); // Passing
// value 1
// i.e
// username
// from
// array
driver.findElement(By.name("password")).sendKeys(uarr[1]); // Passing
// value 2
// i.e
// password
// from
// array
driver.findElement(By.name("login")).click();

// Validation
if (driver.getTitle().equals("Find a Flight: Mercury Tours:")) {
System.out.println("Test Passed");

} else {
System.out.println("Test failed");

}

driver.findElement(By.linkText("Home")).click();
}

}

What Is Big Data?

Big Data refers to large volume of data, that cannot be processed using traditional databases. When we have reasonable amounts of data we typically use traditional relational databases like Oracle, MySQL, SQL Server to store and work with the data. However when we have large volume of data then traditional databases will not be able to handle the data.

API Testing Interview Questions

If you're looking for API Testing Interview Questions, you are at right place. There are lot of opportunities from many reputed companies in the world. According to research API Testing has a market share of about 16.7%. So, You still have opportunity to move ahead in your career in API Testing. Mindmajix offers advanced API Testing Interview Questions that helps you in cracking your interview & acquire dream career.

ETL Design Process & Best Practices

Introduction

ETL stands for Extract Transform and Load. Typical an ETL tool is used to extract huge volumes of data from various sources and transform the data dependi­ng on business needs and load into a different destination. In the modern business world the data has been stored in multiple locations and in many incompatible formats. The business data might be stored in different formats such as Excel, plain text, comma separated, XML and in individual databases of various business systems used etc. Handling all this business information efficiently is a great challenge and the ETL tool plays an important role in solving this problem.

Career in Software Testing for Non-IT Graduates

I am from non – IT background. Can I make my career in Software Testing industry? What is growth path in this field? Can I expect stable career in this sector? Being a non IT graduate I am not pretty good in programming. I don’t have sound knowledge of technologies as well. Will it be a good decision to switch into IT field (especially software testing)?

BDD and ATDD – Exploring the Differences

It is a good idea to explore Behavior Driven Development and Acceptance Test Driven Development methods. This post will help you understand how are these two development strategies different from TDD and each other.

Top 12 Tools for Web Services Testing

Thanks to the rising emergence of applications for social networking, online shopping, banking and many more which have triggered the requirement of web services.

Who Earns more, a Software Tester or a Developer?

Everyone remembers the day they joined their first company. Fresh out from college, all freshers are filled with energy and are pumped up to make their presence felt in the Software world. Eventually, the training starts and everyone gets acquainted with the IT terminologies, different roles in a Software firm, and how different the actual world is outside the textual confinement.

Mind-set Difference between Tester & Developer

Some IT professionals think that a tester’s role is redundant and that a developer alone can test the code that was written by him. Such notion is completely wrong. But can you guess why? A developer can definitely test his own code and also the code was written by other developers. But one thing that a developer would lack is the tester’s strategy, approach, and his mindset.

Selenium Common Exceptions

When you work with Selenium you willencounter many exceptions. Solving of those exceptions would be sometimes very tricky but If you read the exception the answer to solving is in the meaning of Exception name. We will be going through some most commonly used Exception in Selenium.

• NoSuchAttributeException find_element_by_* can’t find the element.

• NoSuchElementException find_element_by_* can’t find the element.

• InvalidSelectorException Thrown when the selector which is used to find an element does not return a WebElement. Currently, this only happens when the selector is an XPath expression is used which is either syntactically invalid (i.e. it is not an XPath expression) or the expression does not select WebElements.

• ElementNotVisibleException Thrown to indicate that although an element is present on the DOM, it is not visible, and so is not able to be interacted with.

• RemoteDriverServerException

• NoSuchWindowException

• ElementNotSelectableException

• NoSuchFrameException

• WebDriverException

• UnexpectedTagNameException Thrown when a support class did not get an expected web element

• UnableToSetCookieException Thrown when a driver fails to set a cookie.

• MoveTargetOutOfBoundsException Indicates that the target provided to the actions move() method is invalid.

• InvalidSwitchToTargetException The frame or window target to be switched doesn’t exist.

• InvalidElementStateException

• InvalidCookieDomainException Thrown when attempting to add a cookie under a different domain than the current URL

• ImeNotAvailableException Indicates that IME support is not available. This exception is thrown for every IME-related method call if IME support is not available on the machine.

• ImeActivationFailedException Indicates that activating an IME engine has failed.

• ErrorInResponseException An error has occurred on the server side.

• TimeoutException Thrown when a command does not complete in enough time.

• StaleElementReferenceException Indicates that a reference to an element is now “stale” — the element no longer appears on the DOM of the page.

Importance of QA Specialist in an Agile Environment

Agile Team Work Principle

An agile team relies on greater collaboration and clear communication between the team members. Owing to the frequently changing requirements in an agile project, the designers, developers, testers and business analysts work hand-in-hand with each other.

How to Conduct Database Regression Testing?

Why Test a Database?

Just as developers are human and prone to errors, database containing mission critical information and lines of codes on which applications are built is also vulnerable to errors and requires regular maintenance and updating from time to time. Organizations use several applications day in and day out that work on database which gets altered depending on the requirements. The best way to ensure the functioning of these applications is to test them on a regular basis for the new features and changes made in the database. This is what Regression Testing is all about.

What is Regression Testing of Database?

Database regression testing refers to performing regression testing of the database for any new changes and updates made in the database to keep its integrity intact even after modifications in it.

Steps to database regression testing:

Step 1: Decide What to Test in Database Regression Testing?

When it comes to performing regression testing of a database, it is never pre-defined as to which part or kind of database gets updated that requires retesting. The updates keep on occurring in various areas of database and testers need to target those updates to get the most out of their efforts. Generally, database testing comprises of following:

• Incoming data values,
• Outgoing data values (generated for raw queries),
• Database elements (tables, procedures)
• Metadata for web applications
• Data load that’s somewhat we call “load testing”

While performing regression testing of database, if you are adopting an approach of either Black-box testing or White-box testing, your contents for testing would be something like this

• Black-box Testing at the Interface
• O/R mappings (including the metadata)
• Incoming data values
• Outgoing data values (from queries, stored functions, views ...)
• Stored Procedures & Functions
• White/Clear-Box Testing Internally Within the Database
• Database Schema (tables, procedures, etc.)
• Triggers
• View Constraints
• View definitions
• Referential integrity (RI) rules
• Default values for a column
• Data invariants involving several columns

Step 2: Decide When to Test Database for Regression Testing?

When you change something in the existing database or application that runs on the database to refactor it or add new functionality, you need to ensure that you have not broken anything. This requires you to perform regression testing to either fix the errors or roll back to the changes made in the database. Hence, database regression testing is generally done after introducing changes in the database to check the behavior of the modified code. It requires hundreds of new tests to retest in order to check and rectify the newly written code as well as integration with other systems or applications.

If you are working in an agile environment, you need to take the Test-First Approach or Test-First Development (TFD). In this approach of regression testing, you need to prepare test codes parallel to development in order to test the business logics implemented in the database as well as forms, data validation rules, referential integrity, etc. in the database which works like a cycle as mentioned below

• Add a Test
• Run Your Tests
• Update the codes or make changes
• Run Your Tests Again

Step 3: Decide How to Perform Regression Testing of Database?

This is the most important question that comes to the testers’ mind while performing regression testing of the database. Regression testing can be performed both in manual and automated manner.

A common practice in agile team is that testers have their own “sandboxes” to work. A sandbox is basically a technical environment that isolates untested codes form production repository.You can opt for either of them depending on the size of the tests and database.

In my opinion, automated regression testing is more beneficial since it allows testers to test and manage a large number of databases without manual intervention. In other words, it allows you to perform other tasks while the testing automation tool continues to do its job in the background and sends the report of bugs if any.

Automated regression testing of database seems more easy and convenient and adds to the testers’ productivity saving a huge amount of time along with imparting following benefits

• Entire database regression testing process gets automated
• Reduces testing efforts by 50 to 75%
• Testing is done parallel to development
• Extensive test coverage can be achieved

There are several software testing tools that takes care of regression testing. These tools help in getting the testing team up with the database regression testing requirements in no time overcoming several challenges they face in manual testing.