Hadoop programs are primarily about processing data.
Sorting: MapReduce implements the sorting algorithm to sort the output key-value pairs from Mapper by their keys.
Searching: Mapper passes the pattern to search as a distinctive character.
Indexing: Allocated the position of a pattern
NLP: TFIDF, Word2Vec, doc2vec...
and other independent data processing
When dealing with large data sets, it is inevitable that some records will have errors.
While you should make your program as robust as possible to malformed records, you should also have a recovery mechanism to handle the cases you couldn’t plan for. You don’t want your whole job to fail only because it fails to handle one bad record.
Hadoop MapReduce provides a feature for skipping over records that it believes to be crashing a task.
JobConf option together with mapreduce.map.skip.maxrecords and mapreduce.reduce.skip.maxrecords.Assume we have a large dataset. How will we perform regression data analysis now?
Hadoop MapReduce for linear regression is possible by implementing Mapper and Reducer.
It will divide the dataset into chunks among the available nodes and then they will process the distributed data in parallel.
It will not fire memory issues when we run with an R and Hadoop cluster because the large dataset is going to be distributed and processed with R among Hadoop computation nodes.
Also, keep in mind that this implemented method does not provide higher prediction accuracy than the lm() model.
Assume we have data set contains both $y_{n\times 1}$ and $X_{n\times p}$.
The linear model $y=X\beta +\epsilon$ yields the following solution to $\widehat \beta$
$ \hat\beta = (X'X)^{-1}X'y $
The Big Data problem: $n>>p$
In statistics, logistic regression or logit regression is a type of probabilistic classification model.
Logistic regression is used extensively in numerous disciplines, including the medical and social science fields. It can be binomial or multinomial.
Binary logistic regression deals with situations in which the outcome for a dependent variable can have two possible types.
Multinomial logistic regression deals with situations where the outcome can have three or more possible types.
Logistic regression can be implemented using logistic functions.
The logit model connects the explanatory variables in this way
$P_i=\frac{1}{1+\exp(-(\beta_1+\beta_2X_i))}$
Alternatively we can write the model in this way
$\log \frac{P_i}{1-P_i} = \beta_1+\beta_2X_i$
where $P_i/(1-P_i)$ is called the odds ratio: the ratio of probability of a family will own a house to the probability of not owing a house.
This model can be easily estimated with the glm() function in R or sklearn.linear_model.LogisticRegression() in Python.
The logistic regression is different from linear regressions as it does not have analytical solutions
Bad news: The above estimation requires sequential iterative method.
Will the following hypothetical Hadoop workflow work?
Defining the Mapper function
Defining the Reducer function
Defining the Logistic Regression MapReduce function
Logistic regression is the standard industry workhorse that underlies many production fraud detection and advertising quality and targeting products. The most common implementations use Stochastic Gradient Descent (SGD) to all large training sets to be used. The good news is that it is blazingly fast and thus it is not a problem for Hadoop implementation to handle training sets of tens of millions of examples. With the down-sampling typical in many data-sets, this is equivalent to a dataset with billions of raw training examples. The ready to use solutions:
Divide $n$ sample into $k$ blocks that each block consists of $m$ observations。
Do logistic regression with each block on a single node.
$\widehat \beta_l = \arg~\max \sum _{i =1}^m \{ y_{li}x_{li}'\beta-\log(1+\exp\{x_i'\beta\})\}.$
The Full Logistic Regression model with coefficients
$\widehat \beta$ can be approximated by weighted average of $\widehat \beta_l$
$ \widehat \beta = \frac{1}{k}\sum_{l=1}^k \widehat \beta_l. $