Install and running APACHE SPARK (R) from scratch

I suppose that you use R 3.x.x (in my case 3.5-devel) and Rstudio > 1.1.x (in my case 1.2.308).
Updated the 25/3/2018 with the last version of SPARK.

 # In Rstudio, first install the package HelpersMG from CRAN and update it:

install.packages("HelpersMG.tar.gz")
install.packages("http://www.ese.u-psud.fr/epc/conservation/CRAN/HelpersMG.tar.gz", repos=NULL, type="source")

# Then load the HelpersMG library and the last version of SPARK:

library("HelpersMG")
wget("http://apache.crihan.fr/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz")
system("tar -xvzf spark-2.3.0-bin-hadoop2.7.tgz")

# Change the .profile and .Rprofile for future use

SPARK_HOME <- file.path(getwd(), "spark-2.3.0-bin-hadoop2.7")

HOME <- system("echo $HOME", intern = TRUE)

fileConn<-file(file.path(HOME, ".profile"))
total <- readLines(fileConn)
writeLines(c(total, paste0('SPARK_HOME="', SPARK_HOME, '"'), "export SPARK_HOME"))
close(fileConn)

Sys.setenv(SPARK_HOME = SPARK_HOME)

fileConn<-file(file.path(HOME, ".Rprofile"))
total <- readLines(fileConn)
writeLines(c(total, 
                      'if (nchar(Sys.getenv("SPARK_HOME")) < 1) {',
                             paste0('Sys.setenv(SPARK_HOME = "', SPARK_HOME, '")'), 
             '}'
           ))
close(fileConn)

# Now install the sparkR package:

install.packages(file.path(SPARK_HOME, "R", "lib", "SparkR"), repos=NULL)


# You have SPARK ready to be used
# Now start the master on your computer
# If you return from a previous use, just begin here

SPARK_HOME <- Sys.getenv("SPARK_HOME")

system(paste0(file.path(SPARK_HOME, "sbin", "stop-master.sh"), ";", file.path(SPARK_HOME, "sbin", "start-master.sh")))

# And run a slave on your computer; just to test

x <- system("ifconfig", intern=TRUE)
IP <- rev(gsub("^(.*) ([0-9\\.]+) (.*)$", "\\2", x[grep("inet ", x)]))[1]

system(paste0(file.path(SPARK_HOME, "sbin", "start-slave.sh"), " spark://", IP, ":7077"))


# Let try to run a computing:

library("SparkR")

spark_link <- paste0("spark://", IP, ":7077")
sparkR.stop()
sc <- sparkR.session(master = spark_link,
                     appName = "Nom de la session",
                     sparkEnvir = list(spark.driver.memory = "2g"))

output <- spark.lapply(1:100, function(x) {x*2})

Don't expect to have exceptional result in such a configuration ;) But it works.

Commentaires

Enregistrer un commentaire

Posts les plus consultés de ce blog

Standard error from Hessian Matrix... what can be done when problem occurs

An Hessian matrix is a square matrix of partial second order derivative. From the square-root of the inverse of the diagonal, it is possible to estimate the standard error of parameters. Let take an example: val=rnorm(100, mean=20, sd=5) # Return -ln L of values in val in Gaussian distribution with mean and sd in par fitnorm<-function(par, val) {   -sum(dnorm(val, par["mean"], par["sd"], log = TRUE)) } # Initial values for search p<-c(mean=20, sd=5) # fit the model result <- optim(par=p, fn=fitnorm, val=val, method="BFGS", hessian=TRUE) # Inverse the hessian matrix to get SE for each parameters mathessian <- result$hessian inversemathessian <- solve(mathessian) res <- sqrt(diag(inversemathessian)) # results data.frame(Mean=result$par, SE=res)   library(MASS) (r<-fitdistr(val, "normal")) It works well. However, the inverse of the matrix cannot be calculated if the second order derivative for some paramete
Lire la suite

stepAIC from package MASS with AICc

Due to a bug in dropterm() and addterm() in package MASS, it is impossible to use AICc with stepAIC() . Here is a solution. Enter these commands and now you can use: stepAIC(g0, criteria="AICc") or stepAIC(g0, criteria="BIC") For example: datax <- data.frame(y=rnorm(100), x1=rnorm(100),                     x2=rnorm(100), x3=rnorm(100), x4=rnorm(100)) g0 <- glm(y ~ x1+x2+x3+x4, data=datax) gx_AICc <- stepAIC(g0, criteria="AICc") gx_AIC <- stepAIC(g0, criteria="AIC") gx_AIC <- stepAIC(g0, criteria="BIC")
Lire la suite

Install treemix in ubuntu 20.04

Informations are available here  https://bitbucket.org/nygcresearch/treemix/downloads/ In terminal, enter for  Ubuntu 20.04: wget https://bitbucket.org/nygcresearch/treemix/downloads/treemix-1.13.tar.gz tar -xvf treemix-1.13.tar.gz sudo apt-get install libgsl-dev sudo apt install libboost-dev sudo apt install libboost-all-dev cd treemix-1.13/ ./configure make sudo make install And it works: $ treemix TreeMix v. 1.13 $Revision: 231 $ Options: -h display this help -i [file name] input file -o [stem] output stem (will be [stem].treeout.gz, [stem].cov.gz, [stem].modelcov.gz) -k [int] number of SNPs per block for estimation of covariance matrix (1) -global Do a round of global rearrangements after adding all populations -tf [file name] Read the tree topology from a file, rather than estimating it -m [int] number of migration edges to add (0) -root [string] comma-delimited list of populations to set on one side of the root (for migration) -g [vertices file name] [edges file name] read the g
Lire la suite